Re: Observing long delay with RoCE using ConnectX-4 and SN2700

Hi Weijia,

disregard the article about "HowTo Run RoCE over L2 Enabled with PFC" - this is related to switchX configuration-  just follow the server related part there.

For Spectrum SM2700 please follow the below:

In comparison to SwitchX-2, Spectrum based switch need to have its buffer configured separately to enable PFC.

1. Enable the PFC on the switch (the same procedure as before):

for each port run:

dcb priority-flow-control enable force

dcb priority-flow-control priority 3 enable

interface ethernet 1/17/1 dcb priority-flow-control mode on force

2. Configure the lossless buffer size and queue seperatly:

interface ethernet 1/17/1 ingress-buffer iPort.pg3 map pool iPool0 type lossless reserved 90000 xoff 17000 xon 17000 shared alpha 8

interface ethernet 1/17/1 ingress-buffer iPort.pg3 bind switch-priority 3

3. Use the below command to verify:

show buffers details interfaces ethernet 1/17/1

Flags: Y - Lossy, L - Lossless

       S - Static, D - Dynamic

Shared size is in Bytes for static pool and in alphas for dynamic pool.

Interface: Eth1/17/1

  Buffer        Resv    Xoff    Xon     Shared  Pool       Description

                [Byte]  [Byte]  [Byte]  [%/a]

  ------        ------  ------  ------  ------  ----       -----------

  iPort(Y)      0       -       -       inf     iPool0(D)

  iPort(Y)      0       -       -       0       iPool1(D)

  iPort(Y)      0       -       -       0       iPool2(D)

  iPort(Y)      0       -       -       0       iPool3(D)

  iPort.pg0(Y)  90.0K   -       -       inf     iPool0(D)  Data

  iPort.pg1(Y)  0       -       -       0       iPool0(D)

  iPort.pg2(Y)  0       -       -       0       iPool0(D)

iPort.pg3(L)  87.9K   16.7K   16.7K   8       iPool0(D)

  iPort.pg4(Y)  0       -       -       0       iPool0(D)

  iPort.pg5(Y)  0       -       -       0       iPool0(D)

  iPort.pg6(Y)  0       -       -       0       iPool0(D)

  iPort.pg7(Y)  0       -       -       0       iPool0(D)

  iPort.pg9(Y)  20.0K   -       -       inf     iPool0(D)  Control

  ePort         0       -       -       inf     ePool0(D)

  ePort         0       -       -       inf     ePool1(D)

  ePort         0       -       -       inf     ePool2(D)

  ePort         0       -       -       inf     ePool3(D)

  ePort.tc0     1.5K    -       -       2       ePool0(D)

  ePort.tc1     1.5K    -       -       2       ePool0(D)

  ePort.tc2     1.5K    -       -       2       ePool0(D)

  ePort.tc3     1.5K    -       -       2       ePool0(D)

  ePort.tc4     1.5K    -       -       2       ePool0(D)

  ePort.tc5     1.5K    -       -       2       ePool0(D)

  ePort.tc6     1.5K    -       -       2       ePool0(D)

  ePort.tc7     1.5K    -       -       2       ePool0(D)

  ePort.tc8     0       -       -       0       ePool0(D)

  ePort.tc9     0       -       -       0       ePool0(D)

  ePort.tc10    0       -       -       0       ePool0(D)

  ePort.tc11    0       -       -       0       ePool0(D)

  ePort.tc12    0       -       -       0       ePool0(D)

  ePort.tc13    0       -       -       0       ePool0(D)

  ePort.tc14    0       -       -       0       ePool0(D)

  ePort.tc15    0       -       -       0       ePool0(D)

  ePort.tc16    96      -       -       inf     ePool0(D)  Control

  Switch-priority  Buffer

  ---------------  ------

  0                iPort.pg0

  1                iPort.pg0

  2                iPort.pg0

  3                iPort.pg3

  4                iPort.pg0

  5                iPort.pg0

  6                iPort.pg0

  7                iPort.pg0

  8                iPort.pg0

  9                iPort.pg0

  10               iPort.pg0

  11               iPort.pg0

  12               iPort.pg0

  13               iPort.pg0

  14               iPort.pg0

When you perform a test, you can see the pause frame counter is increasing and no discard counter.