Hello Robert,
Thank you for posting your question on the Mellanox Community.
Based on the information provided, we noticed that you also opened a Mellanox Support case which is already assigned to one of our engineers.
We will continue to assist you further through the Mellanox Support case.
Thanks and regards,
~Mellanox Technical Support
Hi,
we have 2 brand new SN2700 100G Switches. It shipped with Onyx and since the beginning one of them was acting weird on the Management Interface. Even after upgrading to latest version 8190, the switch is rebooted and not pingable, BUT if i start pinging anything from serial connection, the gateway or the second device, the network connecticity is restored.
I ping 10.0.100.100 -t and get no answer from the switch. If i then go on the serial and ping 10.0.254.254, or 10.0.100.101 and the switch begins to be available from my admin host again. Network guys checked the other switch where the mgmt0 is connected to, and point to maybe ARP problems ? Anything else i can do to make it stable ?
The second switch worked fine out of the box, im not sure whats wrong with that device.
Best regards
Tim
Hi,
packets are captured by Wireshark and so far I never got any issues with packet ordering when using it.
When splitting valid UDP header and payload by using linked mbufs (see example code), the order of the received packets (at least captured with Wireshark) is not as expected - anyway all packets are received, none are lost. I expect that the first packet in the TX array is the first one that is received. Which is the case, when I use the same mbuf for header and payload and don't split them.
The strange thing is, that when I set
ip_hdr->version_ihl = 0x40; // (and not 0x45)
the packets are received in the correct order (same order as TX array).
Thanks and best regards
Sofia
Hey guys, sorry did not realize that posting to the Mellanox community was the same as opening a support case. We were told via other email avenues that we needed to open a support ticket to really get anyone technically involved, so we took that avenue and were then instructed to open a support case attached to the card S/N + support contract. Sorry for any confusion.
we recognize slow speed on an AMD H11SSL-i single socked Boards with the latest Bios 1.0C
Setup
Two servers connected over a Mellanox SN2100 switch.MSN2100
Bios settings
C-State disabled
IO-SVR disabled
SMV enabled
Perforamce mode enabled
On this setup, we run an Ubuntu Kernel 4.15.0-37.40,
but we recognize this problem also with the 4.19.1 mainline kernel
and an old ubuntu 4.4 Kernel.
We use to try two different kinds of nic
First the Mellanox MCX556A-ECAT
Second the Mellanox MCX456A-ECAT
The problem is with an iperf3 parallel benchmark test
we only reach 60GBit on the MCX556A-ECAT and 50GBit on the MCX456A-ECAT.
With an Intel Broadwell (SuperMicro X10SRi-F Board), the iperf3 test shows results near 90 GBit what is ok with Spectre and Meltdown mediation.
Has anyone an Idea what could be the reason?
Hi Jorge,
Please open a case with mellanox support supportadmin@mellanox.com
Hello Vivienne,
Based on the original information provided, can you please update the MLNX_OFED version you are using the latest version, which is version 4.4 and check if this resolves the issue.
If the issue still persists, please open a Mellanox Support case through support@mellanox.com
Thanks and regards,
~Mellanox Technical Support
Hello Yao,
Thank you for posting your question on the Mellanox Community.
Based on the information provided, the counters you are mentioning (rx_vport_rdma*/tx_vport_rdma*) do not display drops, these are informative counters related to the RDMA packets received and transmitted.
Please use the following link for a more information related to the mlx5 counters -> https://community.mellanox.com/docs/DOC-2532
In the current upstream kernel, the only way to capture RDMA(RoCE) traffic is through the following validated and tested solution through a Docket container.
Kindly refer to the following procedure:
In order to capture RDMA packets , kindly refer to the following procedure using a Docker (Container)
This container is simple, elegant and fastest way for users to start using RDMA devices to capture and analyze RDMA packets using favorite tool tcpdump.
tcpdump is extended to directly sniff/capture traffic using RDMA verbs making using of latest Linux kernel services.
We tested the Docker on the following setup:
Installation instructions:
In case of using MLNX_OFED, please refer to Section 3.1.16 'Offloaded Traffic Sniffer' of the UM ( http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v4_4.pdf ) and the Mellanox Community document -> https://community.mellanox.com/docs/DOC-2416
Thanks and regards,
~Mellanox Technical Support
Hi Jillis,
In order to better assist you , i suggest to open a support ticket at support@mellanox.com to further discuss this matter.
Thanks,
Samer
Hi Tim,
Can you share the output of:
# show interface mgmt0
# show interface mgmt0 configured
# show ip route
Please recreate the issue and try the following:
Connect to the switch using the console, and run the following command:
# tcpdump -i mgmt0 arp or icmp
After this and while tcpdump is on start a ping from the remote host.
Are you seeing the remote host ARP or ICMP echo requests ?
Hi Victor,
- Did you reset the configuration by running:
#mlxconfig -d /dev/mst/mt4099_pciconf0 reset
#reboot
#flint -d /dev/mst/mt4099_pciconf0 q
#mlxconfig -d /dev/mst/mt4099_pciconf0 q
Note: If this still does not work, you can reflash the FW:
#flint -d /dev/mst/mt4099_pciconf0 -i <FW image> b
#reboot
- Regarding the port type change (permanent) to ethernet , use the following:
mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
Note : (2) ETH (1) IB
Thanks,
Samer
Hello, everyone.
In the community, I found the method to set the DCQCN parameters on ConnectX-4.
It writes "adjust the parameters under the path /sys/kernel/debug/kernel/mlx5/".
However, the method could not be applied to ConnectX-3 Pro. And the path "/sys/class/net/*device name*/ecn" does not exist.
So how can I enable DCQCN on ConnectX-3 Pro?
Thanks.
Hi,
Please note that ECN/DCQCN is not supported on ConnectX-3.
We do support RoCE with PFC in ConnectX-3 Pro, you can refer to the following guides:
HowTo Configure RoCE v2 for ConnectX-3 Pro using Mellanox SwitchX Switches
HowTo Run RoCE over L2 Enabled with PFC
Recommended Network Configuration Examples for RoCE Deployment
Thanks,
Samer
Hi, Samer
Thank you for your help. I saw that the DCQCN paper (Congestion Control for Large-Scale RDMA Deployments) said ConnectX-3 implemented DCQCN in firmware. So I wonder how to enable it.
One more question, can I configure the DCQCN on the 100Gbps ConnectX-5 EN card like ConnectX-4? Actually, I could not find relevant documents for ConnectX-5 EN in the community.
Thank you.
Hello,
I would like to suggest reviewing the following community page:
https://community.mellanox.com/docs/DOC-2489
Also, would like to suggest using iperf2 instead of iperf3 and use more threads by adding the -P flag.
If the performance will still be lower, please open a case with Mellanox support for further investigation by sending an email to support@mellanox.com.
Best Regards,
Viki
Hi Dave,
It is hard to point what could cause MPI not to work with FDR10 after all the changes you made.
I suggest to try installing Mellanox HPC-X driver that includes latest openMPI and run basic osu_bw test without special parameters and check if it works.
For example:
mpirun -hostfile hostfile.txt -np 2 <HPC-X directory>/tests/osu-micro-benchmarks-5.2/osu_bw
HPC-X can be downloaded from here:
http://www.mellanox.com/page/products_dyn?product_family=189&mtag=hpc-x
You can also try using the openMPI from Mellanox OFED driver, but then you should use the latest MLNX_OFED driver that is suitable for your OS.
Make sure also that all edges are configured to FDR10.
In case you still won't be able to run MPI, please open a case with Mellanox support by sending an email to support@mellanox.com.
Best Regards,
Viki
Hi ,
Regarding ConnectX-5 it is the same as ConnectX-4:
HowTo Configure DCQCN (ECN) Parameters for Windows Driver (WinOF-2)
This post illustrates how to set the DCQCN (ECN) parameters for WinOF-2 driver (ConnectX-4 and ConnectX-5 adapter cards).
HowTo Configure DCQCN (RoCE CC) values for ConnectX-4 (Linux)
Regarding ConnectX-3 , DCQCN was not implemented in the firmware.
We recommend using the tested and verified solution RoCE with PFC for ConnectX-3.
Thanks,
Samer
Thanks, very helpful.
Hi,
Do you run over Beegfs over Ethernet or Infiniband ?
Can you run standard benchmark like ib_send_bw for RDMA test just to see if we can get maximum bandwidth and ib_write_lat to check the latency ?
Thanks
Marc
Hi,
Go through this link to see how to configure tc and vxlan offload
https://github.com/roidayan/ovs-tests/blob/master/setup_tc_vxlan_offload.sh
Regards
Marc