That config looks correct. I'm being that guy... I'd be tempted to do a full machine restart.
Make sure you've issued those commands to the other servers, and done a restart to solidify the config.
I haven't used the mlxfwreset command, but looking at the docs, without the level argument, it is only doing the lowest level of what the adapter supports.
A physical 'shutdown -r now' has always worked for me.
Thank you! I was able to configure and run, I had problems with i40e & i40iw drivers.
Sorry I am new to 10G BASE-SR and I can't seem to find a good resource that can confirm if an SFP supported in a Cisco Nexus 5548UP will be compatible with a new service The new service is described as '10 Gigabit Ethernet LAN PHY IEEE 10G BASE-LR10.3125 Gbps +/- 100 ppm 1310nm'
Ultimately I need to understand if a 'cisco sfp-10g-sr' for which the transmitter wavelength spec is described as 850nm is usable.
Thanks for your patient, i plan to take it and any site recommend?
Hi,
I have two switches that i'am very pleased with. An MSX1012B-2BFS and an MSX6012F-1BRS_WT (as shown in web console).
I have to put one switch on a site and the other on an other site : 6 kilometers between sites.
So i want to install on each switch an MC2210511-LR4 module.
The optical fiber between the sites have been tested.
My questions are:
1. Will this optical QSFP+ module (40GbE) work on each switch ? If so, on what port ? Or any port will be OK ?
2. What does mean the '' WT '' letters on the MSX6012 model ? (Wide Transceiver ??) If so, does that mean that i need to buy an another MSX6012F_WT ? Or the MSX1012B will also work fine ?
3. Actually, my MSX6012F is in VPI profil mode. Do i need to put it in single_ethernet mode ? (i don't do infiniband on the network for now, but it is envisaged).
4. As our needs are evolving, and my two switches already have almost all their ports in use, I plan to buy two MSX6036F to replace the MSX1012B and MSX6012F. Will both MC2210511-LR4 modules will work and, if so, on which ports?
Many thanks for your help !
Regards.
Hi Sophie,
I doublechecked the packages. Instead of tcl8.4 I have tcl8.6 installed and instead of libnuma I have libnuma1. Could this be the issue?
Version 4.4-2.0.7.0 does give me the same error messages as posted before. I can't even run mlnxofedinstall and get it to finish properly.
What can I do to provide more information so you can help me?
Thank you for your reply!
Hi Chris,
have you received a solution for your problem?
I'm quite interested because we just have installed a new cluster in our compagny and, unfortunately, we have the same error messsage than you when we launch some MPI jobs.
It is a very strong problem for us since, up to now, we are not able to start our studies.
Thank you.
Hi,
I'm looking the best value of max_vfs on SR-IOV configuration of virtualization host. I find following statement on ConnectX-5 EN brochure.
SR-IOV: Up to 1K Virtual Functions
SR-IOV: Up to 16 Physical Functions per host
Let's say that the virtualization host is equipped with total of 40 cores (80 threads). Should I use 1024 or 64 (1024/16) for max_vfs? If not, what is the optimal one for that environment? How to utilize all ConnectX-5 EN VF? How to utilize all ConnectX-5 EN PF?
Best regards,
Hi Raphael,
RoCE v2 is a UDP based protocol, and UDP, unfortunately, does not guarantee delivery, ordering or duplicate protection of the packets.
In the case if you have an additional programming questions, I would suggest to ask the question on linux-rdma mailing list.
Hi Pratik,
Oohh I see. Will try this then and will give an update.
Thank you.
Hi,
netdev mailing list - netdev mailing list might be a better place to raise the question.
Hello Gerald,
Many thanks for reaching out to the Mellanox Community.
Unfortunately, the ConnectX-2 adapters you have are EOL/EOS for a while now. Even though the firmware you are running is the latest available, the current versions of the driver do not support the ConnectX-2 adapter anymore.
Also we notice, that you are running on an AMD Ryzen platform, which we do not test. The only AMD platform we test and certify the driver and adapters for is the EPYC platform.
Never the less, you can try if applying the recommendation from the following community post, improves the performance. The link is https://community.mellanox.com/docs/DOC-3086
Our recommendation is to try to get your hands on a pair of ConnectX-3 adapters which are fully supported with our current drivers.
Many Thanks and regards,
~Mellanox Technical Support
Hi Tibor,
Many thanks for reaching out to the Mellanox Community.
Please check if reburning your firmware when the card is booted in the Livefish mode, resolve your issue. If this does not resolve the issue, the card is dead.
For booting you card into Livefish mode, please follow the instructions in the MFT (Mellanox Firmware Tools) User Manual ( http://www.mellanox.com/related-docs/MFT/MFT_user_manual_4_10_0.pdf ), Appendix C: "Booting HCA Device in Livefish Mode".
For correctly burning the firmware in this mode, please use the following command:
# flint -d <device> -i <firmware bin file> -nofs -guid <card guid> burn GUID can be found on the HCA sticker.
Example:
# flint -d /dev/mst/mt26428_pciconf0 -i MNPH29C-XTR.bin -nofs –guid 0002C903000B6D1C burn
After successful burning the firmware, please reboot the card in Normal mode.
Many thanks,
~Mellanox Technical Support
Hi Garegin,
Many thanks for reaching out to the Mellanox Community.
As the ConnectX-4 is using the same mlx5 PMD driver as the ConnectX-5, the ConnectX-4 is supported for using with VPP.
For VPP support, please refer to the VPP Mailings list.
Many thanks.
~Mellanox Technical Support
Hi, Experts:
When deploying VM, I have meet an issue about mlx5_mac_addr_set() to set a new MAC different with the MAC that VMWare Hypervisor generated,
and the unicast traffic (ping) fails, while ARP has learned the new MAC. Since Mellanox NIC is not set anti-spoofing by default,
the VMWare lloks to add some anti-mac-spoofing functionality. The VF is configured through pciPassthru.
I have googled online and some link mentions to set the following parameters in VMX, but it looks NOT working:
pciPassthru0.noPromisc = "false"
pciPassthru0.noForgedSrcAddr = "false"
pciPassthru1.noPromisc = "false"
pciPassthru1.noForgedSrcAddr = "false"
(Note: it is not SR-IOV, and there has no up-link vswitch Port Group there for passthru pciDevice).
Someone else may have asked this question already. I have two questions:
1): Does Mellanox community has finalized the method or document? or no solution yet?
2): How to check (which instruction) whether Mellanox NIC anti-MAC-spoofing status?
Thanks.
-- Edward
Hi Community,
I'd like to understand better a problem we have, which seems to be linked to the fact that DPDK's xstats/ethtool -S shows a lot of "rx-out-of-buffer" packets. I found the performance counter document, but it does not say much about why this could happen, which buffer we're speaking about. I quote "Number of times receive queue had no software buffers allocated for the adapter's incoming traffic.". As rx_nombufs (DPDK stats) is 0 I guess it does not mean that there is not enough software buffers. Are they some internal MLX buffers? What can be done to prevent that?
Thanks,
Tom
Is there any error messages? Does ibtracert work (#ibtracert <src lid> <dst list>?
ibtracert works
We actually have connection but we are only able to ibping to the GUID that is binded on OpenSM but can't ibping to the other GUIDs now.
If your question is if it possible to use different MOFED version for different kernel, then it is not . However, you should be able to install Mellanox OFED on one kernel with --disable-kmp option, reboot to other kernel, recompile same Mellanox OFED using mlnx_add_kernel_support and after that install only kernel modules.
According to you, the issue somewhere in OS (I/O, memory allocation, other) and not in the network. 20Gbps on ConnectX-2 will give you maximum theoretical 16 Gbps because of 8/10 encoding, so 15.6 Gbps is pretty close.
I would suggest to use perf to analyze ssh/rsync behaviour, or maybe 'strace -ttt -T' option in order to see how much time it spends in the system calls
I found one bug in the Mellanox drivers. In the RT kernel on the Centos 7.5 system, the ptp4l or phc2sys processes do not want to start automatically as a service (systemctl). There is no problem with drivers built into the kernel.