We have a small network for testing. Its layout is shown below.  The orange-colored IP addresses are the current IPoIB setup. They are done with /etc/sysconfig/network-scripts/ifcfg0-ib0 using IP addresses in an existing subnet 192.168.11.0/24.  The switch is a Mellanox SN2410.  As indicated in the picture, the gateway IP address for this subnet is 192.168.11.3, assigned to a 10GbE interface of the bastion host.

At present, the SB7700 runs SM.

With this IPoIB setup,

• I have run into problems with them using e.g. ipv_rc_pingpong - please see below for test session:
• We can't get more than 10Gbps with iperf3 or iperf over such IPoIB IPv4 addressse, despite the fact that we have a SB7700 IB switch!

A. Server:

I noticed the GID as :: which is not right to me.

B. Client:

As above, the GID is shown as ::.

ping to each IP address works however.

ibping also works:

A' Server:

B' Client:

All nodes run:

OFED:  MLNX_OFED_LINUX-3.3-1.0.4.0-rhel7.2-x86_64

OS: CentOS 7.2: Linux 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

As such, I suspect that we can't use the 192.168.11.0/24 for IPoIB.  But to use another subnet, the question is where and how do I setup the gateway?

Setting up IPoIB seems to be simple, especially the Mellanox Academy has this short video: "IPoIB Performance Measurement".  Towards the end, the speaker for the video showed the use of ifconfig to setup temporary IPv4 addresses for the two IB equipped servers.  But IMHO that's not enough!   According to my understanding, an Ethernet broadcast domain like what IPoIB uses is an "emulated" one. Thus, somewhere there must be a gateway.  I believe this should be done in the IB switch. But I have Googled a lot without finding any info as to how to do this part. I read carefully my SB7700 IB switch manuals, no clues there either.

I am still getting up to speed with setting up InfiniBand, although I am very comfortable in configuring switches and HCAs already.  If there is something that I missed, please let me know.

Thanks!

Hello,

I advise you to contact Mellanox Support (support@mellanox.com) and describe your project, configuration and further elaborate on what you're trying to do with Mellanox NICs over ARM platforms.

Hi, we have installed compute nodes with RHELS6.3 and MLNX_OFED_LINUX-2.0-2.0.5 using an Infiniband network. We intend to upgrade to RHELS6.7 and mlnx-ofed 3.x.

In a rolling upgrade scenario, can we have both versions of OFED co-existing on the same fabric?

thanks, Greg

Should I expect any difference between ConnectX 3 Pro and ConnectX 4 EN in this case?

Title says about all;

We are starting our cloud-hosting company at 2 sites, in the future it will be 3 sites in total. For now however, we start at 2 sites.

I have a SX1036 with dual PSU that I am using as my main switch for both sites. At each site i have 1x10gbe transit/internet, and 2x (redundand ) 10gbe wave interconnect between the sites.

My question; How will I connect both switches so I can live migrate vm's (Use RDMA) from one DC to another? Is there a max. distance that I can cross? How far is this distance?

How far can I go with 1310ns fiber using SX1036 / ConnectX-3 EN / VPI PRO without losing my RDMA capabilities?

Is the latency important? I am still choosing which area my colocation be in, and i am wondering if datacenters with short distance between eachother (Lets say  250microseconds, or 0.25ms) has any real benefits in terms of RDMA and general application-load balancing. If there is no real benefit in having sub 1ms latency between sites, instead of say 10ms I will consider going with long distance between my sites for geo-redundancy.

Can you guys advise me on the benefits of having multiple datacenters that are very close together, other than redundancy? Would you guys consider this to be a bad idea ? I'm guessing that DC's that are further away from eachother do have some benefits in terms of latency to clients/customers. This is a big plus ofcourse. The thing is though, I am using SSD, NVME and RAM only as storage, and my internal network is very low latency. My gut tells me I should try to also have a minimal latency between my DC's, but I cant really think of any major benefits that would justify not having a wider area of customers that I can offer sub 10ms services. Could you guys share your opinion on this subject?

Anyone managing multiple sites that can give me some advice?

Hi David,

1) Driver version 2.4.0 is an Ethernet driver only so if you have a VPI card, please make sure the port protocol is configured as 2 which is Ethernet.

#/opt/mellanox/bin/mlxconfig -d /dev/mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

2) Enable SRIOV with the MFT tool (Mellanox FW tool) as follow:

#/opt/mellanox/bin/mlxconfig -d /dev/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=16

3) Confirm BIOS setting supports SRIOV and/or virtualization.

4) Set the driver module parameters for VF:

#esxcli system module parameters set -m mlx4_core -p 'num_vfs=<VFs over Port1, VFs over port2, 0> port_type_array=2'

This is all documented in the driver UM.

Thank you,

Sophie.

I afraid it is tricky to understand without debugging and source code, but most likely there is a problem there. Try to print the values of lkey/rkey, QP numbers, protection domains, other related data in every packet you send/receive. Check that a receiver is running first, before sending posting its send request. What is you run only one sender and one receiver, does it work?

Regarding performance, you might check the Mellanox Tuning guide. What is the test you are using, it is part of 'perftest'?

And probably most important question - do you use Mellanox OFED or this is inbox driver? is it Linux?

Thank you eddie.notz for your response.

Yes.

Both port 1 and port 2 status had been 'active' till then. (I did not apply any updates etc in recent and all updates are always manually installed to avoid any potential issues, perhaps...).

The second port status is still 'Initializing' but the port is properly working.

7 days later and no reply from Mellanox... Seriously?

Hi Sopie

we don't want to use it as Ethernet card but as IB over IP. That's the problem that I can't get it to work as a 40GB IBoIP card in ESX 6 anymore, which worked perfectly with ESX 5.5 and the 1.8.2.4 drivers (which are not supported anymore).

Would you please also link to the driver UM you are referring too?

We have Mellanox Infiniband FDR cards and cables supporting it. We want to reduce the speed of the adapters from 56 Gbps to 40 Gbps. We tried using ibportstate <base lid> <port id> of the node. However, it does not help. Please let me know the exact sequence of commands to do so.

Card :

Mellanox Tech. MT27500 Family [ConnectX-3]

Switch : Mellanox 6536 Switch

ibportstate <lid > <port> fdr10 0 espeed 30
ibportstate <lid> <port> reset

Hi guys,

I have a question that is real important for me; How can I connect my two sites so that I am able to use RDMA between them?

I have 2 racks, each located in a different datacenter. Each rack has got its own SX1036 Ethernet switch, and all servers are connected using connectx-3 EN or VPI PRO adapters. The server-switch cabling is all in the same rack, so these cables are passive. The time has come to connect my the sites together, so I will be needing active fiber between the sites.

I have the following options to choose from, please note that I am fairly low on budget, so buying a metro-x long haul switch is out of the question.

I can rent a fiber-ring, giving me 2 Darkfibers, or either go with a 10gbe L2 or L3 ring.

Lets say I take the fiber-ring, can I just buy 4 QSFP+ LR 10km tranceivers and install these into my SX1036's, connect both darkfibers to both switches and I'm good to go?

I really hope someone can explain how to tackle this problem without losing RDMA capability between the servers.

Thanks,

CloudBuilder

I have been kept digging, and here is what I have observed:

I just did a

[root@sc2u0n0 ~]# lsinitrd /boot/initramfs-3.10.0-327.28.2.el7.x86_64.img|less

[...]

drwxr-xr-x  5 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core

-rw-r--r--  1 root    root        21149 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_addr.ko

-rw-r--r--  1 root    root        82669 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_cm.ko

-rw-r--r--  1 root    root      159989 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_core.ko

-rw-r--r--  1 root    root        77565 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_mad.ko

-rw-r--r--  1 root    root        51765 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_sa.ko

-rw-r--r--  1 root    root        33549 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_ucm.ko

-rw-r--r--  1 root    root        35829 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_umad.ko

-rw-r--r--  1 root    root        87141 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_uverbs.ko

-rw-r--r--  1 root    root        68989 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/iw_cm.ko

-rw-r--r--  1 root    root        75765 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/rdma_cm.ko

-rw-r--r--  1 root    root        37637 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/rdma_ucm.ko

drwxr-xr-x  12 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb3

-rw-r--r--  1 root    root      233861 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb4

-rw-r--r--  1 root    root      286621 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ipath

-rw-r--r--  1 root    root      439149 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ipath/ib_ipath.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx4

-rw-r--r--  1 root    root      250733 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx5

-rw-r--r--  1 root    root      192053 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mthca

-rw-r--r--  1 root    root      221773 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/nes

-rw-r--r--  1 root    root      274037 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/nes/iw_nes.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ocrdma

-rw-r--r--  1 root    root      131165 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/qib

-rw-r--r--  1 root    root      600661 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/qib/ib_qib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/usnic

-rw-r--r--  1 root    root      135229 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko

drwxr-xr-x  7 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/ipoib

-rw-r--r--  1 root    root      161509 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/iser

-rw-r--r--  1 root    root        85917 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/iser/ib_iser.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/isert

-rw-r--r--  1 root    root        91245 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/isert/ib_isert.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srp

-rw-r--r--  1 root    root        85757 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srp/ib_srp.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srpt

-rw-r--r--  1 root    root        92813 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko

[...]

Yes. I am new to IB, but I am very sure the above is from the distro.  The mlnxofedinstall perl script doesn't run dracut -f, as proved by the following:

Thus, even the Mellanox drivers/kernel modules are loaded with its openibd running, I suspect that both the inbox and Mellanox ones are loaded.  The outcome under such a circumstance is not something I have much experience - since I am new to IB. But my experience with other drivers tells me that it's not going to be good.  If this is indeed the case, then there is a bug in the mlnxofedinstall perl script (missing a critical step!) that should be fixed ASAP.

I have started thinking whether I should use the uninstall.sh that comes with MLNX_OFED to uninstall it, and then just use the inbox IB modules/drivers.

Driver 2.4.0 include IPoIB driver and works!

Absolutlely! driver 2.4.0 located in ethernet driver section, but why this driver 2.4.0 include IPoIB driver & works?

You can do that using MetroX and InfiniBand connection for 10Km, we don't support RoCE for such distance.

This is mainly due to packet loss and buffer size of the switch on each side (depends on the speed as well).

Ophir.

I resolved the problem by moving all IPoIB IPv4 addresses to a different subnet.

Good!

Looking at the current driver selection across different OS/cards/medium (IB vs. ETH) it looks like the only space consistently supported by Mellanox is Linux. Indeed, for Linux you have everything:

• every card is supported (all the way from Connect-X2 to Connect-X5)
• IB and ETH and the possibility to switch from one to another for the cards that support it (VPI), and you can even use IB on one port and ETH on another
• iSER initiator is supported across the board and iSER target is supported with both LIO and SCST
• SRP initiator is supported across the board and SRP target is supported with SCST

So, if you use KVM as your hyperwisor , there is no problem.

However, if you want to use Mellanox IB technology in conjunction with currently the most popular hyperwisor (VMware ESXi), you're in trouble:

• there is no official support for ESXi 5.5 and up for any card older than Connect-X3
• the only VPI cards supported in IB mode are Connect-X3/Pro
• Connect-IB cards are not supported at all
• Connect-X4 cards are supported only in ETH mode
• dual-port VPI cards support only the same protocol (IB or ETH) on both ports, not a mix
• SRP initiator is no longer available
• iSER initiator is available only with 1.9.x.x drivers only over ETH and only for Connect-X3/Pro cards
• the current IB driver 2.3.x.x is compatible only with ESXi 5.5 (not 6.0!), works only with Connect-X3/Pro cards and includes neither SRP nor iSER initiator

My question is very simple: what's the long term strategy of Mellanox with regards to hyperwisor support? Are they suggesting that everyone considering Mellanox products should switch to KVM as their hyperwisor of choice? Or they should abandon RDMA and use Mellanox adapters/switches only as 56/100Gbe network infrastructure?

I would REALLY appreciate some reaction from Mellanox staff, who no doubt have already seen this thread, but, for some reason, chose not to react to it...

please refer to this post: Is it possible to connect 2 ethernet switches (sx1036) located up to 10km apart without losing RDMA?