Hi,
I am trying to set up an Infiniband network to support 10Gbps IPoIB for my vSphere 6 environment.
I have 2 ESXi 6.0.0 hosts with the OFED 2.3.3.1 installed each with a Voltaire 500 EX-D dual port HCA.
These are connected to a SilverStorm 9024-CU24-ST2-DDR switch
I can see the adapters and have created my vSwitch with port groups.
I can ping between the hosts when connected to a port group with no VLAN set
I want to configure VLANs/partitions, I have created several port groups on my vSwitch with VLAN IDs and created pkeys on the subnet manager within the 9025 switch but I am unable to get any traffic to work over these.
I have read about using 0x8000 + VLAN ID so have the following set up:
-> smPKeys
Index: 0 PKey: 0xffff Desc: Default PKey
Index: 1 PKey: 0x8010 Desc: 10
Index: 2 PKey: 0x8011 Desc: 11
Index: 3 PKey: 0x8009 Desc: 9
I have also tried using 0x0000 + VLAN ID
In my vmkernel.log I am seeing things like:
2015-11-15T19:43:59.734Z cpu2:33083)<4>vmnic1:ipoib_mcast_join_complete:642: multicast join failed for ff:12:40:1b:80:10:00:00:00:00:00:00:ff:ff:ff:ff, status -22
2015-11-15T19:43:59.734Z cpu1:33075)<4>vmnic1000102:ipoib_mcast_join_complete:642: multicast join failed for ff:12:40:1b:80:10:00:00:00:00:00:00:ff:ff:ff:ff, status -22
2015-11-15T19:34:31.412Z cpu3:33080)<3>vmnic1000102:ipoib_vmk_def_init:1516: failed to create Multicast QP for pkey = 9
2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1:ipoib_mcast_restart_task:1017: Unable to find Broadcast QP for pkey = 9
2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1:ipoib_mcast_restart_task:1023: Unable to find Multicast QP for pkey = 9
2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1000102:ipoib_mcast_restart_task:1017: Unable to find Broadcast QP for pkey = 9
2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1000102:ipoib_mcast_restart_task:1023: Unable to find Multicast QP for pkey = 9
I can see the pkeys I have created appear on the esxi hosts:
[root@esx21:/var/log] grep -v "0000" /proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/*
/proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/0:0xffff
/proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/1:0x8010
/proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/2:0x8011
/proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/3:0x8009
Have I missed a step somewhere or am I doing something wrong?
Any help would be appreciated.