I have a "InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]" NIC in a Ubuntu 14.04 host serving multiple VM's. I also have a Cisco Topspin 90 switch (6 ethernet ports x 12 infiniband ports). I have the Infiniband ports configured as well as I can find in all of the documentation I can find but I cant get Link:
ifconfig:
ib0 Link encap:UNSPEC HWaddr A0-00-02-20-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.11.1.11 Bcast:10.11.1.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:4092 Metric:1
RX packets:0 errors:0 dropped:228 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
ib1 Link encap:UNSPEC HWaddr A0-00-03-00-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.12.1.11 Bcast:10.12.1.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Under the Cisco TopSpin 90:
port : 1/1
name : 1/1
type : ib4xTX
desc : 1/1 (65)
last-change : Sun Jun 7 16:26:42 2015
mtu : 0
auto-negotiate-supported : yes
auto-negotiate : enabled
admin-status : up
oper-status : down
admin-speed : 10gbps
oper-speed : unknown
link-trap : enabled
phy-state : port-config-training
port : 1/2
name : 1/2
type : ib4xTX
desc : 1/2 (66)
last-change : Sun Jun 7 16:17:54 2015
mtu : 0
auto-negotiate-supported : yes
auto-negotiate : enabled
admin-status : up
oper-status : down
admin-speed : 10gbps
oper-speed : unknown
link-trap : enabled
phy-state : port-config-training
When I try to run opensm:
sudo opensm -o
-------------------------------------------------
OpenSM 4.3.0.MLNX20141222.713c9d5
Reading Cached Option File: /etc/opensm/opensm.conf
Loading Cached Option:guid = 0x0008f104039a72c9
Loading Cached Option:guid = 0x0008f104039a72ca
Loading Cached Option:hm_ca_manual_action = report
Loading Cached Option:hm_sw_manual_action = report
Command Line Arguments:
Run Once
Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 4.3.0.MLNX20141222.713c9d5
Entering DISCOVERING state
SM port is down
Tail of the opensm log file:
Jun 07 18:58:44 879440 [D9948740] 0x03 -> OpenSM 4.3.0.MLNX20141222.713c9d5
Jun 07 18:58:44 879799 [D9948740] 0x80 -> OpenSM 4.3.0.MLNX20141222.713c9d5
Jun 07 18:58:44 881509 [D9948740] 0x02 -> osm_vendor_init: 1000 pending umads specified
Jun 07 18:58:44 884644 [D9948740] 0x80 -> Entering DISCOVERING state
Jun 07 18:58:44 885621 [D9948740] 0x02 -> osm_vendor_bind: Mgmt class 0x81 binding to port GUID 0x8f104039a72ca
Jun 07 18:58:45 092989 [D9948740] 0x02 -> osm_vendor_bind: Mgmt class 0x03 binding to port GUID 0x8f104039a72ca
Jun 07 18:58:45 093169 [D9948740] 0x02 -> osm_vendor_bind: Mgmt class 0x04 binding to port GUID 0x8f104039a72ca
Jun 07 18:58:45 093337 [D9948740] 0x02 -> osm_vendor_bind: Mgmt class 0x21 binding to port GUID 0x8f104039a72ca
Jun 07 18:58:45 093502 [D9948740] 0x02 -> osm_opensm_bind: Setting IS_SM on port 0x0008f104039a72ca
Jun 07 18:58:45 100734 [CA2C3700] 0x80 -> SM port is down
I configured the opensm.conf file with the two guid's from ibstat -p
There arent many commands that one can use to configure the IB ports with and I cant get link... Any help would be greatly appreciated!
Thanks!