Dear Mellanox Support,
I've problems with OFED 2.2. Neither raw_ethernet_bw / raw_ethernet_lat nor libvma can create a QP flow / call verbs.
At some point on that machine it used to run fine but then we added more network interfaces and it seems to break it somehow.
I got all the latest MLNX software & firmware. Flow steering is disabled and there is nothing useful in dmesg. Could you please help me to figure out what's wrong? See basic info below.
[root@aurarb01 log]# uname -a
Linux aurarb01 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11 17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@aurarb01 log]# ofed_info
MLNX_OFED_LINUX-2.2-1.0.1 (OFED-2.2-1.0.0):
[root@aurarb01 log]# raw_ethernet_bw --server
Max msg size in RawEth is MTU 1518
Changing msg size to this MTU
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx4_1
Number of qps : 1 Transport type : IB
Connection type : RawEth Using SRQ : OFF
RX depth : 512
CQ Moderation : 100
Mtu : 1518[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
MAC attached : 24:BE:05:81:95:90
error: Function not implemented
Couldn't attach QP
[root@aurarb01 log]# LD_PRELOAD=libvma.so sockperf sr
VMA INFO : ---------------------------------------------------------------------------
VMA INFO : VMA_VERSION: 6.6.4-0 Release built on 2014-04-23-16:54:38
VMA INFO : Cmd Line: sockperf sr
VMA INFO : OFED Version: MLNX_OFED_LINUX-2.2-1.0.1:
VMA INFO : Log Level 3 [VMA_TRACELEVEL]
VMA INFO : ---------------------------------------------------------------------------
sockperf: == version #2.5.233 ==
VMA ERROR : rfs[0x23888f0]:188:create_ibv_flow() Create of QP flow ID failed with flow dst:172.17.38.37:11111, src:0.0.0.0:0, protocol:UDP
VMA ERROR : rfs[0x241aa70]:188:create_ibv_flow() Create of QP flow ID failed with flow dst:172.17.38.38:11111, src:0.0.0.0:0, protocol:UDP
VMA ERROR : rfs[0x243e240]:188:create_ibv_flow() Create of QP flow ID failed with flow dst:172.17.38.36:11111, src:0.0.0.0:0, protocol:UDP
sockperf: [SERVER] listen on:
[ 0] IP = 0.0.0.0 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
[root@aurarb01 log]# ibv_devinfo
hca_id: mlx4_1
transport: InfiniBand (0)
fw_ver: 2.31.5050
node_guid: 24be:05ff:ff81:9590
sys_image_guid: 24be:05ff:ff81:9593
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: HP_0280210019
phys_port_cnt: 2
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
port: 2
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.31.5050
node_guid: 24be:05ff:ff94:fae0
sys_image_guid: 24be:05ff:ff94:fae3
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: HP_0230240009
phys_port_cnt: 2
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
[root@aurarb01 log]# ifconfig
eth0 Link encap:Ethernet HWaddr 24:BE:05:94:FA:E1
inet addr:172.XX.XX.XX Bcast:172.17.38.39 Mask:255.255.255.248
inet6 addr: fe80::26be:5ff:fe94:fae1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3987 errors:0 dropped:0 overruns:0 frame:0
TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:300512 (293.4 KiB) TX bytes:812 (812.0 b)
eth2 Link encap:Ethernet HWaddr 24:BE:05:81:95:90
inet addr:172.XX.XX.XX Bcast:172.17.38.39 Mask:255.255.255.248
inet6 addr: fe80::26be:5ff:fe81:9590/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3978 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:299904 (292.8 KiB) TX bytes:684 (684.0 b)
eth3 Link encap:Ethernet HWaddr 24:BE:05:81:95:91
inet addr:172.XX.XX.XX Bcast:172.17.38.39 Mask:255.255.255.248
inet6 addr: fe80::24be:500:181:9591/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3972 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:299492 (292.4 KiB) TX bytes:684 (684.0 b)
eth5 Link encap:Ethernet HWaddr 38:EA:A7:36:99:6A
inet addr:10.128.XX.XX Bcast:10.128.178.63 Mask:255.255.255.224
inet6 addr: fe80::3aea:a7ff:fe36:996a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:869904 errors:0 dropped:0 overruns:0 frame:0
TX packets:1318759 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:447971625 (427.2 MiB) TX bytes:1615917933 (1.5 GiB)
Memory:f3f00000-f4000000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:747 errors:0 dropped:0 overruns:0 frame:0
TX packets:747 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:70889 (69.2 KiB) TX bytes:70889 (69.2 KiB)