Mellanox Switch SX6036
Head Node
Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
total_cores: 16
MLNX_OFED_LINUX-3.0-1.0.1 (OFED-3.0-1.0.1)
ib0: 192.168.89.2
# ibstat
Rate: 56
Storage Node
Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
total_cores: 16
MLNX_OFED_LINUX-3.0-1.0.1 (OFED-3.0-1.0.1)
ib0: 192.168.89.1
# ibstat
Rate: 56
On the head node I have /share/nas that's pointed to the storage node that's across the IB subnet.
When I do an rsync -Pv from the head node to /share/nas it will copy at ~130MB/s. What's odd is that I'll let it finish, delete the files on /share/nas and do the rsync again and the speeds will be at 330MB/s.
I did an ib_send_bw test and it does show BW peak of 6000MB/s.
ib_send_bw -F 192.168.89.2
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 2048[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x04 QPN 0x021c PSN 0xd33f3e
remote address: LID 0x03 QPN 0x043c PSN 0x3e09e6
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 1000 6013.12 6006.08 0.096097
---------------------------------------------------------------------------------------
So I'm confused. Could it be a limitation of my hard drive? If so why do I see speeds of 133MB/s and then 330MB/s? I've ran mlnx_tune -P default to see if it improves but the behavior is the same. Thanks.
EDIT Ran a qperf test
qperf 192.168.89.1 tcp_bw tcp_lat
tcp_bw:
bw = 1.06 GB/sec
tcp_lat:
latency = 7.96 us