Thanks for the reply! I got some downtime on the storage server to test and have two further clarifications/questions.
- The WinOF version of ib_write_bw doesn't have the -R flag, and the -d flag only works w/ the GUID (0x002c90300519b90)
- On the Linux machine, I can set the -R flag when using it as a server, when trying as a client, I get <Unable to perform rdma_client function>
So here's my output when trying ib_write_bw:
Linux as Server: # ib_write_bw -d mlx4_0 --report_gbits
Win10 as Client: > ib_write_bw -d 0x0002c90300519b90 <server IP>
Server output:
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x01 QPN 0x023d PSN 0x90bfd9 RKey 0x28002b00 VAddr 0x007f13223e2000
Couldn't parse line <c2:30001200:000002983183baa0:00000000000080fe:919b510003c90>
Unable to read to socket/rdam_cm
Failed to exchange data between server and clients
Client output:
------------------------------------------------------------------
RDMA_Write BW Test
Number of qp's running 1
Number of iterations 5000
Massege size 65536
Connection type : RC
Each Qp will post up to 100 messages each time
max inline size 28
local address: LID 0x200, QPN 0x57000000, PSN 0xc23c0000, RKey 0x120030 VAddr 0x0000003183baa0
pp_read_keys: No error
Couldn't read remote address Unknown error (0)
In other news, previously I had both machines @ MTU of 2044 (win default is 4078), since I had read it's a good idea to have matching MTU's throughout the network. In one test, I set the windows MTU back to 4078 and am getting ~5.7Gbps per iperf3 when running w/ 4 threads, which is enough to saturate my server (~600MB/s) at the very least. So then I tried to set the Linux box to 4078 too, but I get <SIOCSIFMTU: Invalid argument>. So even though it's more useable now, I'd like to get the low-level functions working and figure out the fundamentals a little more.