Thanks for sharing your experience. I did the following:
[root@sc2u0n0 ~]# dmidecode |grep PCI
Designation: PCIe Slot 1
Type: x8 PCI Express 3 x16
Designation: PCIe Slot 3
Type: x8 PCI Express 3
lspci -vv
[...]
02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
[...]
LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
So, the theoretical speed should be 8Gbps/lane x 8 lane x 128b/130b = 63 Gbps. In fact, we just did a fio sweep using fio-2.12. The read is quite reasonable. We are now investigating why the write is so low.
A. Read test results
- Chunk size = 2 MiB
- Num. Jobs = 32
- IO Depth = 128
- File size = 500 GiB
- Test time = 360 seconds
| Mode | Speed, Gbps | IOPS |
|---|---|---|
| psync, direct | 47.77 | 2986 |
| psync, buffered | 24.49 | 1530 |
| libaio, direct | 49.17 | 3073 |
B. Write test results
- Chunk size = 2 MiB
- Num. Jobs = 32
- IO Depth = 128
- File size = 500 GiB
- Test time = 360 seconds
| Mode | Speed, Gbps | IOPS |
|---|---|---|
| psync, direct | 24.14 | 1509 |
| psync, buffered | 9.32 | 583 |
| libaio, direct | 22.51 | 1407 |