Thank you se-rulez for explaining this.
The reality does not look so brilliant as the commonly used figures.
20Gbps * 0.8 (DDR overhead) * 0.8 (PCI<=2.0 overhead) = 12.8 Gbit
40Gbps * 0.8 (QDR overhead) * 0.8 (PCI<=2.0 overhead) = 25.6 Gbit
Is a dual port ConnectX-3 (PCI 3.0) on the storage server my option for this scenario to saturate storage side IB?
Dual port PCI 3.0 IB 40/Gbps card ======== QDR IB switch ====== 8 clients single port cards 10 or 20 or 40Gbps (ConnectX or ConnectX-2)
As PCI 3.0 encoding overhead is negligible should I expect a 64 Gbps pure data coming to the storage server?