Remember that this limitation is from server to switch and is a function of the PCI capabilities rather than Infiniband protocol.
Is a dual port ConnectX-3 (PCI 3.0) on the storage server my option for this scenario to saturate storage side IB?
Ans: ConnectX-3 FDR card will push upto 53Gbps, Connect-IB will push a little over 100Gbps on one PCI slot, with dual ports.
Dual port PCI 3.0 IB 40/Gbps card ======== QDR IB switch ====== 8 clients single port cards 10 or 20 or 40Gbps (ConnectX or ConnectX-2)
Ans: Each port from Connect-IB card to the QDR switch will max out at QDR speeds which is 32Gbps of actual data
If you go with FDR switch, you will benefit from the Connect-IB switch. Note that in most cases at 100Gbps, BW is not the limiting factor unless you have state of the art storage with highly capable CPUs.
As PCI 3.0 encoding overhead is negligible should I expect a 64 Gbps pure data coming to the storage server?
Ans: No see my calculation from above, also pasted below.
FDR:
a) Between two switches: 64Gbps theoretical *0.3, plus accounting for some preamble and CRC overhead, actual data rate is 56Gbps. (Note, this is almost double of what QDR can provide)
b) From server to switch with PCI Gen 3.0 x8 slot: I think there is a small overhead wrt the slot but you should see 52~53Gbps on each port. Again, dual port card makes sense for redundancy.
Please feel free to email support@mellanox.com. One of the SE's can help you design your system per your requirement.