Let me try to explain PCI Generations, lane width and the cards as well.
There are three components here:
1. PCIe Generation, lane width,
2. IB speed- QDR/FDR etc
3. Encoding schemes
1. PCI Gen 2 speed supports upto 5Giga Transfers per second per lane. Assuming 8 lanes, the maximum theoretical BW is 40GT/s. That translates to 40Gbps theoretically.
PCI Gen 3 supports upto 8GT/s, assuming 8 lanes, theoretical maximum will be 64Gbps, with 16 lanes, approximately 128Gbps theoretical.
Ill combine #2 and #3 below.
FDR: Fourteen Data Rate,so 14Gbps per lane. IB runs over 4 lanes so 56Gbps, needs minimum of PCI Gen 3x8 slot
QDR: Quad data rate. so 10Gbps per lane. IB runs over 4 lanes so 56Gbps, Minimum of PC Gen 2 x8 slot
Here is the kicker: FDR uses 64b/66b encoding which translates to 3% overhead.
QDR uses 8b10b encoding which translates to 20% over head.
Factor this back into 1.
QDR:
a) Between two switches: 40Gbps theoretical*0.8 (accounting for 20% over head) = 32Gbps
b) From server to Switch: PCI Gen 2 also uses 8b10b so you will see a further loss in actual rate. 32*0.8 = 25.8 Gbps. You will see close to 25Gbps on some motherboard/CPUs where as some others will not be able to push beyond 20Gbps.
Note, this limitation is from the PCI slot and not the card. If you were to get a dual port card, you could use it either for redundancy or for running ethernet on one port and IB on the other. But from purely a BW perspective, there isnt much use to getting a dual port ConnectX-2 card.
FDR:
a) Between two switches: 64Gbps theoretical *0.3, plus accounting for some preamble and CRC overhead, actual data rate is 56Gbps. (Note, this is almost double of what QDR can provide)
b) From server to switch with PCI Gen 3.0 x8 slot: I think there is a small overhead wrt the slot but you should see 52~53Gbps on each port. Again, dual port card makes sense for redundancy.
However, we do have a Connect-IB card that is able to make use of x16 lane on PCIe Gen 3.0 and you can get close to or just over 100Gbps with the dual port Connect-IB card.
This card is not a VPI card.
Let me know if you have any further questions