Yes 40Gb/s data rate, but sending 8 bits data in a 10 bit packet giving 32Gb/s max data thruput; however the PCIe bus will limit you to about 25Gb/s.
Keep in mind that the performance for hardware to hardware is better than software to software. I've only used Mellanox cards with Linux and the performance for hardware to hardware hits 25Gb/s with ConnectX-2 cards.
The IB equipment you are using has 4 pairs of wire running at 10Gb/s each - hence 40Gb/s total.
Real world file sharing, even with older 10Gb/s InfiniHost cards is better than 10Gb/s ethernet. My MAXIMUM performance tests (using the Linux fio program) are below. That being said we've avoided Windows file servers since at least Windows 2000 - the performance has been terrible compared to Linux; esp. when one factors in the cost of the hardware required.
I would suggest that you compare the exact servers using an ethernet link to see how it compares. In the end theoretical performance is nice - but what really matters is the actual software you are using. In my case going to 10Gb ethernet or QDR IB things like data replication (ZFS snapshots, rsync) went from 90 minutes to sub 3 minutes. It was often not the increased bandwidth but the lower latency (IOPs) that mattered. For user applications accessing the file server - compile times were only reduced by about 30% going to InfiniBand or 10Gb ethernet - but the ethernet is around 10x as expensive. I've not performance tested our Oracle database - but it went to 10Gb ethernet because my IB setup is for the students and I don't trust it yet on a "corporate" server.
In the case of file sharing you'll want to see if you're using the old ports 137 to 139 instead of 445 as that can impact performance.
Also - there is no way to exploit the exceptionally low latency of InfiniBand unless you've got SSDs or your data in RAM.
| Network | GB Data in 30 sec | Aggregate Bandwidth (MB/s, Gb/s) | Bandwidth (MB/s, Mb/s) | latency (ms) | iops |
|---|---|---|---|---|---|
| QDR IB 40Gb/s NFS over RDMA | 94 | 3,100, 25 | 802, 6.4 | 0.615 | 12,535 |
| DDR IB 20Gb/s NFS over RDMA | 24.4 | 834, 6.7 | 208, 1.7 | 2.4 | 3256 |
| SDR IB 10Gb/s NFS over RDMA | 22.3 | 762, 6.1 | 190, 1.5 | 2.57 | 2978 |
| QDR IB 40Gb/s | 16.7 | 568, 4.5 | 142, 1.1 | 3.4 | 2218 |
| DDR IB 20Gb/s | 13.9 | 473, 3.8 | 118, 0.94 | 4.1 | 1845 |
| SDR IB 10Gb/s | 13.8 | 470, 3.8 | 117, 0.94 | 4.2 | 1840 |
| 10Gb/s ethernet | 5.9 | 202, 1.6 | 51, 0.41 | 9.7 | 793 |
| 1Gb/s ethernet | 3.2 | 112, 0.90 | 28 | 17.8 | 438 |
| 100Mb/s ethernet | 346MB | 11.5 | 2.9 | 174 | 45 |
| 10Mb/s ethernet via switch | 36MB | 1.2 | 279kB/s | 1797 | 4 |
| 10Mb/s ethernet via hub | 33MB | 1.0 | 260kB/s | 1920 | 4 |