KVM?
No! KVM also have many limitations.
For example EoIB, etc.
Infiniband communication rely on SM
- Subnet Manager
SM consist of some components and API.
But SM architecture didn't designed for hypervisor world.
Historically many problems were exist in vSphere environment.
1st. ESXi
When vSphere 4.x age VMware give us two choice.
ESX and ESXi
ESX consist hypervisor and OEMed Red Hat console.
ESXi consist hypervisor only.
Some IB tools didn't work in ESXi host in my experience. But that IB tools work in ESX host nicely.
ESXi isn't a general purpose kernel.
I think it cause a major IB driver porting problem.
2nd. Infiniband design itself!!!
Hypervisor control all communication guest vm and host network. RDMA have a kernel by-pass feature called zero copy or RDMA R/W.
This feature controlled by SM, but add hypervisor to this network, many complex modification must add to SM and IB APIs.
There isn't IBTA standard yet.
This will be stadardize in near future.
3rd. RDMA storage protocol.
Infiniband specification concern all of RDMA and ULP protocols.
The improvement of Linux OFED is very fast.
No one know Which OFED version will be port to latest ESXi version.
Many complexity exist and must resolve issues in ESXi environment.
iSER also good candidate for ESXi RDMA protocol, but some critical problem exist.
I think that we must check the latest Linux OFED release note that include so many bugs, limitations.
Linux is a good platform but also suffer from IB's unique limitations.
Conclusion.
I think IB is fastest and eiffient high speed protocol in the planet. But not ready to enterprise network environment yet.
Mellanox said in their product brouchur that they can support major OS environments.
But many case wasn't.
Beta level driver, manual, bug, limitation, etc.
Absolutely!
Many time later all problem will be overcome with new standard and product.
But not now...