Hello,
I have a big problem to get the right performance for KVM virtualization (SR-IOV) on top of CentOS7 Host.
I'm using ConnectX-3 card and MLNX_OFED_LINUX-3.0-2.0.1 version. The firmware is up-to-date (2.34.5000).
I've checked that MPI performance is OK for CentOS6.5 Host/bare-metal and CentOS6.5 KVM image (both Infiniband/Ethernet).
I've used exactly same OFED/firmware version and same applications for two CentOS6.5 and CentOS7 cases.
But, the KVM-SR-IOV performance is too bad on top CentOS7.
I tried different KVM-OS (CentOS6.5 and CentOS7 VM images) on top of CentOS7 host, but the result is same.
Between two CentOS7 host/bare-metal machines, MPI performance is OK.
Between two CentOS6.5 or CentOS7 KVM images on top of CentOS7 host, MPI performance becomes too bad in case of <=32KB message size.
For example, only 17% bandwidth is gotten within KVM-SR-IOV in case of 4KB MPI message size.
I'm using 3.10.0-229.4.2.el7.x86_64 kernel.
To enhance the KVM performance on top of CentOS7, I could upgrade kernel to v.4.0.1, but that kernel is not supported by OFED.
While following the section 3.12.1 of "Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf", I tuned for Hypervisor.
The bios setup is totally same between CentOS6.5 test and CentOS7 test.
Is there any way to get the similar performance (bandwidth/latency) between two KVM-SR-IOV images on top of CentOS7 also? Any help is welcome!