Thanks, Blair! I've setup exactly same OFED/FW/NIC/Switch/OS versions on each host/VM (CentOS6.5/7.1).
...
Host Driver Version .................... MLNX_OFED_LINUX-3.0-2.0.1 (OFED-3.0-2.0.0): modules
Firmware version: 2.34.5000
vlan 1000 is setup on Switch/NIC
<HOST>
Linux xxx 3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
6: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether e4:1d:2d:01:12:40 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
...
vf 9 MAC 5a:16:3e:6c:d9:2f, vlan 1000, spoof checking off, link-state auto
...
vf 15 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
07:01.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
<VM>
Linux mk-test-inst1.novalocal 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 5a:16:3e:6c:d9:1f brd ff:ff:ff:ff:ff:ff
00:04.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
...
To check result on both 2 CentOS 6.5 hosts and 2 CentOS 7.1 hosts, I'm using same machines/setup. Even though I tried to make new VM images (CentOS 6.5/7.1) again, the bandwidth result is still bad. Should I have to do something more (that's not needed on CentOS6.5 host) on CentOS7.1 to get reasonable bandwidth between 2VMs? Could you please explain the QoS parameters in the guest?? I tried same guest/VM (default QoS) on CentOS6.5 and CentOS7.1. Only the VM on CentOS7.1 have a problem.