So I'm doing some experiments with SRIOV on XenServer host and I faced some peculiarities which I don't quite understand.
A few words about the setup. I have two HCA installed: one is a dual-port FDR VPI ConnectX-3 adapter and the second is a dual-port 10GbE ConnectX-3 adatper.
My goal is to get a couple of VFs and pass them through to VMs.
The issues I face:
0) I've enabled SRIOV but I get conflicting values regarding the maximum vfs:
mstflint -dev 81:00.0 dc
...
[HCA]
num_pfs = 1
total_vfs = 16
sriov_en = true
...
vs:
# mlxconfig -d /dev/mst/mt4099_pci_cr1 q
Device #1:
----------
Device type: ConnectX3
PCI device: /dev/mst/mt4099_pci_cr1
Configurations: Current
SRIOV_EN 1
NUM_OF_VFS 63
LOG_BAR_SIZE 3
Which one should I trust? I'd prefer the second one since I don't want to modify and burn the firmware
And also what is the parameter "LOG_BAR_SIZE" responsible for? I have some understanding that it stands for base address register but I don't understand what does it influence and if one would benefit from increasing decreasing it.
1) If I try to generate more than 7 VFs on the 10GbE card I get unassigned class devices is lspci:
# lspci | grep -i mel
...
81:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
81:00.1 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:00.2 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:00.3 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:00.4 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:00.5 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:00.6 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:00.7 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
81:01.0 Unassigned class [ffff]: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] (rev ff)
81:01.1 Unassigned class [ffff]: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] (rev ff)
81:01.2 Unassigned class [ffff]: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] (rev ff)
...
What does it mean, why do they appear and if it is possible to get more than 7 VFs what is the correct way to do it?
I have the following option: num_vfs=01:00.0-15,81:00.0-10
2) Then I would like to have different ports on my HCAs represented by different VFs. According to the manual it can't be done for Infiniband ports unless they are in Ethernet mode. My experiments confirm that. Could someone please explain the reasons for that limitation.
3) I then tried to have a couple of VFs for Infiniband HCA and a couple of VFs for 10GbE card with various combinations of ports:
num_vfs=01:00.0-15,81:00.0-2;2;2
Trying to apply that config I dicovered the following error in dmesg:
[ 879.149339] mlx4_core: The value of 'num_vfs param' is incorrect. The value is discarded!
[ 879.452128] mlx4_core: value '15,81:00.0-2;2;3' of bdf '01:00.0-' in 'num_vfs param' is not valid
Is this by design or is it a bug? The following works fine: num_vfs=01:00.0-0;0;15,81:00.0-2;2;2. But it feels a bit excessive.