Hi all,
I'm testing SR-IOV on Xen 4.2 with a ConnectX-3 MT27500 VPI adapter card. I have installed the mlnx ofed 2.2-1.0.1 with sr-iov support on Dom0 (Xen 4.2 over CentOS 6.4). The installation script has finished correctly and file mlx4_core.conf looks like this:
[root@hypervisor ~]# cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core num_vfs=8 port_type_array=1,2 probe_vf=1 enable_64b_cqe_eqe=0
All seems to be ok because the lspci shows the eight virtual functions:
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
01:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.4 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.5 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.6 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.7 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:01.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
In Dom0, I can use the PF and one VF:
[root@hypervisor ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:f452:1403:0006:9421
base lid: 0x7
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 1 status:
default gid: fe80:0000:0000:0000:0014:0500:0000:0088
base lid: 0x7
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
I try to pass through a VF to a full virtualized guest and I can use the card in guest without problems.
Full virtualized guest detects the VF and it works correctly:
[root@VM1 ~]# /etc/init.d/openibd restart
Unloading HCA driver: [ OK ]
Loading HCA driver and Access Layer: [ OK ]
Setting up InfiniBand network interfaces:
Bringing up interface ib0: [ OK ]
Setting up service network . . . [ done ]
[root@VM1 ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
00:04.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
00:05.0 Multimedia audio controller: Ensoniq ES1370 [AudioPCI]
[root@VM1 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0014:0500:0000:008d
base lid: 0x7
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
However when I try to pass through a VF to a paravirtualized guest.
The card seems to be detected correctly:
[root@VM0 ~]# lspci
00:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
But I cannot restart the openibd.
[root@VM0 ~]# /etc/init.d/openibd restart
Unloading HCA driver: [ OK ]
Terminado (killed)
Loading Mellanox MLX4 HCA driver: [FAILED]
^C
Please wait ...
Loading Mellanox MLX4_IB HCA driver: [FAILED]
^C
Please wait ...
Loading Mellanox MLX4_EN HCA driver: [FAILED]
Loading HCA driver and Access Layer: [FAILED]
And the dmesg command shows:
[root@VM0 ~]# dmesg
mlx4_core: Mellanox ConnectX core driver v1.1 (Oct 22 2014)
mlx4_core: Initializing 0000:00:00.3
mlx4_core 0000:00:00.3: enabling device (0000 -> 0002)
mlx4_core 0000:00:00.3: Xen PCI mapped GSI0 to IRQ29
mlx4_core 0000:00:00.3: Detected virtual function - running in slave mode
mlx4_core 0000:00:00.3: Sending reset
mlx4_core 0000:00:00.3: Sending vhcr0
mlx4_core 0000:00:00.3: HCA minimum page size:512
mlx4_core 0000:00:00.3: Timestamping is not supported in slave mode.
BUG: unable to handle kernel paging request at ffffc9000032c00c
IP: [<ffffffff812ebacc>] msix_capability_init+0x29c/0x300
PGD 782ba067 PUD 782bb067 PMD 783be067 PTE 8010000000000464
Oops: 0003 [#1] SMP
Modules linked in: mlx4_core(O+) compat(O) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 knem(O) coretemp hwmon crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 microcode joydev xen_netfront pcspkr ext4 jbd2 mbcache xen_blkfront dm_mirror dm_region_hash dm_log dm_mod [last unloaded: compat]
CPU: 0 PID: 1609 Comm: modprobe Tainted: G O 3.10.55-11.el6.centos.alt.x86_64 #1
task: ffff880004fdc040 ti: ffff880004e1e000 task.ti: ffff880004e1e000
RIP: e030:[<ffffffff812ebacc>] [<ffffffff812ebacc>] msix_capability_init+0x29c/0x300
RSP: e02b:ffff880004e1fa28 EFLAGS: 00010286
RAX: ffffc9000032c00c RBX: ffff880074f14000 RCX: 0000000000000005
RDX: 0000000000000001 RSI: ffff880077ff0280 RDI: ffff880077ff0280
RBP: ffff880004e1fa78 R08: 00000000f3cd794f R09: 000000005448e430
R10: 00000000000a1dfa R11: 00000000000a1ca7 R12: 0000000000000000
R13: ffff880004d367c0 R14: 0000000000000000 R15: ffffc9000032c00c
FS: 00007f64e9ba4700(0000) GS:ffff88007f200000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff80010ffb8010 CR3: 0000000005213000 CR4: 0000000000042660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880004cd2b80 ffff880074f14860 00000000000080d0 c06b880000002000
ffff880074f14000 ffff880004cd2b80 ffff880074f14000 ffff8800749b0000
ffff880004cd2b80 ffffffff81812328 ffff880004e1faa8 ffffffff812ebc3f
Call Trace:
[<ffffffff812ebc3f>] pci_enable_msix+0x10f/0x160
[<ffffffffa019c7bd>] mlx4_enable_msi_x+0x14d/0x200 [mlx4_core]
[<ffffffffa019e2c4>] mlx4_load_one+0x9c4/0xdb0 [mlx4_core]
[<ffffffffa019eb3c>] __mlx4_init_one+0x48c/0x590 [mlx4_core]
[<ffffffff815f5801>] ? printk+0x4d/0x4f
[<ffffffffa019ecb2>] mlx4_init_one+0x32/0x60 [mlx4_core]
[<ffffffff812d4d0e>] local_pci_probe+0x4e/0x90
[<ffffffff812d4e21>] __pci_device_probe+0xd1/0xe0
[<ffffffff812d6162>] ? pci_dev_get+0x22/0x30
[<ffffffff812d61aa>] pci_device_probe+0x3a/0x60
[<ffffffff813d181a>] really_probe+0x7a/0x250
[<ffffffff813d1a2e>] driver_probe_device+0x3e/0x60
[<ffffffff813d1afb>] __driver_attach+0xab/0xb0
[<ffffffff813d1a50>] ? driver_probe_device+0x60/0x60
[<ffffffff813d1a50>] ? driver_probe_device+0x60/0x60
[<ffffffff813cfca4>] bus_for_each_dev+0x94/0xb0
[<ffffffff813d164e>] driver_attach+0x1e/0x20
[<ffffffff813d1098>] bus_add_driver+0x1e8/0x250
[<ffffffffa0236302>] ? mlx4_init+0xaa/0xaa [mlx4_core]
[<ffffffff813d2194>] driver_register+0x74/0x160
[<ffffffffa0236302>] ? mlx4_init+0xaa/0xaa [mlx4_core]
[<ffffffff812d62bc>] __pci_register_driver+0x4c/0x50
[<ffffffffa02362d2>] mlx4_init+0x7a/0xaa [mlx4_core]
[<ffffffffa0236310>] __init_backport+0xe/0xcfe [mlx4_core]
[<ffffffff81002042>] do_one_initcall+0x42/0x170
[<ffffffff810c43f0>] do_init_module+0x80/0x1f0
[<ffffffff810c6168>] load_module+0x3e8/0x5a0
[<ffffffff810c3ad0>] ? mod_sysfs_teardown+0x150/0x150
[<ffffffff810c3b67>] ? copy_module_from_user+0x67/0xc0
[<ffffffff810c31f0>] ? module_sect_show+0x30/0x30
[<ffffffff810c6463>] SyS_init_module+0x93/0xa0
[<ffffffff81601dd9>] system_call_fastpath+0x16/0x1b
Code: 41 83 c7 0c e8 56 ca df ff 4d 63 ff 4d 03 7d 20 41 8b 17 41 0f b7 45 02 41 89 55 08 83 ca 01 c1 e0 04 83 c0 0c 89 c0 49 03 45 20 <89> 10 41 89 55 08 49 8b 45 10 41 83 c6 01 48 39 45 b8 4c 8d 68
RIP [<ffffffff812ebacc>] msix_capability_init+0x29c/0x300
RSP <ffff880004e1fa28>
CR2: ffffc9000032c00c
---[ end trace 74e9d763ef2ef85b ]---
Any help is going to be appreciated.
Thanks in advance,
Javi