Hi, All,
Host:
Processor: Intel SandyBridge dual 8 core
HCA: Connect-X3 FDR
OS: Linux 2.6.32-431.el6.x86_64
MLNX_OFED: MLNX_OFED_LINUX-2.4-1.0.0 (OFED-2.4-1.0.0)
I'm using Qemu-1.7 and KVM for VM.
mlx4_core loading is fine when booting host, and VFs are up.
The issue is that VF can not be activated within VM, when launching VM with ivshmem-size>=512M. However, its fine with size<=256m.
| /usr/local/qemu-1.7/bin/qemu-system-x86_64 | \ |
| -enable-kvm -boot c -cpu host | \ |
| -smp 8 -m 8192 -hda a.img | \ |
| -net nic,macaddr=52-54-00-12-34-4,model=virtio | \ |
| -net tap,ifname=tap0,script=no | \ |
| -device ivshmem,shm=nahanni,size=XXXXXXm | \ |
| -device pci-assign,host=02:01.0,id=hostdev0 | \ |
The dmesg in VM outputs:
Backport based on mlnx_ofed/mlnx-ofed-2.x-kernel.git f10d938
compat.git: mlnx_ofed/mlnx-ofed-2.x-kernel.git
mlx4_core: Mellanox ConnectX core driver v2.4-1.0.0 (Feb 5 2015)
mlx4_core: Initializing 0000:00:05.0
mlx4_core 0000:00:05.0: setting latency timer to 64
mlx4_core 0000:00:05.0: Detected virtual function - running in slave mode
mlx4_core 0000:00:05.0: Sending reset
mlx4_core 0000:00:05.0: Got slave FLRed from Communication channel (ret:0x1)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:1)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:2)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:3)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:4)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:5)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:6)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:7)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:8)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:9)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave is currently in themiddle of FLR. retrying...(try num:10)
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: slave driver version is not supported by the master
mlx4_core 0000:00:05.0: Communication channel is not idle.my toggle is 1 (cmd:0x0)
mlx4_core 0000:00:05.0: Failed to initialize slave
mlx4_core: probe of 0000:00:05.0 failed with error -5
I googled and found this post LKML: Yinghai Lu: [PATCH v2 6/7] PCI: Make sure VF's driver get attached after PF's.
Basically, it's kernel trying to load mlx4 drivers for VFs before PF's is loaded.
Any suggestions? Thanks in advance!