Manual for getting Infiniband in KVM

Hi there!

We've been using Infiniband (IPoIB) for a while now quite successfully. We now want to virtualize some of our servers with KVM. Everything works quite fine - apart from Infiniband. I don't know whether it is important but I have a CentOS host and guest.

I already found out, that I have to use SR-IOV and basically followed this manual. The only thing I changed, is the /etc/modprobe.d/mlx4_core.conf to emulate an Infiniband device instead of an Ethernet device:

[host]# options mlx4_core port_type_array=1,2 num_vfs=16 probe_vf=0

My SR-IOV seems to be installed correctly as my lscpi-output is quite the same as it is in the manual:

[host]# lspci  | grep -i Mel

07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

07:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

...

Trying the same in my guest this also shows the SR-IOV-Function

[guest]# lspci | grep -i Mel

00:06.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

I can even set up the Infiniband-Device and bring it up:

[guest]# cat /etc/sysconfig/network-scripts/ifcfg-ib0

DEVICE=ib0

TYPE=InfiniBand

ONBOOT="yes"

BOOTPROTO="static"

IPV6INIT="no"

IPADDR="10.10.100.15"

BROADCAST="10.10.255.255"

NETMASK="255.255.0.0"

NETWORK="10.10.0.0"

So far, so good. I can ping an external machine from the host - but I cannot ping it from the guest.

Furthermore ibstatus tells me that the state of the device is down:

[guest]# ibstatus

Infiniband device 'mlx4_0' port 1 status:

        default gid:     fe80:0000:0000:0000:0000:0000:0000:0000

        base lid:        0x15

        sm lid:          0x1

        state:           1: DOWN

        phys state:      5: LinkUp

        rate:            10 Gb/sec (4X)

        link_layer:      InfiniBand

I am quite new to InfiniBand but something tells me - so I wonder what's the difference between the phys state and the state. (By the way: The host has the state ACTIVE.)

So, I have two questions:

1. What did I do wrong or forget to do to activate the guests Infiniband?
2. Is there a complete manual or anything like that which I can follow to understand what I am doing?

Thanks a lot in advance!

Daniel