ok thanks to a google with another 3 commands on each node I can now using the eipoib interfaces.
the first command creates the ib0 sub interface in this case ib0.1
root@blade01:~# echo .1 > /sys/class/net/ib0/create_child root@blade01:~# ifconfig ib0.1 ib0.1 Link encap:UNSPEC HWaddr A0-00-01-10-FE-80-00-00-00-00-00-00-00-00-00-00 UP BROADCAST RUNNING SLAVE MULTICAST MTU:2044 Metric:1 RX packets:1986136 errors:0 dropped:0 overruns:0 frame:0 TX packets:5610181 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1024 RX bytes:79455932 (79.4 MB) TX bytes:24752145636 (24.7 GB) root@blade01:~#
then the last two commands completed the enslavement
root@blade01:~# echo +ib0.1 > /sys/class/net/eth2/eth/slaves root@blade01:~# echo +ib0.1 00:1b:78:33:6e:95 > /sys/class/net/eth2/eth/vifs
And speed tests are similar to normal ipoib withour larger mtu or SDP setup
root@blade01:~# netperf -H 20.20.20.102 -c -C -- -m 1400 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.20.20.102 (20.20.20.102) port 0 AF_INET : demo Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 65536 1400 10.00 5388.11 21.91 25.73 1.333 1.565 root@blade01:~#