Time for a dumb question from my side: the ib ports go into an "HP 4X QDR InfiniBand Switch Module for c-Class BladeSystem" (part number 489184-B21). Since I haven't seen any mention in the switch's specs for supporting ethernet mode, should I even attempt to use mlx4_en ? It seems to me that special support would be required in the switch ports for that, and the HP QDR switch is (ipo)ib - only.
In any case, the performance over ipoib went up from 3Gb/s to 11Gb/s curiously after I did this:
- run a netperf via ipoib -> 3Gb/s (100% CPU)
- run a netperf via ipoib with SDP -> 18Gb/s (100% CPU)
- run a netperf via ipoib, no SDP -> 11Gb/s (80% CPU)
so maybe enabling SDP flipped some setting that now gets me 11Gb/s at ~80% CPU. Which is plenty for what we need.
Thanks yairi, Sorin and Justin for your persistence in helping me get this set up.