Sorin Iszlai - Generally that's good advice (ie "you need to run opensm"), but for the specific topology mentioned here it's not quite right.
In a 3 node setup, where two nodes have only a single port, and the middle box has two ports doesn't work quite like that.
OpenSM only binds to one port on a server (by default the first one on the first card), then explores/discovers network topology through just that connection. So, if it gets started on the middle box, it won't see the 2nd port nor the other server connected through it. (this is specific to only this topology, and doesn't happen with a switch)
The workaround is super easy, but counter-intuitive. Just run OpenSM on both of the nodes with the single port cards, and don't run it on the middle box.
I do similar to this with some test boxes here, and run IPoIB over the top of it. Works fine that way.
(note - edited slightly for clarity)