Hi Daniel,
you have no loops in this setup ...
in any case, I suggest you have a look here:
Designing an HPC Cluster with Mellanox InfiniBand Solutions
And maybe start with this:
Understanding Up/Down InfiniBand Routing Algorithm
it will help you understand the routing algorithms and networking for InfiniBand.
This is very small network. I would check that all the ports are being used, if there are not so many flows - let's say just one flow, only one port will be utilized - (same in Ethernet)....