Ok, it appears that the 2.9 firmware is not playing nice with the ConnectX cards in ESXi servers.
It is resulting in the "IB_Timeout" and "mkey incorrectly set" errors. That explains the results by swapping the cards above. The 2.7 firmware does seem to work though.
Thanks to Chuckleb (at Serve The Home forums) for the results of his investigation.
RB