Additional info.
Have tested with ethernet, same setup, just using ethernet instead and it works perfect for failover. a few seconds with the datastore not contactable and then comes back online.
Other testing done with infiiniband is that when 1 pool/datastore is failed over to the other node, if it does come up, then it will most likely work for the most of the time, but if you failover a second pool, then this second pool will not work initially, if after around 5 minutes it does actually come up as visible from the other node, then the pool that was running ok will almost always then fall off and no longer be visible, even though nothing has actually been done with it. Then sometimes after again 5 minutes or so it will come back.
I thought that it may have been arp updates being the problem, but looking at the arp tables on the esxi servers shows that they are updating correctly.