I remember coming across this exact problem a long time ago setting up a similar config.
I went back through some old correspondence and indeed found that it was ARP requests not being honoured by the infiniband HCAs that was the root cause of the problem.
Apparently whilst IB cards do have ARP functionality, they do not honour the command to update their ARP for any given IP address.
This contradicts your experiences as you suggest that you can see the ARP updates on the interface so I am a little confused.
I would not think that the subnet manager would become involved at all in this scenario as the LID assignment would only ever change on a reboot.
Have you tried using connected mode? Packet headers are slightly different compared to Datagram mode, and is worth a shot.