Regarding the MTU, I think that is part of the Mellanox OFED driver. What size are you using? It looks like it can be set up to be as high as 4k. In the older releases, the cap was 1500 and everything above it was "silently dropped" when in IPoIB.
http://www.mellanox.com/related-docs/prod_software/ESX_5_X_release_notes.txt
Another option that I have been meaning to try is to see if the 10GbE mode lets you set and user higher MTU.
I was just messaging someone else a second ago, one thing to check is the cables, some of the ones sold online say that they are rated for 10Gb, others say DDR. Not sure if that's true, but I had no problems. I don't think there's anything special with my config but here are some of the results for my esx machine. I'll keep thinking as well.
iblinkinfo:
2 5[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 3 1[ ] "esx2 mlx4_0" ( )
ibnetdiscover:
vendid=0x2c9
devid=0x6732
sysimgguid=0x2c90300015233
caguid=0x2c90300015230
Ca 2 "H-0002c90300015230" # "esx2 mlx4_0"
[1](2c90300015231) "S-0008f104004128f4"[5] # lid 3 lmc 0 "ISR9024D Voltaire" lid 2 4xDDR