A few ideas where to look:
1. Most likely you do not have 4K MTU set on the IB fabric itself. You need to make sure opensm is set with MTU 4K, it is likely set to 2044 – default. If you just have one partition you need to add following line to /etc/opensm/partitions.conf and then restart opensm:
pkey0=0x7fff,ipoib,mtu=5 : ALL=full;
if your opensm runs on the switch – you will need to upload this file to the switch
2. Your switch may or may not support 4K MTU
3. Your card (quite old) and driver may or may not support 4K MTU. See page 18 at http://www.mellanox.com/related-docs/prod_software/Mellanox_IB_OFED_Driver_for_VMware_vSphere_User_Manual_Rev_1_8_2_4.pdf
it says that “maximum value of JF supported by the InfiniBand device is: 2044 bytes for the InfiniHost III family and 4052 / 4092 bytes for ConnectX® IB family”
4. In any case it also make sense to go with the latest
driver: http://www.mellanox.com/downloads/Drivers/MLNX-OFED-ESX-1.8.2.4-10EM-500.0.0.472560.zip