For MTT values, you no longer need to set them if you are using MLNX_OFED 2.x versions. So which version of MLNX_OFED are you using?
This seems that if you are getting memlock limit to 64KB, it has something to do with environment when you are running the job. I have seen similar for other job schedulers (Setting Up TORQUE for Running HPC Jobs), but I haven't run into it for SLURM. I thought SLURM would just use whatever you have set, unlike other job schedulers. So I think it has something to do with your environment.
Usually the MLNX_OFED install would update some values in /etc/sysctl.conf, then after a reboot you should be all set. So if you are running outside of a job scheduler, you should check for ulimit -l.
Maybe upgrade to the latest MLNX_OFED and see if it improves?