Hi Roman,
1. What is the OS running on the servers? It does not appear to be Inbox driver nor Mellanox OFED driver either but open source as you downloaded it from:
Homepage: http://www.openfabrics.org/
Description: OpenIB system files
2. We have (Mellanox) a Performance Tuning Guide which document recommended settings at the BIOS & OS level.
3. Two servers have their HCA cards installed on PCI GEN 2 x4 and the third one PCI GEN 2 x8
Thank you,
Sophie.