Hi Thomas,
In order to link at 56GbE in a back-2-back setup, you need to change the HCA ini file and re-burn the firmware.
1. Download the .mlx file (fw code file) and ini file ( configuration file )from:
Creating a Customized Firmware Binary - Mellanox Technologies -> Firmware DISCLAIMER
2. Extract the files
3. Find the ini file that belongs to your HCA,
a. ibv_devinfo | grep board_id
b. grep <board_id> *
for example:
# tar xzf ConnectX3-rel-2_31_5050.tgz
# ibv_devinfo | grep board_id
board_id: MT_1090120019
# grep MT_1090120019 *
MCX354A-FCB_A2-A5.ini:PSID = MT_1090120019
4. create a copy of the ini file as a backup
5. edit the ini file and add the following lines under the [IB] section:
port1_802_3ap_56kr4_enable = true
port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_enable = true
port2_802_3ap_56kr4_ability = true
6. create a new firmware bin file:
l-supp-11:/tmp/2.31.5050 # mlxburn -fw fw-ConnectX3-rel.mlx -c MCX354A-FCB_A2-A5_56GbE.ini -wrimage fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin
-I- Generating image ...
-I- Image generation completed successfully.
7. backup the firmware currently loaded on the HCA:
mst start
flint -d mlx4_0 ri backup.bin
8. burn the new firmware onto the card:
flint -d mlx4_0 -i fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin -y b
9. reboot the server
You're done. you can use the same .bin file for the other server if he has the same board_id
Please note that the fw release notes states:
• Added support for 56GbE in all devices supporting 40GbE
• Establishing 56GbE link with SwitchX® requires 56GbE enablement on the
switch
• Establishing 56GbE link on back to back setup requires additions to the INI.
• For further information, please contact Mellanox Support