Hi Philip:
I like your idea about the cables; the Mellanox IB is hyper-proprietary and no doubt this bleeds over into the VPI world; I just had a conversation with a cable engineer about this a few weeks ago of how proprietary the IB world and Mellanox are.
If you see pg 158 in theory if you set AUTO and restart the card should see the 40Gbps pipe. I've not found any setting in any of the manuals (OFED, Linux driver or Performance tuning) that allows you to hard-code a 40Gbps link (Which is unfortunate as otherwise there are many tunable parameters).
http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v2.2-1.0.1.pdf
I'm not familiar with their PCI based card, only chip based (And I concur you have plenty of PCI headroom) and I am wondering are you running an external transceiver or is integrated? If its external Mellanox has an approved list somewhere (its pretty small) and here is one of theirs: http://www.mellanox.com/page/products_dyn?product_family=78&mtag=cables_optical_trans
I'm a huge fan of Mellanox but they are uber-enterprise and it very well may be the card simply won't work with more generic cabling. There background is IB where they are used to charging for the privilege, and its possible they treat those of us in the ETH world the same.
I would enjoy learning how it turns out.
Gage