It turned out to be something funny with Ubuntu 12.04 Desktop. A clean install of Ubuntu 13.04 Server with MLNX_2.0.5 seemed to get everything working off of one port. With 2 ports in a configuration with port 0 to port 0 and port 1 to port 1, I am getting the following error when I try to look at iblinkinfo on the second port guid.
ibwarn: [3161] ib_path_query_via: sa call path_query failed
ibwarn: [3161] main: Failed to resolve 0x0002c903009f5742; attempting full scan
I can ping and run netperf however over the second port ib interfaces.