Yeah, as Yair Ifergan mentioned, don't connect port 0 to port 0, then port 1 to port 1.
If you're connecting two boxes directly to each other, connect both of their port 0's to port 1 on the other. eg Kind of like crossing over the cables. Then run OpenSM (opensmd) on both. That way all 4 ports should come up as "ACTIVE".
I do this (using RHEL/CentOS) and it works fine with IPoIB layered on top. I can give you the CentOS ifcfg-ethx files, but I'm not sure they'd be helpful as Debian/Ubuntu does networking differently.