1. I have really heavy duty intel infiniband cx4 cables.
1. With subnet manager enabled on cisco switch, I see same activity as listed above, I took that info with openSM running and both ports plugged into the switch and the cisco subnet manager disabled.
1. I enabled the cisco subnet manager and plugged the same cable into two ports, no connection lights.
2. I take same cable and run it between both ports on the HCA and connections light up.
It does appear that i'm now stuck with a dead infiniband switch, or some subnet manager weirdness?