Sweet Im in and able to manage the switch vi the infiniband connection.
The issue was that one of the nics on the box has the same subnet as the switch (192.168.2.x). The SM on the switch itself is trying to manage that subnet so its conflicting with my router. So i changed the IP of the ether port, change the IP of the ib0 interface to be on 192.168.2.x and bam, access granted.
Thanks for your help with eliminating such a trivial problem. cant believe i didnt catch that sooner....
SO the IB interface does inherit the IP i set for the ether on the ISR switch. Good to know.