Hi everyone,
I have a question regarding the configuration of a computing grid using IB network cards.
We have three computing nodes connected to a master node for jobs submission. Each server has Debian 7.8 and the computing nodes have dual-port Mellanox IB cards (with Mellanox OFED installed on each computing node). We do not use an IB switch, the computing nodes are directly conneted via bidirectional cables:
Node01 IB port 1 to Node02 IB port 2
Node02 IB port 1 to Node03 IB port 2
This cluster/grid is used for parallel computing, using OpenMPI 1.8.4 and Torque/PBS for job scheduling.
We'd like to configure the nodes as a grid, but unfortunately I didn't find any suggestion/configuration for this purpose. I tried using OpenSM on each cabled port, but this didn't work, OpenMPI complaints about multiple GID in the network.
I'm sure the IB cards are working, as ibstat reports the active connections between the nodes.
Please, could you help me?
If you need other information, please, do not hesitate to ask.
Thank you!
Gian