Hi there,
Changes in the MTU size for an IPoIB pkey can be tricky. Its always possible to decrease, but increasing might not.
The first device connected will set the MTU size for the broadcast domain. i.e. subsequent connection of IPoIB interfaces will not be allowed to set an MTU size larger than the previous one.
So my suggestion is:
-disconnect all IPoIB interfaces attached to the partition (pkey interfaces)
- configure Subentmanager with the requested MTU size (like MTU=5 in partition.conf)
-restart SM
-make sure that the first interface added to the partition will use mtu size 4092
-- example: in linux I would do as follows
echo 0x0001 > /sys/class/net/ib0/create_child
echo 4092 > /sys/class/net/ib0.8001/mtu
ifconfig ib0.8001 10.10.10.10/24 mtu 4092 up
and then check that the mtu really IS 4092 before adding the interfaces on the second node.
ref: RFC4391 Transmission of IP over InfiniBand (IPoIB)
5. Setting Up an IPoIB Link
The join operation is successful only if the Subnet Manager (SM)
determines that the joining node can support the MTU registered with
the broadcast group [RFC4392] ensuring support for a common link MTU.