Hello,
I have a kernel module enhancement request to mlx4_core because of an issue I have running auto-install on systems using ConnectX VPI cards over 10G ethernet.
My issue is that the default behavior of the kernel module is set to infiniband by default instead of auto detect under /sys/module/mlx4_core/drivers/..../mlx4_port1. If I use it for 10G ethernet, and I do, I have 3,500+ ConnectX cards in my environment, auto-installation fails when the initial image loads after PXE booting. Because instead of initializing the connection as 10G ethernet it sets it to IB and network connectivity is lost until I manually intervene.
I have found a way to circumvented the issue by compiling and maintaining my own boot installation image that runs the following code during module insertion:
Inside of /etc/modprobe.d/mlx4_core.conf I put this line:
install mlx4_core /usr/bin/connectx
Which runs the following code in the /usr/bin/connectx script:
#!/bin/bash
/sbin/insmod /lib/modules/`uname -r`/initrd/mlx4_core.ko
PORTARRAY=( `for BUS in /sys/bus/pci/devices/*; do if [ -e ${BUS}/mlx4_port1 ]; then for PORT in $BUS/mlx4_port?; do echo ${PORT}; done; fi; done` );
for PORTS in ${PORTARRAY[@]}; do
/lbin/echo auto > ${PORTS};
done;
/sbin/insmod /lib/modules/`uname -r`/initrd/mlx4_en.ko
exit 0;
It is very similar to the init.d framework you release in the ofed stack where you can set the port type in a configuration file. The problem is that the ofed stack does not exist in this preinstallation environment. I do not see why the default behavior would not be auto detect. This seems like it will be problematic for anyone who uses 10G ethernet on these VPI cards for deployment.