Apparently the port-type gets reset to ib after I rmmod/modprobe mlx4_core:
# echo eth > /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port2
# echo eth > /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port1
# dmesg | tail -n 3
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.8.3 (June 2012)
mlx4_en 0000:06:00.0: Activating port:1
mlx4_en: 0000:06:00.0: Port 1: Port: 1, invalid mac burned: 0x0, quiting
# rmmod mlx4_en mlx4_core
# modprobe mlx4_core
# modprobe mlx4_en
# cat /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port*
ib
ib