Yes, you can plug a cable between two ports directly but you need to start the Subnet Manager on one machine (/etc/init.d/opensmd, it can be on any machine)
What you lose is scalability, of course. But you get slightly better latency without a switch:)