hi,
What cards/switches do you have? what OS, is this linux?
so, you don't want to use RDMA? just pure ethernet traffic with TCP let's say?
See here for example,
HowTo Run RoCE and TCP over L2 Enabled with PFC
Focus on the Web Services VLAN, and see the examples to enable it the traffic over priority 0, let's say, you can change the priority as well.
Ophir.