Hi Tal,
Thanks for your response, from the author of mlnx_tune too! Great to e-meet you!
Regarding your point 1: For most (if not all) OS, it is next to impossible to query for the I/O non-posted prefetch value. May I make a suggestion? Given Mellanox's industry-wide weight, and the fact there are only two major COTS BIOS makers: AMI and Phoenix, both publish Windows/Linux based BIOS review tools, so perhaps calling such tools is a way to do more reliable queries for BIOS values, including the I/O no-posted prefetch value?
Regarding your point 2: The issue with I/O non-posted prefetch is only relevant when trying to run traffic at rates higher than 150Gb/s (bidirectional). Indeed, we are using four EDR IB HCAs, with SB7700 IB switch. We are pushing for more than 100Gbps traffic so we are in the territory that you mentioned
Finally, one suggestion: the -p comes with the following options, but AFAICT, the only Mellanox doc that explains somewhat is this HowTo Tune Your Linux Server for Best Performance using mlnx_tune Tool published by the prolific Ophir in Oct 22, 2014. IMHO, there is a lack of actual application examples for each profile. An excellent one for illustration would be again Ophir's HowTo Configure and Test BeeGFS with RDMA.
['HIGH_THROUGHPUT',
'IP_FORWARDING_MULTI_STREAM_THROUGHPUT',
'IP_FORWARDING_MULTI_STREAM_PACKET_RATE',
'IP_FORWARDING_SINGLE_STREAM',
'IP_FORWARDING_SINGLE_STREAM_0_LOSS',
'IP_FORWARDING_SINGLE_STREAM_SINGLE_PORT']
Could you please talk to him about adding this piece of info? Since the naming of these profiles all start with IP, which means they apply to Ethernet mostly (IB traffic doesn't use kernel networking stack!). So, perhaps somewhere there should be more explinations regarding
- Is this tuning tool really usable to IB fabric? Ophir said in his 2014 post it is. My review of the Python script didn't convince me so.
- What if all my NIC interfaces use jumbo frames? If so, what is the applicability of IP_FORWARDING_MULTI_STREAM_THROUGHPUT and IP_FORWARDING_MULTI_STREAM_PACKET_RATE?
I look forward to hearing your comments about the above.
Best,
Chin