What are driver and fw versions for HCA, what OS and switch model/version? What numbers are you getting?
Start with checking that IB utilities give you expected BW for IB links (32Gb/s data for QDR and 54Gb/s for FDR - due to 8/10 and 64/66 encoding correspondingly). If your numbers too off look for flopping ports, bad cables, errors counters.