Even your SSD's local max of 500 MB/s is only 4 Gb/s which is nowhere near of PCIe ver2 limit of about 25Gb/s for x8 slot (ConnectX3 cards are x8 and I assume you have it in x8 slot).
To eliminate concerns about end-to-end connectivity - check it with ib_read_bw and ib_write_bw utilities first. If you get 25-26 Gb/s with PCIe gen2 or full 54-55 Gb/s with PCIe gen3 (btw, the direction of the transfer doesn't matter in case of a gen2 talking to gen3 node) then nothing wrong with your fabric - you need to look for bottlenecks somewhere else.
Check with Linux utilities like mpstat/vmstat if you have a lot of IO waits and what your CPU cores are doing. IO tests can be tricky as your SCSI layer on top of SRP and the test utility itself may not interact nicely with each other: try to use more threads, async i/o options, and different io queue depth settings just to name a few.