How to increase and control loopback bandwidth

[singletrack]

Hi All!

I'm testing a USB 3.2 Gen 1 interface under Linux (Ubuntu 20.04.3 LTS) using the API/example version 2.1.1002 and am seeing ~329 MB/s (2632 Mbit/s) when running the benchmark test (USB_3_BenchmarkTest()) which seems very reasonable and as expected. See below:

Found USB3 plug : firmware: 2.5 PMU356ZYJS speed: 5000 at 0:0
Connecting [Serial: PMU356ZYJS]
Connecting [Serial: PMU356ZYJS]
<snip>
Transfer count 64/1024 - Transfer rate: 1807.55 Mbit/s (Max rate: 1807.55 Mbit/s)
Transfer count 128/1024 - Transfer rate: 2493.68 Mbit/s (Max rate: 2493.68 Mbit/s)
Transfer count 192/1024 - Transfer rate: 2623.66 Mbit/s (Max rate: 2623.66 Mbit/s)
Transfer count 256/1024 - Transfer rate: 2616.75 Mbit/s (Max rate: 2623.66 Mbit/s)
Transfer count 320/1024 - Transfer rate: 2627.85 Mbit/s (Max rate: 2627.85 Mbit/s)
Transfer count 384/1024 - Transfer rate: 2632.03 Mbit/s (Max rate: 2632.03 Mbit/s)
Transfer count 448/1024 - Transfer rate: 2620.37 Mbit/s (Max rate: 2632.03 Mbit/s)
Transfer count 512/1024 - Transfer rate: 2622.42 Mbit/s (Max rate: 2632.03 Mbit/s)
Transfer count 576/1024 - Transfer rate: 2625.30 Mbit/s (Max rate: 2632.03 Mbit/s)
Transfer count 640/1024 - Transfer rate: 2626.49 Mbit/s (Max rate: 2632.03 Mbit/s)
Transfer count 704/1024 - Transfer rate: 2622.89 Mbit/s (Max rate: 2632.03 Mbit/s)
Transfer count 768/1024 - Transfer rate: 2623.98 Mbit/s (Max rate: 2632.03 Mbit/s)

For the loopback test however I'm seeing only ~16 MB/s (I added data rate measurement):

Error Count: 0 - Packets sent: 400 - Bytes sent: 26214400 - Bytes received 26214400, TX rate: 16.83 Mb/s, RX rate: 16.83 MB/s
Error Count: 0 - Packets sent: 500 - Bytes sent: 32768000 - Bytes received 32768000, TX rate: 16.79 Mb/s, RX rate: 16.79 MB/s
Error Count: 0 - Packets sent: 600 - Bytes sent: 39321600 - Bytes received 39321600, TX rate: 16.83 Mb/s, RX rate: 16.83 MB/s
Error Count: 0 - Packets sent: 700 - Bytes sent: 45875200 - Bytes received 45875200, TX rate: 16.80 Mb/s, RX rate: 16.80 MB/s
Error Count: 0 - Packets sent: 800 - Bytes sent: 52428800 - Bytes received 52428800, TX rate: 16.82 Mb/s, RX rate: 16.82 MB/s
Error Count: 0 - Packets sent: 900 - Bytes sent: 58982400 - Bytes received 58982400, TX rate: 16.74 Mb/s, RX rate: 16.74 MB/s
Error Count: 0 - Packets sent: 1000 - Bytes sent: 65536000 - Bytes received 65536000, TX rate: 16.70 Mb/s, RX rate: 16.70 MB/s

If I disable the actual checking of the data at the application level I see a slight bandwidth increase but not very much:

Error Count: 0 - Packets sent: 100 - Bytes sent: 6553600 - Bytes received 6553600, TX rate: 20.04 Mb/s, RX rate: 20.04 MB/s
Error Count: 0 - Packets sent: 200 - Bytes sent: 13107200 - Bytes received 13107200, TX rate: 19.74 Mb/s, RX rate: 19.74 MB/s
Error Count: 0 - Packets sent: 300 - Bytes sent: 19660800 - Bytes received 19660800, TX rate: 19.80 Mb/s, RX rate: 19.80 MB/s
Error Count: 0 - Packets sent: 400 - Bytes sent: 26214400 - Bytes received 26214400, TX rate: 18.63 Mb/s, RX rate: 18.63 MB/s
Error Count: 0 - Packets sent: 500 - Bytes sent: 32768000 - Bytes received 32768000, TX rate: 18.07 Mb/s, RX rate: 18.07 MB/s
Error Count: 0 - Packets sent: 600 - Bytes sent: 39321600 - Bytes received 39321600, TX rate: 17.67 Mb/s, RX rate: 17.67 MB/s

I would like to increase the bandwidth of the loopback testing and retain data checking at the application level.

I note that when calling ConnectUSBPlug() that for USB_3_TEST_MODE there are different modes for benchmark/loopback testing which makes we wonder if there is some hardware related upper bandwidth limit for loopback testing?

typedef enum _USB_3_TEST_MODE
{
USB_3_TEST_LOOPBACK = 0,
USB_3_TEST_BENCHMARK_READ,
USB_3_TEST_BENCHMARK_WRITE,
USB_3_TEST_BENCHMARK_RW
} USB_3_TEST_MODE;

If not, how can I increase the loopback test bandwidth please? I tried with the same buffer size and count as for the benchmark test but this led to an error on the dongle hence why I think there may be a hardware related restriction.

Many thanks in advance for your time and assistance!
Ken

[David (PassMark)]

Expected actual data throughput on real hardware is between 2.4Gbit/s and 3.5Gbit/s depending on the host controller.
In theory, the maximum bandwidth of USB3 port is limited to 3.8Gbit/s (after 8b/10b encoding & protocol overhead).
You can find a more detailed discussion on this page.

Loopback mode is slower as the data needs to do a round trip. Meaning data is sent, then there is a pause, until the data is looped and returned (and then checked for correctness).
So the bus is idle for short periods between bursts of data. This is just the way the loopback test works.

Benchmark mode sends the data as fast as possible. There is no checking of the data and the data is discarded at the other end.

[singletrack]

Thanks for the fast reply! I saw at the link below that for the USB3Test software under Windows that even only in 480 Mbps mode quite a lot higher data rates than I'm seeing in 5 Gbps mode are possible (46.x MB/s).

Is there a coding difference between this software and the API example that accounts for this that I could also apply?

Also, since my port is running at 5 Gbps I would for sure think I could achieve at least this? Or am I missing something perhaps?

Many thanks!


https://www.passmark.com/products/usb3test/index.php

[David (PassMark)]

Expected loopback speeds with default settings on a USB3.0 host controller are around 160MB//Sec (1300 Mbit/sec). See example from USB3Test below.

If you are not seeing these lookback speeds (but the speeds is OK in benchmark mode), then you might have got your calculations wrong when you added data rate measurement to the code. Maybe you are including idle periods? Note that we don't split the speed up into Sent / Receive speeds for loopback mode as they should always be exactly the same for a loopback situation.

[singletrack]

Thanks again for the fast reply. With some tweaks to the loop and how I'm measuring including combining of course the amount of data read/written to give an overall bandwidth I am now seeing ~100 MB/s! Thanks for the hint.

There are no "idle periods" I can see in the LoopbackTest() code itself so is it the case that the hardware is introducing them in loopback mode?

My timer runs continuously around the entire while loop since the individual libusb_bulk_transfer() calls are too short to meaningfully time so any idle periods are included in my measurement. I guess they now account for the rest of the "missing" bandwidth...?

Many thanks!