USB3Console.exe Multi-threading in Python

[helloeasyji]

Good Day,

Here we are using USB 3.0 loopback hardware in our manufacturing test environment. 4ea Passmark are connected to the downstream port of our product.
During current testing, we are using Python subprocess to test the USB 3.0 loopback for each Passmark loopback, so basically they are running loopback one by one.
In order to reduce the total test time, we try to modify the code to use threading to run the loopback on different loopback hardware simultaneously.

Here is our partial python code to call the different loopback hardware to run the USB3Console.exe, but looks like the result is quite unstable, sometimes the return code will indicate the loopback hardware can't be find. I would like to confirm if USB3Console.exe is friendly to multi-threading application?


---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
def Passmark_Attach_1():

p = subprocess.Popen(r"M:\\Result\\FunctionalTestScrip tGE\\USB3Console.exe -f", stdout = subprocess.PIPE)
output = p.communicate()
returncode = p.wait()
data = output[0].decode("utf-8").splitlines()
time.sleep(2)
serial_number_array_1 = []
for i in data:

find_device_data = i.split("=")

serial_number_array_1.append(find_device_data[1])

p.kill()
return serial_number_array_1

def run_usb_test(device, usb2_result_array, usb3_result_array):

p_usb2 = subprocess.Popen(r"M:\\Result\\FunctionalTestScrip tBB\\USB3Console.exe -d {} -s USB2 -t 2 -l 1".format(device), stdout=subprocess.PIPE)
returncode_usb2 = p_usb2.wait()
if returncode_usb2 == 0:

usb2_result_array.append("Pass")

else:

usb2_result_array.append("Fail")

p_usb3 = subprocess.Popen(r"M:\\Result\\FunctionalTestScrip tBB\\USB3Console.exe -d {} -s USB3 -t 2 -l 1".format(device), stdout=subprocess.PIPE)
returncode_usb3 = p_usb3.wait()

if returncode_usb3 == 0:

usb3_result_array.append("Pass")

else:

usb3_result_array.append("Fail")

def Passmark_USB_Test(m):

usb3_result_array = []
usb2_result_array = []
result = {}
if len(m) < 1:

result.update({'USB 2 Result': 'Fail', 'USB 3 Result': 'Fail'})

else:

threads = []
for i in m:

thread_usb = threading.Thread(target=run_usb_test, args=(i, usb2_result_array, usb3_result_array))
threads.append(thread_usb)
thread_usb.start()

for thread in threads:

thread.join()

if len(usb3_result_array) != len(m) and len(usb2_result_array)!=len(m):

result.update({'USB 2 Result': 'Fail', 'USB 3 Result': 'Fail'})

else:

if "Fail" in usb3_result_array:

result.update({'USB 3 Result': 'Fail'})

else:

result.update({'USB 3 Result': 'Pass'})

if "Fail" in usb2_result_array:

result.update({'USB 2 Result': 'Fail'})

else:

result.update({'USB 2 Result': 'Pass'})

return result

if __name__ == "__main__":

while(True):

usb_result = None
time.sleep(2)
passmark_serial_1 = Passmark_Attach_1()
print(passmark_serial_1)
usb_result = Passmark_USB_Test(passmark_serial_1)
print(usb_result)

[helloeasyji]

Good Day,

Here we are using USB 3.0 loopback hardware in our manufacturing test environment. 4ea Passmark are connected to the downstream port of our product.
During current testing, we are using Python subprocess to test the USB 3.0 loopback for each Passmark loopback, so basically they are running loopback one by one.
In order to reduce the total test time, we try to modify the code to use threading to run the loopback on different loopback hardware simultaneously.

Here is our partial python code to call the different loopback hardware to run the USB3Console.exe, but looks like the result is quite unstable, sometimes the return code will indicate the loopback hardware can't be find. I would like to confirm if USB3Console.exe is friendly to multi-threading application?


---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
def Passmark_Attach_1():

p = subprocess.Popen(r"M:\\Result\\FunctionalTestScrip tGE\\USB3Console.exe -f", stdout = subprocess.PIPE)
output = p.communicate()
returncode = p.wait()
data = output[0].decode("utf-8").splitlines()
time.sleep(2)
serial_number_array_1 = []
for i in data:

find_device_data = i.split("=")

serial_number_array_1.append(find_device_data[1])

p.kill()
return serial_number_array_1

def run_usb_test(device, usb2_result_array, usb3_result_array):

p_usb2 = subprocess.Popen(r"M:\\Result\\FunctionalTestScrip tBB\\USB3Console.exe -d {} -s USB2 -t 2 -l 1".format(device), stdout=subprocess.PIPE)
returncode_usb2 = p_usb2.wait()
if returncode_usb2 == 0:

usb2_result_array.append("Pass")

else:

usb2_result_array.append("Fail")

p_usb3 = subprocess.Popen(r"M:\\Result\\FunctionalTestScrip tBB\\USB3Console.exe -d {} -s USB3 -t 2 -l 1".format(device), stdout=subprocess.PIPE)
returncode_usb3 = p_usb3.wait()

if returncode_usb3 == 0:

usb3_result_array.append("Pass")

else:

usb3_result_array.append("Fail")

def Passmark_USB_Test(m):

usb3_result_array = []
usb2_result_array = []
result = {}
if len(m) < 1:

result.update({'USB 2 Result': 'Fail', 'USB 3 Result': 'Fail'})

else:

threads = []
for i in m:

thread_usb = threading.Thread(target=run_usb_test, args=(i, usb2_result_array, usb3_result_array))
threads.append(thread_usb)
thread_usb.start()

for thread in threads:

thread.join()

if len(usb3_result_array) != len(m) and len(usb2_result_array)!=len(m):

result.update({'USB 2 Result': 'Fail', 'USB 3 Result': 'Fail'})

else:

if "Fail" in usb3_result_array:

result.update({'USB 3 Result': 'Fail'})

else:

result.update({'USB 3 Result': 'Pass'})

if "Fail" in usb2_result_array:

result.update({'USB 2 Result': 'Fail'})

else:

result.update({'USB 2 Result': 'Pass'})

return result

if __name__ == "__main__":

while(True):

usb_result = None
time.sleep(2)
passmark_serial_1 = Passmark_Attach_1()
print(passmark_serial_1)
usb_result = Passmark_USB_Test(passmark_serial_1)
print(usb_result)

Hi There,

i also noticed when we running 3 passmark HW loopback simultaneously, it's quite stable, but when we increase them to 4 HW loopback running at the same time, it's quite unstable.

[helloeasyji]

Hi There,

i also noticed when we running 3 passmark HW loopback simultaneously, it's quite stable, but when we increase them to 4 HW loopback running at the same time, it's quite unstable.

Hi There,

We confirm that while there is a USB 3.0 loopback ( #1st) running the loopback test, the enumeration of a new added USB 3.0 loopback HW (#2nd) sometimes will disturb the loopback testing of #1st one, which may result some low level error or even stopping loopback, please advise if there is any plan to fix that issue.

Thank you

[Dimiter (PassMark)]

Yes, the enumeration should be able to handle multiple threads.

As far as we're aware, how well that happens is a function of how well (from a threading point of view) the drivers handle the enumeration requests.

USB has some strict time limits for the handshake that takes place after a device is connected.

We ran your script (the parts that we could) and could not reproduce the fault on our test hardware.

If you can give us a working script that reproduces the fault we could run it on some more machines.

Also there is a fair chance the problem might be specific to your machine. e.g. bad host controller device driver. So we are going to need more system details if we were to look into it in more detail.

Finally. You don't need to write your own code. You can use BurnInTest for this type of USB port testing and save yourself a lot of time.

[helloeasyji]

Yes, the enumeration should be able to handle multiple threads.

As far as we're aware, how well that happens is a function of how well (from a threading point of view) the drivers handle the enumeration requests.

USB has some strict time limits for the handshake that takes place after a device is connected.

We ran your script (the parts that we could) and could not reproduce the fault on our test hardware.

If you can give us a working script that reproduces the fault we could run it on some more machines.

Also there is a fair chance the problem might be specific to your machine. e.g. bad host controller device driver. So we are going to need more system details if we were to look into it in more detail.

Finally. You don't need to write your own code. You can use BurnInTest for this type of USB port testing and save yourself a lot of time.

Hi Dimiter,

Thank you for sharing the update.
USB 3.0 loopback is partial section of the MFG testing, that's why we integrate the light console program into our script, instead of using the BurnIn Test software.
I can confirm using BurnIn Test, i never see the similar issue before.

I can demonstrate the issue using the script sent in previous post.
As our device has 4 DFP ports, now i group the detected 4 PassMark into two group, and run the loopback on 2 passmrk simultaneously, this can save us half of the test time. So far it looks works well.

I will try to see if different host exhibit the different behavior when running 4 loopback at the same time.

Thank you for your time to look into this.

[David (PassMark)]

As our device has 4 DFP ports

Maybe you uncovered a flaw in your device?

[helloeasyji]

Maybe you uncovered a flaw in your device?

Hi David,

Even we by-pass our device and use a self-power 4 ports USB hub to demonstrate the same testing, we still can see the same issue.