USB3.0 Loopback Plug error reported on Intel Braswell Platform Cherry Hill CRB

[MiaB]

In case anyone else has the same problem.

We have received three separate reports of an intermittent USB error on Intel's IOTG Braswell Platform Cherry Hill Customer Reference Board using the USB3.0 loopback plug. The specific error is reported as:


"Benchmark read/write failed (31)
A device attached to the system is not functioning.
Overlapped I/O operation is in progress.
USB device driver error reported: 0xc0000001"


We have observed that the behaviour changes with different cabling where the USB3.0 controller exhibits marginal behaviour with some cables resulting in disconnection errors. We believe it is an electrical fault on the board. The fault has been reported to Intel by OEMs.

[Tim (PassMark)]

We have also seen this error occur in BurnInTest with the message "USB driver reported send error".

[David (PassMark)]

National Instruments did a protocol trace for us on a similar Intel board and it looks like it is a combination of two bugs.

The Intel USB3 host controller is failing to send Isochronous Timestamp Packets (ITP) on a regular basis. The ITP contains a timestamp value based on the Bus Interval Counter and it's value varies from 0 to 0x3FFF before it rolls over, back to zero.

The ITP packet should be send from the host each bus interval boundary to all links that aren't in a low power state. In our understanding each bus interval boundary is 125us, but the Intel host controller failed to send anything for 5.512ms.
5ms is an eternity when you are expecting a transmission every 125us. So it seems something is wrong here.

When the host controller does start transmitting again the interval count skips forward (e.g to a value of 902 instead of 858.

This unexpected behaviour confuses the USB devices, it could be argued that that our plug should eventually recover from this bad behaviour, but the problem is really in the underlying silicon chip (a Cypress FX3 in this case).

In summary there seems to be a compatibility bug between some Intel system on a chip (SoC) USB3 host controllers and at least the Cypress FX3 device chip.

It wouldn't be too surprising if the problem also effected other devices but we haven't done extensive testing on 3rd party devices.

There is also some evidence that the problem might also be linked to load. The problem seems more frequent when the host controller is under high load.

We will be reporting the problem to Intel.

[Jordan]

I also meet this problem using Braswell Platform,
I have report this issue to Intel.
If you got some feedback or solution from Intel, could you share the information to me?


Fri Nov 20 09:01:19 2015: Benchmark - (1) Write: 2186.8 Mb/s (273.3 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Write: 2157.2 Mb/s (269.7 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Read: 1866.5 Mb/s (233.3 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Write: 2152.2 Mb/s (269.0 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Read: 1874.6 Mb/s (234.3 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Write: 2117.5 Mb/s (264.7 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Read: 1708.3 Mb/s (213.5 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Write: 2208.2 Mb/s (276.0 MB/s)
Fri Nov 20 09:01:19 2015: Benchmark - (1) Read: 1861.4 Mb/s (232.7 MB/s)
Fri Nov 20 09:01:19 2015: ERROR - Serial PMQONCPY: Test start time: Fri Nov 20 09:01:18 2015
Fri Nov 20 09:01:19 2015: ERROR - Test Stopped due to Error: Benchmark write failed (31)
Fri Nov 20 09:01:20 2015: ERROR - A device attached to the system is not functioning.
Fri Nov 20 09:01:20 2015: ERROR - Overlapped I/O operation is in progress.
Fri Nov 20 09:01:20 2015: ERROR - USB device driver error reported: 0xc0000001
Fri Nov 20 09:01:20 2015: ERROR - {Operation Failed} The requested operation was unsuccessful.
Fri Nov 20 09:01:20 2015: ERROR - USB request status reported: 0xc0000011
Fri Nov 20 09:01:20 2015: ERROR - Unknown.
Fri Nov 20 09:01:20 2015: ERROR - Endpt UsbdStatus: [state=STALLED status=UNKNOWN]
Fri Nov 20 09:01:20 2015: Benchmark - (1) Write: 0.0 Mb/s (0.0 MB/s)
Fri Nov 20 09:01:20 2015: Information- PMQONCPY (SuperSpeed) added to list
Fri Nov 20 09:01:21 2015: Benchmark - Errors 0, 0, 1

[Taiwan]

Hi, Passmark

intel response as below information to us:
=============================

Hi,

We have passed on SI and margin test. It means board design (physical layer) is good, no problem.
If you have problems on real devices that you will sell out, we will help you to debug.
But for this 3rd party tool, we have done so much. I think you must contact Passmark on this issue and ask them how to do.
BTW, we have gotten USB-IF TID certification for Braswell. You can refer to below website.

http://www.usb.org/kcompliance/view/...120e7f0815b9c2 &referring_url=/kcompliance


==========================================
Do you have any suggestion?

Thanks
Taiwan

[David (PassMark)]

Taiwan,

Can you provide more details of the hardware and test setup that you are using.

There was a firmware update in Dec 2015 for the USB3 plugs, to V1.6
http://www.passmark.com/products/usb3updater.htm

This seemed to improve the behaviour slightly, but not totally resolve the problem.
So you might want to try out this FW and report back.

As noted above the problem doesn't appear to be a physical later problem. So saying the board passed physical layer testing is not really relevant.

[David (PassMark)]

We had a similar report from a similar system today (CPU N2840, Silvermont - Bay Trail-M).

ERROR - Test Stopped due to Error: Write failed (995)
ERROR - The I/O operation has been aborted because of either a thread exit or an application request.
ERROR - Overlapped I/O operation is in progress.
ERROR - USB device driver error reported: 0xc0000120
ERROR - The I/O request was canceled.
ERROR - USB request status reported: 0xc0010000
ERROR - The USB stack reports this error whenever it completed a transfer because of an AbortPipe request from the client driver.
ERROR - Endpt UsbdStatus: [state=STALLED status=UNKNOWN]
ERROR - RESULTS:
ERROR - Test Start time: Wed Feb 10 12:35:42 2016
ERROR - Duration: 008h 19m 46s
ERROR - 2568747 operations. 1 Write Error. 0 verification errors.

This one failure only occurred after 8 hours of testing however, so it isn't clear if this is the same problem occurring more rarely, or some other one off problem. More investigation is required.

[yzstd88]

Hi Passmark

I also see the same problem on Intel Broadwell-DE CPU (Grangeville platform) . And I also find that I can't update the firmware of USB3.0 loopback plug to 2.0 . It always shows fatal error . but rev1.7 firmware can update

The issue on our platform can be replicated randomly . It seems it will be easier to be replicated when system is in higher loading (add more test item in Burn-in test)

Do you have any update on this issue?

Thanks

[HamidK (PassMark)]

Hi,

About fatal error that happened with firmware V2.0, can you let us know:
When this fatal error occurs? During Benchmark test or Loopback test? What are the test configurations?
What are the error codes returned by fatal error message? (In FW Ver 2.0 LCD shows two error codes when fatal error occurs)

Can you also make sure that BIOS and Host controller drivers are up-to-date, and the latest driver for USB Loopback plug (Ver 1.2.3) is used.

Hi Passmark

I also see the same problem on Intel Broadwell-DE CPU (Grangeville platform) . And I also find that I can't update the firmware of USB3.0 loopback plug to 2.0 . It always shows fatal error . but rev1.7 firmware can update

The issue on our platform can be replicated randomly . It seems it will be easier to be replicated when system is in higher loading (add more test item in Burn-in test)

Do you have any update on this issue?

Thanks

[yzstd88]

Hi,

About fatal error that happened with firmware V2.0, can you let us know:
When this fatal error occurs? During Benchmark test or Loopback test? What are the test configurations?
What are the error codes returned by fatal error message? (In FW Ver 2.0 LCD shows two error codes when fatal error occurs)

Can you also make sure that BIOS and Host controller drivers are up-to-date, and the latest driver for USB Loopback plug (Ver 1.2.3) is used.

Hi

When we use USB3.0 loopback plug and use BurnIn test V8.1 pro to do the test . LCD will show the following message "Fatal error update software and reconnect the plug "

It only occurs when we update firmware 2.0 . Firmware 1.7 can run correctly

Our USB driver is Intel_USB_3.0_xHC_Driver_Skylake_MR5_PV_4.0.5.55

By the way, about the USB driver report send error issuel Is there any update on this issue? Does Intel have any response on this issue?

Thanks

[yzstd88]

Hi,

About fatal error that happened with firmware V2.0, can you let us know:
When this fatal error occurs? During Benchmark test or Loopback test? What are the test configurations?
What are the error codes returned by fatal error message? (In FW Ver 2.0 LCD shows two error codes when fatal error occurs)

Can you also make sure that BIOS and Host controller drivers are up-to-date, and the latest driver for USB Loopback plug (Ver 1.2.3) is used.

The issue occurs on loopback test in BurninTest V8.1 pro. We run CPU/RAM/network/2D graphic/ video playback/COM port/disk/USB with 100% loading

We have installed l.2.3 loopbackd driver and latest Intel USB3.0 host driver 4.0.5.55

[Tim (PassMark)]

Are you using the latest build of BurnInTest (Version 8.1 build 1016)?

The USB3 firmware version 2 cannot be used with older versions of BurnInTest.

[yzstd88]

Got it . originally we use burn-in test V8.1 (build 1014) instead of build 1016 . So firmware 2.0 will show fatal error. Now it works on build 1016 .We are testing again to check if firmware 2.0 improves the issue "USB driver reported send error"

The test is completed with firmware 2.0 burn-in test V8.1(build 1016). But we still see "USB driver reported send error " message . But now it shows in "black" instead of "RED" warning .
Does it mean that passmark ignore this message?

Please confirm

Thanks

[Integrator]

Hi,
we see the same Problem.

"USB driver reported send error " message and BIT it shows it in "black" instead of "RED".
But in the Tracelog file the Error is marked in "red".

BIT V8.1 Build 1016
Loopbackplug FW V2.0 (with newest driver)
USB 3.0 Driver:4.0.5.55
Skylake platform
OS: Win7 64Bit Ultimate SP1