Request for Certification / Compliance Documents for PassMark Tools

[Dogukan Celik]

Dear PassMark Support Team,

We are currently planning to integrate PassMark software and hardware tools into a testing system for a product that must comply with military standards. As part of our validation process, we require documentation or certificates that verify the accuracy and reliability of the tools used.

Could you please confirm whether the following are available for your products:

• Calibration certificates for USB or Serial loopback devices
• Compliance or validation documentation
• Any documentation proving test result accuracy traceable to national or international standards

If anyone from the PassMark team or community has experience with this or can point me in the right direction, I’d really appreciate it.

Thanks in advance!

[David (PassMark)]

No calibration is required or even possible for a serial loopback plug.

For USB devices, we have a number of different ones. For some no calibration is possible. For others calibration is possible at an additional cost.

There is also no national standards for data loopback plugs. But there is for functionality like voltage and current measurement however.

[Dogukan Celik]

Do you have any Compliance or verification documents? As an example, we are conducting HDMI interface tests within our system. According to the HDMI 1.4 specification, transmitting a 4K@30Hz signal with RGB 4:4:4 color sampling and 8-bit color depth requires approximately 8.91 Gbps of bandwidth. We would like to understand what technical criteria your software uses to determine a “Passed” result in such tests.

[David (PassMark)]

We don't have any devices for testing HDMI interfaces. We suggest just connecting a monitor and checking the monitor output.

[Dogukan Celik]

What I actually want to understand is which standards are taken into account when determining whether a test result is marked as "Pass" or "not Pass." How do you define and evaluate these test outcomes as passed or failed?

[David (PassMark)]

In the case of HDMI we would suggest setting the video card to it's maximum resolution & refresh rate and then and checking the monitor output.
The easy pass or Pass or Fail check for a monitor / HDMI output is via visual inspection of the image for corruption, correct colors, dead pixels, etc...

Of course of you wanted to get more sophisticated you could generate eye diagrams for HDMI and start doing measurements using an oscilloscope and similar high end gear. There are definitely standards for this. But we aren't selling oscilloscopes. This kind of testing makes sense during the design phase. It makes less sense during the bulk production validation QA stage.
Also you really need to be an electrical engineer to setup the tests and understand the results.

Example eye diagram.

[Dogukan Celik]

I am an Electrical and Electronics Engineer, and we are currently conducting tests on a device developed within our company. Since leading companies in the industry include test results obtained using PassMark software and hardware in their Certificates of Conformance (CoC), we have decided to test our product using the test systems we procured from you.

However, these tests need to comply with various standards. We plan to test all interfaces covered by PassMark, including HDMI, USB 2.0 and USB 3.0, RS232, CPU, and RAM.
Do you have any technical test result documentation that verifies compliance with these standards? If so, how can we access these technical documents?

[David (PassMark)]

Our software & hardware is targeted at load testing devices for production lines & fault finding.

Our products aren't targeted at testing full compliance with the standards documents. Typically this kind of testing is only done during the design phase of a new chipset. For example if you were designing a new USB4 host controller chip, then confirming full compliance with the USB4 standard would be important. But this standard is around 1000 pages of documentation. The USB power delivery aspect is another 1000 pages. And USB4 has to work with USB3 as well, so that's another 1000 pages. It would take many many months to confirm full compliance and very high end equipment is required (oscilloscopes, protocol analyzers, custom break out boards, etc). This is not practical for a production line environment, where typical test times are measured in minutes.

Addressing the specific interfaces you mention:
HDMI: See my comments above. We suggest using a monitor with MonitorTest software. We don't verify the low level electrical characteristics of the HDMI port. BUT it can be reasonable assumed that if you can see a clear image on a monitor at maximum resolution, then the port it working.

USB2/3: We have various hardware for USB testing. None of the hardware ensures full compliance with the specification. The specifications are too vast and the options & permutations too many. Instead it is better to assume your host controller chip was designed correctly and then verify the device actually works (i.e. sends and receives data at the correct speed without data corruption). If sending and receiving work without error, then you don't really need to check all the low level things like voltage ramping, slew, timings, protocol sequences, etc... Faults in this low level stuff will manifest themselves at the higher level as corrupt data, CRC errors, or the port being missing entirely from the system, etc...

RS232: We have a loop back device for RS232. Again it doesn't test full compliance with the standards. For example we don't test the RI pin, as ring indicator isn't really used anymore, but it is still in the standards. There is again the assumption that if data sending and receiving works without error at expected speeds, then the low level electrical requirements are more or less in spec.

CPU: The CPU really isn't an "interface". But we have software for testing the CPU. Again, the specification for a modern CPU are vast. 5200 pages for the x86 instruction set specification, and around another 2000 pages per Intel CPU family in the external design specifications. Even Intel themselves can't fully test their CPUs. So we don't attempt to test CPUs comply fully with their specifications (often the specifications are secret in any case).

RAM: Same story. The specifications are vast. We don't fully test against all the specifications.

I understand what you want. It would be nice to run a two minute test and say you fully comply with standard XXX. But these standards are way to complex for that.

So you either accept higher level testing, that we do, to gain a good level of confidence that the device works (and is thus very likely compliant). Or you embark on a project of real compliance testing that will likely take many months.