PerformanceTest 9.0 score variations between boards with the same spec

[harry.cheng]

Hi,

We ran PerformanceTest 9.0 on 3 boards of the same model (same CPU, BIOS settings and peripheral devices). However, there are both inter-board and intra-board (3 trials per board) variations of the scores, especially the CPU Mark, usually the 1st trial being the lowest. How large is the typical deviation for the scores from PerformanceTest 9.0?

Thank you!

	Board #1			Board #2			Board #3
H/W Version	ZR0	ZR0	ZR0	ZR0	ZR0	ZR0	ZR0	ZR0	ZR0
BIOS Version	v0.0.3	v0.0.3	v0.0.3	v0.0.3	v0.0.3	v0.0.3	v0.0.3	v0.0.3	v0.0.3
EC Version	90726T01	90726T01	90726T01	90726T01	90726T01	90726T01	90726T01	90726T01	90726T01
CPU	i7-8665UE	i7-8665UE	i7-8665UE	i7-8665UE	i7-8665UE	i7-8665UE	i7-8665UE	i7-8665UE	i7-8665UE
Memory	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)	J3 8G (WARIS2133)
Clockspeed	1.7 GHz	1.7 GHz	1.7 GHz	1.7 GHz	1.7 GHz	1.7 GHz	1.7 GHz	1.7 GHz	1.7 GHz
Turbo Speed	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz	Up to 4.4 GHz
Core	4	4	4	4	4	4	4	4	4
Storage	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB	Waris SSD 128GB
OS	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits	Win 10 64 bits
PassMark Rating	3313	3338	3404	2955	3033	3006	3327	3380	3377
CPU Mark	8438	8480	8438	7946	8365	8347	8254	8885	8818
2D Graphics Mark	811	841	851	705	720	722	834	843	846
3D Graphics Mark	1119	1125	1128	951	948	953	1122	1120	1126
Memory Mark	2053	2078	2103	1841	1863	1859	2081	2089	2065
Disk Mark	3127	2988	3145	2954	3201	2943	3098	3058	3068

Best,
Harry

[David (PassMark)]

Board #2 looks like it has slower RAM in it.

Variations from one run to the next depend on what background tasks are running (often randomly) on the machine. e.g. having a windows update or Anti-virus update occur mid test might result in a large variation. Thermal throttling can also impact the results. As can if the machine is running on battery power (or has a low battery, depending on Windows power settings).

On a modern Windows machine there is always a massive amount of background activity taking place. Most of is random when it occurs. This impacts the consistency of benchmarking.

If you want to take a look download Microsoft's Process Monitor tool
https://docs.microsoft.com/en-us/sys...nloads/procmon

And also have a look at the background Network activity with Wireshark.
https://www.wireshark.org/

You'll be amazed what is going on without your knowledge.

[harry.cheng]

Hi David,

The same RAM was used on all 3 boards as well as the heat sink. No battery was used, only AC source.The internet was not connected so no windows update was available. So i don't think these would be the source of variation.

As for Windows background activities, is there any way of avoiding the possible variations from the background activities when using the PerformanceTest? Or is there a Linux-based PerformanceTest tool available?

In addition, according to your experience, what is the approximate variation of the performance score? We just want to make sure that the variations we got are within the normal range.

Thanks!

Best,
Harry

[David (PassMark)]

Still looks like there was a difference in the RAM's performance to me. Maybe BIOS timing setting or the RAM not being in dual channel mode, or something else. Looking at the individual results might tell you more.

The way to avoid some of the variability is to,
A) Run longer tests AND/OR
B) Run multiple test runs and take the max value.
This can be done in an automated fashion from the Preferences window in PerformanceTest. e.g. do 10 runs on each platform.

If you have a lot of time, you can also go through all the services and startup apps running in the background in Windows and disable a bunch of them. But it is hard to figure out which ones are essential and which are safe to disable.

Linux also has a fair amount of background activity. It is just the way modern operating systems work.

[harry.cheng]

Hi David,

We have double-checked that the testing environment (including RAM and BIOS settings) was all the same except for different boards of the same model. Currently, we are using the PerformanceTest as a part of the testing procedure to demonstrate our capabilities to our customers. Therefore, is there a final suggestion for how large would the variation (in the same board for example) typically be?

Thanks!

Best,
Harry

[David (PassMark)]

It depends on the particular test and the environment. There are valid factors (some mentioned above) that can cause a very large variation in scores.
Disk systems (especially traditional HDD) can have large variations as performance depends on the placement of the files on the disk and fragmentation states. Which can vary from 1 minute to the next. Memory systems can also vary, NUMA & ECC error correction in particular can cause large differences on higher end systems as can the number of channels in use. Windows display settings and DPI settings can effect 2D performance. Drivers & SLI options 3D. The list is nearly endless.

But you can collect the real life variance yourself from within the PerformanceTest software.

Here for example is the real life distribution for the popular i7-8700K CPU in the CPUMark result (as of 23/Aug/2019).



Here for example is the distribution for the popular Samsung 970 EVO SSD in the sequential read result.



If you ran 10 tests and take the max result, in a very tightly controlled environment, then I would expect a variation of less that 3% in pretty much all of the tests.

[harry.cheng]

Hi David,

Got it. So according to the CPU Mark figure you provided, it seems that even with the same CPU but different board configurations (such as more supported IO ports could result in decreased CPU performance, etc.), there would be a large variation in the test scores right?

[David (PassMark)]

Yes, there can be a large variation between machines with the same CPU, for all the reasons stated above and more. e.g. Overclocking was not mentioned yet.

There is another list of factors here that can effect performance.
https://www.passmark.com/forum/perfo...-for-a-slow-pc

[harry.cheng]

OK, thanks a lot for the quick response!