2D Benchmark Error - Resolution and Scaling Impact on 2D Benchmark Score

[SnakeGTX]

After using PerformanceTest to benchmark my new laptop, I noticed the 2D mark was far below average. I messed around with the resolution and scaling options in windows, and fixed it; my 2D benchmark went from 380 to 600, raising my overall Passmark score dramatically. It turns out, Window 10's scaling dramatically effects the 2D benchmark. The resolution also effects the 2D score as well. I spent the last 2 hours running tests of different scaling percentages and resolutions, and attached is a screenshot of the spreadsheet of my results.

So how is this an error? Well, at 1080p, Windows by default sets the scaling to 125%. If you're on 4k (like me), the scaling default is set at 250%. I expect for this to impact the performance of my graphics card, but PerformanceTest should take this into account, or the 2D benchmarks are going to be vastly inaccurate of the real performance of the graphics card. Just as the 3D benchmark takes into account resolution (and prevents Window's scaling by running everything full-screen at a set resolution), the 2D score should also take the resolution and scaling into account. I did test the 3D benchmark, and the DirectCompute test (only test not run in full screen) is also affected by Window's scaling and resolution.

[David (PassMark)]

By asking Windows to scale everything to 200%, you are in effect asking for the of display 4 times the number of pixels. So it is no surprise it is slower.

However the resolution in itself shouldn't have much impact (if the scaling / DPI settings are the same). So that difference is slightly surprising. Would be interesting to see it is was a consistent effect across all video cards, all versions of Windows & various device driver versions. Doesn't seem to effect the lower resolutions which is a bit strange as well.

As to if we should penalise users running 100% scaling (or augment the scores for users running 200% scaling), there are arguments on both sides.
2D really is slower with high DPI monitors and surely people need to have benchmark tools that measure this. Also the comparing of video cards for 2D isn't so important because the video card hardware doesn't impact on the 2D performance very much. (i.e. high end video card don't really do 2D any faster than low end). The CPU plays an important role however. So the whole concept of comparing video cards on their 2D performance (in modern versions of Windows) doesn't make much sense.

What we should be doing however is exposing the current scaling setting in the system information.

[IFeelYou]

To be true, PerformanceTest 2D scores are totally inacurate and almost obsolete. Sorry, but that's a fact. Just by lowering resolution (which can be done by anyone) to let's say 480p from 1440p I can easily increase 2D score greatly (by up to 30%). Just by taking out all Windows effects (Advances settings > Performance) I can go up to 20% more. And there are another tweaks, which I will not mention here. So all this is making this test total obsolete and inconsistent. That's why PerformanceTest will not be taken seriously in HWbot.org or similar websites till it fixes this kind of problems. It's just not serious. If there is a way to alter benchmark so easily - it's bad. And that's a fact. It is never happening in 3DMark or Unigine tests.
I mentioned only 2D, but there is plenty inconsistency in almost all other tests. Especially in Disk tests, when I can get score like 13000, and another time 22000+ more

I hope admins will fix this in near future, and take this problems more seriously, cause I like this benchmark very much and I think it can have a bright future.

[David (PassMark)]

I think you are confusing the serious process of benchmarking with a competition. There is no prize for getting the highest score.

In comparing our 2D tests to 3DMark, you are further assuming that 2D performance purely depends on the video card (and should scale with the 3D performance). But this isn't correct. 2D performance depends on a lot of factors.

In our opinion it is the function of our benchmark to measure the performance of your PC. If you can change the settings of your PC to get better performance, then the benchmark should reflect this. So if turning off the Windows's desktop visual effects results in the system getting faster, then the benchmark should reflect this. If you overclock your CPU & GPU, and the system is faster as a result, then the benchmark should reflect this. To say that a 2D benchmark should return the same result regardless of system settings renders the benchmark useless.

Same goes for the disk test, If there is a difference in the score, then there is a difference in performance. There are many factors that can cause variable performance in hard drives. For example, caching effects, background load, garbage collection, disk errors / retries, fragmentation, bad drivers, overheating, using the wrong PCIe slots, etc...

[David (PassMark)]

Just to add this this. Starting from V9.0 Build 1023 - 8th/Jan/2018, we are collecting and reporting on the DPI setting in the system information window (with the video card resolution). This will help somewhat in making apples to apples comparisons.

[IFeelYou]

I think you are confusing the serious process of benchmarking with a competition. There is no prize for getting the highest score.

In comparing our 2D tests to 3DMark, you are further assuming that 2D performance purely depends on the video card (and should scale with the 3D performance). But this isn't correct. 2D performance depends on a lot of factors.

In our opinion it is the function of our benchmark to measure the performance of your PC. If you can change the settings of your PC to get better performance, then the benchmark should reflect this. So if turning off the Windows's desktop visual effects results in the system getting faster, then the benchmark should reflect this. If you overclock your CPU & GPU, and the system is faster as a result, then the benchmark should reflect this. To say that a 2D benchmark should return the same result regardless of system settings renders the benchmark useless.

Same goes for the disk test, If there is a difference in the score, then there is a difference in performance. There are many factors that can cause variable performance in hard drives. For example, caching effects, background load, garbage collection, disk errors / retries, fragmentation, bad drivers, overheating, using the wrong PCIe slots, etc...

Thanks. Very good answer. Now I understand that Your benchmark mainly focuses on Real performance, not just synthetic performance, which is fair. But probably 480p is quite obsolete now for most of the users. 2DMark score could have a penalty for using resolutions like 480p, cause we have penalty on 3D for not using 4k, so...

[David (PassMark)]

Yes, we should probably not run the tests below the recommended resolution, 1280 x 1024. But we figured there wasn't much harm. People are just fooling themselves for the most part.