Single Thread Score rating

[touristguy87]

Dave, if I may say so, after reading 5 or so of these whinges I realized something that you guys have done by changing CPUmark etc so the scores are so different on different processors.
You've exposed the fact that old cpus are old!

It's a classic case of a "collector" suddenly finding out that his priceless treasure-trove of priceless relics is just a room full of junk.
You've really got two choices. Keep benchmarking using code that your team built 5 years ago on a 5 year old PC, with a 5 year old compiler, library files that are at least 5 years old and the settings used 5 years ago. And never, ever, change it because it's more holy than the Holy Grail.

Or update your benchmark and let the tears fall where they may.
I say that you come out with a new version every month and just flush the results that are over a month old.

That way you can optimize its utility and usefulness without having to worry about how different the scores are from earlier versions on earlier hardware.
And that way it is more likely to remain RELEVANT.

Unless you want to peg it to a K5 at 10 or something, as someone suggested earlier, laugably.
Like it's only good if a K5 system scores a 10 on it
Maybe run it by NIST to get the proper scores.

Or should we go back to the days when benchmarks would score lower on AMD chips than Intel simply because the compilers didn't recognize AMD chips as fully Intel compatible and so would revert to using standard x86 calls instead of SSE calls? Or should you try every compiler, compiler setting and compiler version out there until you get a Ryzen 5 to beat a Xeon gold?

You can't go around worrying about what will make these fanbois happy, that will just hobble your benchmark with a label of being biased towards Intel or AMD, for or against modern CPUS and so on. Pick a goal and code to it. then defend the goal and the coding.

cheers

[David (PassMark)]

What I would like see is real results and not regression scores

I'm not sure what you mean. What even is a regression score in the context of a CPU benchmark?
What's not 'real' about the results?

I know that a weakness of weighted scores is the weightings

If we thought weightings was a weakness we wouldn't be doing it. People like having a single number like the CPUMark, and to get a single number from many different measurements, all with different scales and units, then some type of weighting system isn't a weakness, it is a necessity.

The purpose of this type of analysis is to look at best fit charts

What chart? Fitted to what value? Without context, this doesn't mean anything.

As an IT person I want to see MOPs results

I assume MOPS is Millions of Operations Per Second?
This is exactly what the PerformanceTest already reports. Have you used the software?

[solesurvivor]

First post for me. I'm ex platform mangement for a telco and so used to have access to professional benchmark databases. What I would like see is real results and not regression scores. Having a background in stats and operations research, I know that a weakness of weighted scores is the weightings. The purpose of this type of analysis is to look at best fit charts. It's the complete set that is being analysed not individual results. Is the trend linear, geometric type of stuff. As an IT person I want to see MOPs results, mostly because it's the only way I can assess if your benchmark is useful, in other words - real. Is this possible? I feel very uneasy with scores derived from weightings.

[David (PassMark)]

Yes, it seems to be a big jump over the previous generation. But it is early days at the moment, as we only have one sample for the AMD Ryzen 5 5600X, so the numbers will change.

We are using Visual Studio 2019. So a modern compiler. We haven't changed compiler versions since the release of PerformanceTest. So the version has remained at Version 16.4.5. which came out in Feb 2020.

I had a quick look through the release notes for Visual Studio 2019, but didn't see anything obvious that was included especially for Ryzen 5000. So I would be pretty surprised if using the current 16.7.6 release of VS2019 would make any significant difference.

And generally speaking compilers don't like including multiple code paths. So as a developer we pick a target CPU instruction set and the compiler creates code that runs on that CPU and all later CPUs. It doesn't create 6 different code paths for the same piece of code and then picks the optimal path at run time based on the CPUs capabilities. Some libraries do this however. Here is an example of memcpy that has multiple code paths.

[Dariop]

I see the single thread performance ranking is now topped by a yet to be released Zen 3 based processor, and yes, its improvement is significant. I am wondering to what extent the used compiler and its optimizations are taking advantage of the architecture of this new processor. If the benchmark program is compiled with a standard set of options, how fair is the ranking in assessing the actual capability? Any performance trick not known (yet) by the compiler or which is just not enabled so the program can run on older architectures would of course underestimate the potential of the hardware.

[David (PassMark)]

Intel's i3-9100F had a 'monstrous' 8900 cpumark and some 2400 ST. Now it's 6900 and some 2500 ST.

As of today the i3-9100F single threaded results are,
V9 - 2406
V10 - 2486
~3% increase

Ryzen 5 1600
V9 - 1825
V10 - 2108
~15% increase.

So are you
1) Upset that AMD benefited more than Intel in this case (i.e. the opposite of some others in this topic)? It is a no win situation for us. Any move up or down upsets roughly half our user base.
2) Or just upset because you don't like change. Again it is a no win. As eventually the numbers will become irrelevant, unless we update it every decade or so. Leaving people upset about the change, or upset that we didn't change.

[vt8235]

... and you telling me that in less than month Intel magically upgraded older CPUs?

I was irritated to read this as I remembered the cpus I was interested in buying and the way they performed across this whole V9 V10 thing.
Intel's i3-9100F had a 'monstrous' 8900 cpumark and some 2400 ST. Now it's 6900 and some 2500 ST.
Meanwhile AMD's Ryzen 1600/2600 had some 1800 ST now they are 300 points higher.

[Leiwu]

What do u think about Ryzen 5 1600 and 1600x Does it make sense to overpay?

[David (PassMark)]

From pure maths point of view margin of error is very high for Pro 3900. Probably over 15%. It's a super rare CPU.
But for the 3800X we have 170+ samples in PT10. So margin for error is more around 5%.
(but the maths isn't precise as results don't fit perfect bell curves & population sizes per CPU model aren't known).

[demonsavatar]

Those two AMD CPUs are so close that they are below the margin for error.
But we agree the results for the Pro 3900 look at little high compared to its specs. I suspect it will drop a few percentage points once we get a few more results. Even a 1% drop would move it down a few places, all the CPUs are so close in single threaded performance.

This makes sense, they really are all very close in scoring. What is the error margin on the scores?

Considering they are all clustered, does showing them in a sequential ranking make sense then? Maybe the list should be shown in groups/clusters of the error margin, or maybe the rows with scores within the error margin can be highlighted a different color? E.g. if you hover over an item in the list, all the processors above and below it within the error margin could also be highlighted?

[David (PassMark)]

Those two AMD CPUs are so close that they are below the margin for error.
But we agree the results for the Pro 3900 look at little high compared to its specs. I suspect it will drop a few percentage points once we get a few more results. Even a 1% drop would move it down a few places, all the CPUs are so close in single threaded performance.

[davide445]

Was to create a new post but I think this thread is here for this reason
Looking at the single core performance for a new customer PC I didn't find myself (as others I was reading) with the AMD Ryzen results.
Looking at the currently top two Ryzen processors, I was wondering how is possible a 3900 PRO 12 core 3.1Ghz/4.3Ghz turbo with 65W TDP is beating a 3800X 8 core 3.9Ghz/4.5GHz turbo with 105W TDP.
To me didn't make sense, I find the ranking totally changed with only Intel processors at the top (not a bad thing itself, just disorienting due to this weird results from AMD).

Strangely the results does have a logic for Intel, where the first three processors are ranked in a logic way with the 9900KS 4 Ghz and 127W ranked more tan 9900KF 3.6Ghz 95W that's over the less binned 9900K with same specs

[David (PassMark)]

Most web application on the internet is based either on .NET or on JAVA

Ridiculous claim, with zero evidence to back it up.
Most common web server on the internet is now Nginx. Second most common is Apache. Both these run on Linux. And no one runs a Linux server if they want .NET (a Microsoft API).
Proof, https://news.netcraft.com/archives/c...server-survey/
Microsoft only has a 14% market share. And even then many of those Microsoft web servers won't be running .NET.
If you are using Java server pages, then you should be running Tomcat, but that doesn't even rate a mention in the stats.
No one runs Java on the desktop anymore. That all died 10 years old. It is still a popular language, but it's use is now restricted to business apps.

Javascript & HTML5 & SQL is where it is all at on the server side. (plus a bit of PHP, Python, Ruby, C#, Perl, and twenty other languages).

I am a .NET developer since more than 10 years and the tests in BenchmarkDotNet was fully single-threaded.

Is it? You've read through all the source code to confirm this, and it doesn't create a single thread, and you've also checked all past and future releases of .NET to confirm that threading will never be used? Of course you can't, it isn't in your control.

[a7751218]

Secondly - .NET is a foundation of millions real-world applications in the wild.

That's not exactly true, and as you later stated, millions of web applications - yes, but not non-server applications.
I only know of 1 well-known desktop C#/.net application - Paint.NET.
Regardless, Javascript is more widely used than even .NET

Most applications, whether it's compression, photo editing, encoding, the web browser, office apps, or games, are mostly written using C/C++, including Windows itself.

---

I was also initially unhappy about the change, especially to single-threaded scores, but realise that since all other software has moved on, it's good that PassMark has also decided to use updated algorithms and a newer compiler (VS2019).

I'm still some-what sceptical for example of i7-7700k having better threaded performance compared to Ryzen 5 3600x, let alone the Ryzen 9 3900 and 3950x, even if only a few points.
But at the same time, I agree with David (PassMark) explanation of other workloads, such as photoshop, javascript, and excel, etc.

Regardless, I've learnt to live with new results.

[proboszcz]

We don't use .NET, we have full access to the source code, there are no disk or network dependencies, we known for a fact the code is single threaded & any changes to the code or compiler are fully under our control. So nothing at all like the same situation.

.NET is open source from many years and the source is available on GitHub. I am a .NET developer since more than 10 years and the tests in BenchmarkDotNet was fully single-threaded. So I don't understand your point here.

It isn't.
.NET was only ever used for business applications and mostly for user interface work (business data entry forms). There are dozens of incompatible versions and it was replaced to UWP to some extent. Win32 and UWP (and a multitude of web based stuff) are what are now actually being used most of the time. But there is now talk of UWP being killed or morphed into something else as well. If you want to benchmark hardware it makes sense to get as close to the hardware as possible. Not build something upon layers of software you don't control.

This is a completely false statement from your side. Most web application on the internet is based either on .NET or on JAVA. Why do you say such things?