Single Thread Score rating

[proboszcz]

Running more tests means a longer run time, and our aim was to have a relatively quick benchmark (as opposed to others on the market that can take hours to run).
So we selected representative tests and short test periods. That decision was made around a decade ago. There is obviously a test time / accuracy trade off that we made. It wouldn't matter what we did, people are still going to come up with conspiracy theories as soon as they see a result they don't like, or doesn't favor the CPU they just bought.
To really understand CPU performance you need a degree in computer science and a in depth study of the domain. There are only a few people who truly understand x86 assembler, SIMD, NUMA, pipelining, variable alignment, caching, compilers, the windows kernel, branch prediction, microcode, etc.... (and to be clear, some of this stuff we also only half understand). So it is all to easy to explain a complex issue as just being a conspiracy. Plus is makes great click bait for the publishers.

Well, you do not have to remove workloads for single thread tests to speedup the execution. You can just decrease the amount of data to process leaving the number of workloads untouched, which will give much more consistent results between multithread and singlethread results than it is currently.
Also does the SingleThread tests include workloads that utilize AVX512? If yes then you should definitely add also the workloads utilizing IA SHA Extensions for fair comparisons.

As to the conspiracy theories - the problem with them is that in the past they came true (Intel was paying benchmarks' vendors and suppliers to favor only their processors). Moreover compilers often were written in such a way, they did not "see" additional instructions on processors on which the CPUID instruction did not return "GenuineIntel" phrase. Having that in mind you should not change the tests so they favor only one Vendor, because it starts to look like the "good" old times have returned...

[Xeinaemm]

PassMark really? Again are you playing with Intel? As a software developer when I saw this and results in BenchmarkDotNet I laughed... BenchmarkDotNet is for microbenchmarking of various topics and you telling me that in less than month Intel magically upgraded older CPUs?

[David (PassMark)]

As noted several times, all the old PerformanceTest V9 results have been archived on this V9 page. There is no need to post unreadable screen shots from the Wayback machine, like you were unearthing some hidden secret.

BenchmarkDotNet isn't a single thread benchmark. So pretty much has no relevance to single thread scores. It is also a benchmark primarily aimed at tracking the performance of .NET across difference releases (different compilers and versions of .NET). It isn't really a hardware benchmark. But if you are really a software developer I guess you already know this, and are just trolling.

We welcome an informed technical discussion, but from this point forward any posts that are trolling, baseless accusations, time wasting repetition or just plain dumb will be deleted without notice.

[proboszcz]

BenchmarkDotNet isn't a single thread benchmark. So pretty much has no relevance to single thread scores..... [/COLOR]

What? Then please explain - why when running the tests using that BenchmarkDotNet library the task manager is showing the full utilization of only one core?

[David (PassMark)]

.NET functions are opaque & out of the control of the benchmarker. Some functions might be single thread, some might not be, some might be threaded in certain .NET releases, but not others. Some might be hardware accelerated in some .NET releases, some might not be. You can't even be sure what the bottleneck will be. For some .NET functions it is likely limited by disk speed, or RAM speed and isn't even a CPU benchmark. To make a broad claim that benchmarkdotnet is the gold standard reference for measurement of single threaded CPU performance doesn't make any sense.

[proboszcz]

.NET functions are opaque & out of the control of the benchmarker. Some functions might be single thread, some might not be, some might be threaded in certain .NET releases, but not others. Some might be hardware accelerated in some .NET releases, some might not be. You can't even be sure what the bottleneck will be. For some .NET functions it is likely limited by disk speed, or RAM speed and isn't even a CPU benchmark. To make a broad claim that benchmarkdotnet is the gold standard reference for measurement of single threaded CPU performance doesn't make any sense.

Firstly - the same can be said about PassMark and the workloads that were chosen for tests.
Secondly - .NET is a foundation of millions real-world applications in the wild. So the behavior of it on certain CPUs is by far more better measurement than synthetic benchmarks like PassMark, where the Passmark is not a foundation of any real-world applications.

[David (PassMark)]

the same can be said about PassMark

We don't use .NET, we have full access to the source code, there are no disk or network dependencies, we known for a fact the code is single threaded & any changes to the code or compiler are fully under our control. So nothing at all like the same situation.

.NET is a foundation of millions real-world applications

It isn't.
.NET was only ever used for business applications and mostly for user interface work (business data entry forms). There are dozens of incompatible versions and it was replaced to UWP to some extent. Win32 and UWP (and a multitude of web based stuff) are what are now actually being used most of the time. But there is now talk of UWP being killed or morphed into something else as well. If you want to benchmark hardware it makes sense to get as close to the hardware as possible. Not build something upon layers of software you don't control.

[proboszcz]

We don't use .NET, we have full access to the source code, there are no disk or network dependencies, we known for a fact the code is single threaded & any changes to the code or compiler are fully under our control. So nothing at all like the same situation.

.NET is open source from many years and the source is available on GitHub. I am a .NET developer since more than 10 years and the tests in BenchmarkDotNet was fully single-threaded. So I don't understand your point here.

It isn't.
.NET was only ever used for business applications and mostly for user interface work (business data entry forms). There are dozens of incompatible versions and it was replaced to UWP to some extent. Win32 and UWP (and a multitude of web based stuff) are what are now actually being used most of the time. But there is now talk of UWP being killed or morphed into something else as well. If you want to benchmark hardware it makes sense to get as close to the hardware as possible. Not build something upon layers of software you don't control.

This is a completely false statement from your side. Most web application on the internet is based either on .NET or on JAVA. Why do you say such things?

[a7751218]

Secondly - .NET is a foundation of millions real-world applications in the wild.

That's not exactly true, and as you later stated, millions of web applications - yes, but not non-server applications.
I only know of 1 well-known desktop C#/.net application - Paint.NET.
Regardless, Javascript is more widely used than even .NET

Most applications, whether it's compression, photo editing, encoding, the web browser, office apps, or games, are mostly written using C/C++, including Windows itself.

---

I was also initially unhappy about the change, especially to single-threaded scores, but realise that since all other software has moved on, it's good that PassMark has also decided to use updated algorithms and a newer compiler (VS2019).

I'm still some-what sceptical for example of i7-7700k having better threaded performance compared to Ryzen 5 3600x, let alone the Ryzen 9 3900 and 3950x, even if only a few points.
But at the same time, I agree with David (PassMark) explanation of other workloads, such as photoshop, javascript, and excel, etc.

Regardless, I've learnt to live with new results.

[David (PassMark)]

Most web application on the internet is based either on .NET or on JAVA

Ridiculous claim, with zero evidence to back it up.
Most common web server on the internet is now Nginx. Second most common is Apache. Both these run on Linux. And no one runs a Linux server if they want .NET (a Microsoft API).
Proof, https://news.netcraft.com/archives/c...server-survey/
Microsoft only has a 14% market share. And even then many of those Microsoft web servers won't be running .NET.
If you are using Java server pages, then you should be running Tomcat, but that doesn't even rate a mention in the stats.
No one runs Java on the desktop anymore. That all died 10 years old. It is still a popular language, but it's use is now restricted to business apps.

Javascript & HTML5 & SQL is where it is all at on the server side. (plus a bit of PHP, Python, Ruby, C#, Perl, and twenty other languages).

I am a .NET developer since more than 10 years and the tests in BenchmarkDotNet was fully single-threaded.

Is it? You've read through all the source code to confirm this, and it doesn't create a single thread, and you've also checked all past and future releases of .NET to confirm that threading will never be used? Of course you can't, it isn't in your control.

[davide445]

Was to create a new post but I think this thread is here for this reason
Looking at the single core performance for a new customer PC I didn't find myself (as others I was reading) with the AMD Ryzen results.
Looking at the currently top two Ryzen processors, I was wondering how is possible a 3900 PRO 12 core 3.1Ghz/4.3Ghz turbo with 65W TDP is beating a 3800X 8 core 3.9Ghz/4.5GHz turbo with 105W TDP.
To me didn't make sense, I find the ranking totally changed with only Intel processors at the top (not a bad thing itself, just disorienting due to this weird results from AMD).

Strangely the results does have a logic for Intel, where the first three processors are ranked in a logic way with the 9900KS 4 Ghz and 127W ranked more tan 9900KF 3.6Ghz 95W that's over the less binned 9900K with same specs

[David (PassMark)]

Those two AMD CPUs are so close that they are below the margin for error.
But we agree the results for the Pro 3900 look at little high compared to its specs. I suspect it will drop a few percentage points once we get a few more results. Even a 1% drop would move it down a few places, all the CPUs are so close in single threaded performance.

[demonsavatar]

Those two AMD CPUs are so close that they are below the margin for error.
But we agree the results for the Pro 3900 look at little high compared to its specs. I suspect it will drop a few percentage points once we get a few more results. Even a 1% drop would move it down a few places, all the CPUs are so close in single threaded performance.

This makes sense, they really are all very close in scoring. What is the error margin on the scores?

Considering they are all clustered, does showing them in a sequential ranking make sense then? Maybe the list should be shown in groups/clusters of the error margin, or maybe the rows with scores within the error margin can be highlighted a different color? E.g. if you hover over an item in the list, all the processors above and below it within the error margin could also be highlighted?

[David (PassMark)]

From pure maths point of view margin of error is very high for Pro 3900. Probably over 15%. It's a super rare CPU.
But for the 3800X we have 170+ samples in PT10. So margin for error is more around 5%.
(but the maths isn't precise as results don't fit perfect bell curves & population sizes per CPU model aren't known).

[Leiwu]

What do u think about Ryzen 5 1600 and 1600x Does it make sense to overpay?