Disk test doesn't max NVME speeds.

[t0aster]

Why doesn't the PerformanceTest drive benchmark test nvme drives at top speed? Running the benchmark on 2 980pro's separately and combined in a raid0 array the test never goes over 4,343MB/s. The top speed on 1 - 980pro is over 7,000MB/s and 2 in a raid0 have a top speed over 11,000. I've noticed this since the 980pro came out and I've been meaning to post this question for a while.

[David (PassMark)]

I think you'll find the disks are 100% busy during the test run. Meaning they can't go any faster in your system under this test scenario. The vendors of these drives like to claim huge speeds, but in real life scenarios they almost never hit the claimed speeds.

Try running some of the advanced tests in PerformanceTest instead.

(if they aren't 100% busy then maybe there is some other bottleneck in your system. e.g. your RAID controller or the CPU, or the PCIe bus).

[t0aster]

No the disks are not 100% busy in my system during the tests. I will take screenshots to show what the top speed is and the disk utilization during passmark testing and show what I am talking about. I've been noticing this for a while and I assure you that the disk test isn't running my disks at full speed.

[David (PassMark)]

You should see something like this (screen shot below) for the standard disk read test.

Note the disk "Active time" is 100%.
This is for a Western Digital BLACK SN850 NVMe SSD (PCIe 4, 500GB)
CPU: Ryzen 5 5600X
RAM: 32GB DRR4

Western Digital claim this drive does 7000MB/sec reads, but this largely marketing bullshit. Here you can see the drive is 100% busy reading, but only doing 4.2GB/Sec.

But also note that there is some CPU load. At these high speeds it is relatively easy that a slow CPU or PCIe bus (especially if slow in single threading) can be a bottle neck. Also note that there is 106KB/sec in writing going on. This isn't from the benchmark, but is background activity from Windows (it is a system drive being benchmarked in this case). Too much of this and it can also have a negative effect on the benchmark.

I'll do a 2nd post with some details from the Advanced tests in PerformanceTest.

[David (PassMark)]

Below is the "Drive Performance" disk benchmark test under the Advanced Test menu in PerformanceTest.

It runs a bunch of disk test scenarios that are reasonably accepted in the industry as reasonable tests to run.

This is for the same WD drive as above (that WD's marketing people claim is a 7000MB/sec drive)

In all scenarios below it doesn't match the marketing claims (yet the drive is still at, or close to 100% busy, all the time, for all the tests). The results are still fantastically fast and it is a great SSD, but they don't match the marketing.

[David (PassMark)]

Next we can have a look at if we can devise some unlikely scenario that will get us numbers that are close to the WD's marketing numbers. For this we can use the "Advanced Disk Test" benchmark under the Advanced Test menu in PerformanceTest.

Scenario 1:
Here we pick a massive block size (8MB), relatively small test file (5GB), and non-random data (which is easy to compress). Not so realistic, but whatever, we don't care at this point.
Result: 5781MB/sec. So we are getting kind of closer.



Scenario 2:
This time we allow caching of the disk activity (into RAM)

Success!! We have matched and slightly exceeded the marketing claims of WD. 7432MB/sec read performance!!!

But wait ,what kind of shenanigans is this? Task manager reports 0% busy time and no disk activity. Conclusion: All the data is being read from RAM and not from disk. Clearly this is a bit of a poor disk test if the disk isn't even being used. What it does illustrate nicely however is the limit on the RAM cache speed. That even fast DDR4 RAM in dual channel can only just hit 7000MB/sec (with the I/O overhead of Disk API calls).




Scenario 3:
This time we use a massive block size (8MB), Asynchronous access (64 queue length), No caching and non-random data & 3 simultaneous threads. No developer would write code to access the disk this way, but whatever.... real life performance this isn't our goal for today.

This time we have done it for real. 100% load on the disk and 7GB/sec performance!!

Note that there is still some measurement differences depending on if you think a MB has 1,000,000 Bytes or the traditional 1,048,576 Bytes.
WD claim a MB has 1,000,000 Bytes, as then their numbers look better that way.
Also, task manager reports on all disk activity not just from the benchmark. So background activity is also counted.

[t0aster]

I made a video testing the array on my setup. It shows the hwinfo read and write speeds nvr reach their maximum with the passmark test. I've noticed this for a while using diff nvme drives. It appears that passmark doesn't give a reading much higher than the one in video no matter what drive combo I have tried. If that is real world sort of limit I would't know. The combination of 2 980pro's in raid0 does report much higher in hwinfo than 1 in at least a couple tests if used.
Passmark would give me test results that were much better when the 980pro first came out (5000+ read) but a few months ago I noticed the test wouldn't report higher than 3800 or so. I also want to clarify that I am referring to tests on new drives not the same drive months later, and I also have used a couple diff 5900x cpu's, 5600x cpus, Asus x570 mobos, MSI x570 mobos and Gigabyte x570 mobos.

[David (PassMark)]

Maybe you didn't read the posts above?

Disk performance is very very dependant on the test scenario. I understand you aren't happy with the performance of your expensive drive setup, but there isn't much point shooting the messenger. You really should be blaming the marketing department(s) of the drive vendors for not publishing numbers from realistic test scenarios.

[t0aster]

Did you watch the video I posted? To be clear, I am not disappointed nor shooting anyone, I am pointing out something. If you watch the top speed achieved with the passmark test and compare it to other disk tests, the passmark test nvr got the disks to perform at top speed. If you say the passmark test is reflecting the disks real world performance, what about copy operations between 2 nvme disks or nvme raid arrays? If I remember right, during those operations I've seen read-write rates that are in line with manufacturer claims. (I don't have a spare drive to test this right now, the 2 980's I have are currently set-up in raid0.) If disk copy operations run at those speeds wouldn't that count as a real-world situation, albeit one which may be uncommon at this time? In the near future, systems with multiple nvme drives should be much more common and copying files between system disks at 5000+ MB/s is a great performance increase.

I think the disk test could be improved is all I am saying. There are situations where the gen4 nmve drives will perform better than what is reflected by the diskmark test or at least I believe there are. I wanted to demonstrate what I observed and I did so by making a video. I noticed this months ago, made that video a couple weeks ago, and got around to posting it yesterday.

For the record, I wouldn't recommend the dual-nvme raid0 setup, it causes a few inconveniences, like the extra configuration work if bios is reset and taking away the ability to monitor temps with hwinfo. Since I had the chance I wanted to give it a try, I plan on going back to the normal setup for normal use. Before I do, I want to run a bunch of tests while I can, then compare.

[Martin5950]

My question is similiar. But I have no RAID and another NVMe: Corsair MP600 Pro XT. There are not so many results for that SSD, but median is about 45.000 in Disk Mark. I started with 33.000 and then found that I should disable CSM support und activate Secure Boot. I was already on UEFI boot. That brought the value to 38.386 which is about 20% less than the median for that drive.
Is there a way to get closer to the median with Windows 10? Is this solely due to fact that this is my C-drive and I did not switch off any particular background services.

[David (PassMark)]

Windows does create a lot of background disk activity. So it could be a Windows update running, it might be Defender doing a scan, SSD TRIM functionality, OneDrive syncing files, Windows Search function indexing files in background, or 20 other periodic background tasks. 3^rd party software does the same. E.g. Steam client downloading new game updates in the background.

Benchmarking a drive as a data drive, rather than a boot drive will certainly give more consistency.

Seems a bit strange that turning on secure boot should have a significant effect on the disk performance (unless it was also linked to some other features, like full disk encryption or data compression).

Could also be related to how full the drive is, what type of PCIe slot it is in (Gen3 or Gen4), if there was thermal throttling, etc..

[Martin5950]

Thank you David for your suggestions. The NVMe is a Gen4 on Gen4 slot. I always have an eye on task manager. So I know that the disk was on 0% activity, with only some KB/s read/write here and then. No encryption running, no STEAM downloads, LAN activity was also in the region of some KB/s.

Regarding Secure Boot I found an entry of a user who also figured his NVMe was way to slow. And the two things did the trick for him. So I just followed that. I have not yet checked whether both things are necessary for the speed improvement. However I can say that the Windows boot performance exploded with both changes, very likely due to the fact that I disabled CSM support. The performance is so good that there is little need for hybernate or energy saving mode in most of my uses. After BIOS is finished asking me for changes it takes 3-5 seconds for the password prompt to appear.
CrystalDiskMark gave the values that are advertised. But at that point the disk was still empty, because here it is drive letter F: instead of C:. I should repeat that test now.
But I was wondering about the PassMark DiskMarks from other users and there it looks less good for my setting.

[Martin5950]

"Could also be related to how full the drive is..." Probably I will reset the drive to empty and test again. It is 65% full, leaving 560 GB. The thermal throttling should be the same as for other buyers of this drive, so expect no difference here. It has native big metal coolers which I kept.

[David (PassMark)]

Can you try running the the "Drive Performance" disk benchmark test under the Advanced Test menu in PerformanceTest. Then post the numbers.

[Martin5950]

Here are the Drive Performance results. Disk temperature was always 39°C.

The Throughput Write has one drop to 166 MB/s at 28 sec, which was captured probably due to the lowered sample rate. All other data points were above 3.400 MB/sec