Relevance of CPU benchmark on cloud/virtualised systems

[Luc]

Hello,

I am trying to understand how the benchmark works exactly and whether it still has any relevance on virtualised systems and CPUs considering there is always a certain degree of time slicing involved, sometimes quite heavy (on busy systems).

I found this page here https://www.cpubenchmark.net/cpu_test_info.html, the tests seem extensive, but I wonder, won't the results vary wildy on a VM sharing its hypervisor with 10 other vs one sharing the hypervisor with 50 other?

Am I right in saying that based on the above scenarios, test results (as a reflection of performance) cannot be guaranteed?

[David (PassMark)]

Yes, it would be better if the (entire) machine was idle when the benchmark was run. If your aim is to measure hardware performance.

But you can still use it as a relative measure, to measure "available performance".
What performance are you really going to get on this shared machine? Maybe you need to run it several times at different periods of the day to get a feel for the range of results.

[Luc]

Hi David and thanks for the reply.

My interest is less about specific numbers and scores, but where I am getting at is whether we can use the benchmark results as a way of showcasing the performance of a VM, in absolute terms, regardless of the load on the hypervisor at any given time.

If the VM is all alone on a hypervisor, the benchmark score will be (significantly) higher than when it shares the hypervisor with 50 other VMs doing various random work loads. Correct?

The single VM might get a score of - say - 500 cpu mark, but when the hypervisor gets busier, it might get 100 etc.

This seems intuitively obvious, just seeking your confirmation that the benchmark does indeed work this way and is indeed susceptible to the cpu time slicing nature of virtualisation.

[David (PassMark)]

If the VM is all alone on a hypervisor, the benchmark score will be (significantly) higher than when it shares the hypervisor with 50 other VMs doing various random work loads.

Of course. All computers have limited hardware resources.

But it also depends on the setup of the VM. If the physical machine has 32 Cores available and you have 20 VMs running with 1 core each. Then you have 12 cores idle. So one would hope that benchmarking an additional VM (that was assigned just 1 core) would give around the same CPU result regardless of what the other 20 VMs are doing. Of course there might be other RAM and disk bottlenecks.

[natey2]

... whether we can use the benchmark results as a way of showcasing the performance of a VM, in absolute terms, regardless of the load on the hypervisor at any given time.

IMHO, no (if you are looking for VM performance measurement in absolute terms).

I use VMs (VMware and Hyper-V) and physical machines daily, and I have seen VMware VMs that gave a PT sequential read speed of around 400MB/s when first provisioned become 10MB/s at certain times when I use them.
I use the VMs quite a bit for software development, and you can feel the VM "slow down" at times, e.g. you right-mouse-click in Window Explorer and it takes quite a while for the popup to appear, or you type "dir" and the VM hesitates a little before showing you the directory listing. That can be disastrous for apps that depend on immediate response, have aggressive timeouts, etc.
That's when I run the PT benchmarks to see what is slowing it down, to give the quantitative (i.e. not subjective, like "it feels slow today") numbers/charts to the VMware admins, to show them that a low-end desktop PC would be more responsive than their VM running on a server-class machine :^)

You can be sure an OS running straight off the hardware would produce better results than throwing any Virtualization layer on top of the hardware.
Virtualization gives you the ability to "split" a machine into a number of sub-machines, but there's a cost to doing that.

I have also found Hyper-V Disk I/O to be a little faster that VMware Disk I/O for Windows VMs, but I digress.