Threaded memory test failed. Error Code 4.

[kquan94401]

Hi folks,

I am a newbie to the Passmark benchmark and using it to help benchmark VM performance under Vmware ESX 5.0. I am hitting this "Threaded memory test failed" error and wondering if anyone has more details on how this particular test works.

My environment is the following:
ESX 5.0 with 10.3GB memory
Windows 7 VM with 2GB memory

I have 5 identical Windows 7 VMs. When I run the memory test on 4 VMs, no error.
However, with 5 VMs, I usually get the threaded memory test fail error on 2-3 VMs.

Thanks,
Kenny

[David (PassMark)]

Looks like you are running out of RAM.
256MB of free RAM is required to run the RAM tests.So 2GB of total RAM should be enough if you don't have much else running. We have had it running on machines with only 0.5GB.

[kquan94401]

Looks like you are running out of RAM.256MB of free RAM is required to run the RAM tests.So 2GB of total RAM should be enough if you don't have much else running. We have had it running on machines with only 0.5GB.

Hi David,

I have a total of 10.3GB RAM for the 5 VMs (2GB each). The catch is that I'm running all 5 VMs simultaneously. With 4 VMs running, the memory test always completes. With 5 VMs, usually 2/5 failed on the Threaded Memory test. I understand that ESX has some sort of dynamic allocation of memory so even though the VM is requested 2GB, all the memory isn't provided until the application/OS really uses it.

http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf

As an experiment, I tried reducing the memory for each VM to 1.5G each (total of 7.5GB) and still saw the error. So if there is plenty of memory, why is the Threaded Memory test still failing? Is there a clue with the Error Code 4? Am I hitting some memory boundary that is shared by two VMs?

Regards,
Kenny

[David (PassMark)]

At a high level here is how the Threaded memory test works.

1) Multiple test threads are launched.
The number of threads is equal to the value in the Edit / Preferences window. But it will match the number of CPU cores available in nearly all cases. There is a maximum of 64 threads. (This will be a max of 64 threads per VM in your case).

2) Each thread allocates a large-ish memory buffer. Then sets a 'ready to go' flag. Then waits.
The size of the memory buffer is 256MB *in total* across all the threads. So the size per thread is 256MB / Number_of_threads.

3) The main thread waits for all the test threads to be ready.
If after 15 seconds, all the threads are not ready you get the Error Code 4.

4) If all threads are ready, the benchmark test starts in all threads

5) When the test stops the results are collected.
If after 15 seconds past the stop time, all the threads have fail to stop, you get the Error Code 5.

So what we can be sure of is that either
A) One or more of the threads failed to allocate a memory buffer.
OR
B) One or more of the threads was super slow. As the allocation of a memory buffer should only take a few millseconds. (Not 15 sec). Being super slow would normally mean the system is in the process of swapping RAM pages out to disk, as a result of a low RAM situation.

In either case A) or B) the cause is a lack of RAM.

Also remember there is 2 ways to run out of RAM and get a memory allocation failure. You can run out of physical RAM, but you can also run out of address space (this is a bigger and bigger problems on 32bit operating system). I believe in your VMs you can set the virtual and physical memory settings separately?


I agree that it should run in the setup described. But clearly it doesn't. But I don't think the fault is in our software. VMWare is doing something wrong or the VMWare config is wrong.

[kquan94401]

At a high level here is how the Threaded memory test works.

1) Multiple test threads are launched.
The number of threads is equal to the value in the Edit / Preferences window. But it will match the number of CPU cores available in nearly all cases. There is a maximum of 64 threads. (This will be a max of 64 threads per VM in your case).

2) Each thread allocates a large-ish memory buffer. Then sets a 'ready to go' flag. Then waits.
The size of the memory buffer is 256MB *in total* across all the threads. So the size per thread is 256MB / Number_of_threads.

3) The main thread waits for all the test threads to be ready.
If after 15 seconds, all the threads are not ready you get the Error Code 4.

4) If all threads are ready, the benchmark test starts in all threads

5) When the test stops the results are collected.
If after 15 seconds past the stop time, all the threads have fail to stop, you get the Error Code 5.

So what we can be sure of is that either
A) One or more of the threads failed to allocate a memory buffer.
OR
B) One or more of the threads was super slow. As the allocation of a memory buffer should only take a few millseconds. (Not 15 sec). Being super slow would normally mean the system is in the process of swapping RAM pages out to disk, as a result of a low RAM situation.

In either case A) or B) the cause is a lack of RAM.

Also remember there is 2 ways to run out of RAM and get a memory allocation failure. You can run out of physical RAM, but you can also run out of address space (this is a bigger and bigger problems on 32bit operating system). I believe in your VMs you can set the virtual and physical memory settings separately?


I agree that it should run in the setup described. But clearly it doesn't. But I don't think the fault is in our software. VMWare is doing something wrong or the VMWare config is wrong.

Hi David,

Thanks for the details on the Memory Thread test. I did a little more investigation and it's looks like I'm running into out of RAM issues. Even though the system has 10GB of memory, ESXi reserves 2GB for itself leaving 8GB of usable memory. With my 5 VMs (2GB), I would require about 10GB memory. ESXi dynamically allocates memory to VMs on an as-needed basis but I'm assuming that the Memory Thread test is trying to use all or most of the 2GB in the VM. I have not seen any other issues running the CPU, 2D Graphics or Disk tests.

Kenny