BurnInTest failed when verify read file

[mars.wen]

Hi,

We run reboot test with PassMark BurnInTest V9.2 1002 (64-bit) on Windows 10.
Recently we got test result fail, and it showed "insufficient system resources..." messages in log file
Do you have any idea to avoid this error?

BurnInTest log:
******************
DETAILED EVENT LOG
******************
LOG NOTE: 2022-03-27 09:03:49, Status, PassMark BurnInTest V9.2 1002 (64-bit)
LOG NOTE: 2022-03-27 09:03:51, Status, Main Tests started
SERIOUS: 2022-03-27 09:04:23, Disk, L: Insufficient system resources to complete test
SERIOUS: 2022-03-27 09:04:23, Disk, J: Insufficient system resources to complete test
SERIOUS: 2022-03-27 09:04:23, Disk, I: Insufficient system resources to complete test
SERIOUS: 2022-03-27 09:04:23, Disk, K: Insufficient system resources to complete test
LOG NOTE: 2022-03-27 09:04:23, Disk, Verify Read File: K:\BurnInTest test files\~bittestK00003, Block Size: 1048576, Block Num: 516, Ops: 5501878272
LOG NOTE: 2022-03-27 09:04:23, Disk, Verify Read File: J:\BurnInTest test files\~bittestJ00003, Block Size: 1048576, Block Num: 517, Ops: 5502926848
LOG NOTE: 2022-03-27 09:04:23, Disk, Verify Read File: I:\BurnInTest test files\~bittestI00003, Block Size: 1048576, Block Num: 512, Ops: 5768216576
LOG NOTE: 2022-03-27 09:04:23, Disk, Verify Read File: L:\BurnInTest test files\~bittestL00003, Block Size: 1048576, Block Num: 473, Ops: 5501878272
LOG NOTE: 2022-03-27 09:04:23, Disk, 配額不足，無法完成要求的服務。 (1453)
LOG NOTE: 2022-03-27 09:04:23, Disk, 配額不足，無法完成要求的服務。 (1453)
LOG NOTE: 2022-03-27 09:04:23, Disk, 配額不足，無法完成要求的服務。 (1453)
LOG NOTE: 2022-03-27 09:04:23, Disk, 配額不足，無法完成要求的服務。 (1453)
LOG NOTE: 2022-03-27 09:04:23, Disk, Disk Error (J Insufficient system resources to complete test in Cycle 2 with pattern Sequential data pattern
LOG NOTE: 2022-03-27 09:04:23, Disk, Disk Error (K Insufficient system resources to complete test in Cycle 2 with pattern Sequential data pattern
LOG NOTE: 2022-03-27 09:04:23, Disk, Disk Error (L Insufficient system resources to complete test in Cycle 2 with pattern Sequential data pattern
LOG NOTE: 2022-03-27 09:04:23, Disk, Disk Error (I Insufficient system resources to complete test in Cycle 2 with pattern Sequential data pattern
LOG NOTE: 2022-03-27 09:04:34, Status, Test run stopped

Script file:
SETDURATION 30000
SETCYCLES 0
Loop 1
{
SETDISK DISK I: MODE CYCLIC FILE 10 BLOCK 1024 SEEK 100
SETDISK DISK J: MODE CYCLIC FILE 10 BLOCK 1024 SEEK 100
SETDISK DISK K: MODE CYCLIC FILE 10 BLOCK 1024 SEEK 100
SETDISK DISK L: MODE CYCLIC FILE 10 BLOCK 1024 SEEK 100
SETLOG LOG yes NAME "D:\Reboot_BurnIn_CLI_Autoit_V3.22.4-SmiWinTools_20220224A_beta2\Autoit_Log\BurnInTest_ Log.log" TIME no
SETLOG LINES 1000000
SETERRORS ACTION STOP WINDOWS Block
RUN DISK
}

Thanks.

[Tim (PassMark)]

This error message is displayed when disk access fails with a windows error code between 1450 and 1455.
The test then sleeps for a full minute before trying again and if there is another error then then "Insufficient system resources error is logged in BurnInTest.

If you enable level 2 trace logging in the logging test preferences you should be able to get the specific windows error code that is triggering the error.

So if this error is occurring immediately after a reboot then the system seems to be taking a very long time to finish loading, perhaps you need to leave more time after the system boots before launching BurnInTest or put a sleep in the BurnInTest script.

[mars.wen]

Thanks for your reply.

In our test flow, BIT is executed after tens of seconds from system reboot, so it should be long enough.
We will try to enable level 2 logging, seeing if there is any helpful information.

BTW, we have replaced BIT v9.2 with v9.1 and run on the same platform, and it seems no error occurred anymore for a whole week.
So using the older version might be the temporary workaround.

[mars.wen]

Hi,
Still want to figure out the failure reason. Here's the log I re-run with level 2 trace logging.
It seems memory load rises dramatically somehow, causing insufficient resource situation.
Could you give any suggestion? Thanks.

[David (PassMark)]

Is this correct, you have 4 disk volumes, and each one is 10GB in size?

I: Local Drive, \\?\Volume, SSD_Test, NTFS, (10.00GB total, 9.97GB free)
J: Local Drive, \\?\Volume, SSD_Test, NTFS, (10.00GB total, 9.97GB free)
K: Local Drive, \\?\Volume, SSD_Test, NTFS, (10.00GB total, 9.97GB free)
L: Local Drive, \\?\Volume, SSD_Test, NTFS, (10.00GB total, 9.97GB free)

Why?

[David (PassMark)]

It also looks like there was a failed Windows update in the middle of one of test runs.

LOG NOTE:     2022-05-09 21:56:08, Application Event, 1118365 - Description: 錯誤容器 ，類型 0  事件名稱: WindowsUpdateFailure3

There is also this,

LOG NOTE: 2022-05-09 21:53:54, Disk , (End of test loop) Mem Total Phys: 8032MB [Available Phys: 11MB], Page: 16496MB [Available Page: 1690MB], Total Virt: 134217727MB [Available Virt: 134217340MB], Memory Load: 99%

So the system ran out of RAM as well (memory load 99%).
As you were only running the disk test, very little RAM should have been used by BurnInTest. So you need to have a look at what process on your system was using all the RAM.

[mars.wen]

Is this correct, you have 4 disk volumes, and each one is 10GB in size?
=> We have a test plan that runs IO workloads with 4 threads in parallel on the OS drive. To avoid problems like disk full, we simply split spaces into 4 10GB partitions and run each workload repeatedly. Is it a bad idea?

As you were only running the disk test, very little RAM should have been used by BurnInTest. So you need to have a look at what process on your system was using all the RAM.
=> Got it, we will monitor the memory usage of processes to see if we can find out the cause