ImageUSB NTFS Verification Problem

[Thor]

ImageUSB does not seem to have a problem creating/verifying image files created from NTFS drives. However, when attempting to verify after writing from an NTFS image file, it always gets a checksum error. I notice that the USB drive is remounted as soon as the writing is complete (before verification starts). I suspect that Windows is writing to the drive when the mount occurs and that is causing the verification error. Is it possible to add an option to not remount the USB drive after writing?

[Richard (PassMark)]

I tried to replicate this on my local machine and was unable to get the verification problem you are seeing.

1. Format USB Stick NTFS
2. Create Image with Verification On (PASSED)
3. Write Image to USB Stick with Verification On (PASSED)

If you run the software in debug mode (see help file) when it does verification, in additional to the checksums checks, it will also do actual byte comparison and output the offset where the contents differ in the logfile.

Other things:

Did you create the image on a different machines? What was the OS used? Different user accounts with different permissions?

[Thor]

Tried XP and Windows 7

I tried to replicate this on my local machine and was unable to get the verification problem you are seeing.

1. Format USB Stick NTFS
2. Create Image with Verification On (PASSED)
3. Write Image to USB Stick with Verification On (PASSED)

If you run the software in debug mode (see help file) when it does verification, in additional to the checksums checks, it will also do actual byte comparison and output the offset where the contents differ in the logfile.

Other things:

Did you create the image on a different machines? What was the OS used? Different user accounts with different permissions?

I'll try the logging. I tried creating the image and writing to the USB drive on two different machines and two different operating systems (XP and Windows 7), they both reacted the same way. The USB drives are: Kingston Digital DataTraveler SE9 8GB USB 2.0 (DTSE9H/8GBZ).

[Thor]

WinDiff and Log Files

I repeated the whole process (creating/restoring the image) on a Windows XP machine. I also ran WinDiff against the original directory structure that the source USB drive was created from. Even though it fails verification, the contents are identical which is why I think it may be an NTFS log issue.

Here's the ImageUSB log you requested:

10-May-2013 - 05:43:52:281 - PassMark Software - imageUSB v1.1.1011
10-May-2013 - 05:43:52:281 - Getting list of available removable USB drives...
10-May-2013 - 05:43:52:296 - Inspecting Drive C:\...
10-May-2013 - 05:43:52:296 - Inspecting Drive D:\...
10-May-2013 - 05:43:52:296 - Inspecting Drive E:\...
10-May-2013 - 05:43:52:312 - Inspecting Drive Z:\...
10-May-2013 - 05:43:52:312 - Ready...
10-May-2013 - 05:45:07:062 - Drive E:\ added to queue.
10-May-2013 - 05:45:07:062 - Creating image from Drive [E:\]...
10-May-2013 - 05:45:07:078 - Creating image D:\Transfer\ThumbDrive\ThumbDriveImage.bin from E:\ (drive 1)
10-May-2013 - 06:02:20:906 - Drive E:\ - Image completed.
10-May-2013 - 06:02:20:906 - Drive E:\ - MD5 Checksum: 46E32625BC7AD46A7FB85A4D48AA6DE2
10-May-2013 - 06:02:20:906 - Drive E:\ - SHA1 Checksum: 84971417F7AFB7AAC0D47715FDB2B736551133D4
10-May-2013 - 06:02:20:921 - Verifying Image D:\Transfer\ThumbDrive\ThumbDriveImage.bin created from Drive E:\...
10-May-2013 - 06:05:06:468 - Drive E:\ checksum completed.
10-May-2013 - 06:05:06:468 - Image D:\Transfer\ThumbDrive\ThumbDriveImage.bin - MD5 Checksum: 46E32625BC7AD46A7FB85A4D48AA6DE2
10-May-2013 - 06:05:06:468 - Verification passed - MD5 matched. (E:\).
10-May-2013 - 06:05:06:468 - Image D:\Transfer\ThumbDrive\ThumbDriveImage.bin - SHA1 Checksum: 84971417F7AFB7AAC0D47715FDB2B736551133D4
10-May-2013 - 06:05:06:484 - Verification passed - SHA1 matched. (E:\).
10-May-2013 - 06:09:13:906 - Getting list of available removable USB drives...
10-May-2013 - 06:09:13:906 - Inspecting Drive C:\...
10-May-2013 - 06:09:13:906 - Inspecting Drive D:\...
10-May-2013 - 06:09:13:906 - Inspecting Drive E:\...
10-May-2013 - 06:09:14:687 - Inspecting Drive Z:\...
10-May-2013 - 06:09:42:062 - Drive E:\ added to queue.
10-May-2013 - 06:09:42:250 - Writing image D:\Transfer\ThumbDrive\ThumbDriveImage.bin to E:\ (drive 1)
10-May-2013 - 06:44:26:218 - Drive E:\ write completed.
10-May-2013 - 06:55:06:140 - Drive E:\ checksum completed.
10-May-2013 - 06:55:06:156 - Drive E:\ - MD5 Checksum: C1534ED10DC46A0385A478F3C20CAD8A
10-May-2013 - 06:55:06:171 - VERIFICATION FAILED!!! MD5 Checksum does not match (E:\). Image MD5 Checksum: 46E32625BC7AD46A7FB85A4D48AA6DE2
10-May-2013 - 06:55:06:171 - Drive E:\ - SHA1 Checksum: 991039FFC0DC49F573C4CE26659F2C632A2823A9
10-May-2013 - 06:55:06:187 - VERIFICATION FAILED!!! SHA1 Checksum does not match (E:\). Image SHA1 Checksum: 84971417F7AFB7AAC0D47715FDB2B736551133D4
10-May-2013 - 06:55:16:812 - Exiting...

[Richard (PassMark)]

That seems to be the is the normal log that is output. To have imageUSB output debug information you need to start imageUSB with the "-d" command line arguement. It will save the debug information into the logfile, "imageUSB.log".

Quick way to do this is create a shortcut to imageUSB.exe. Right click->Properties and then edit the target. (i.e. "C:\Users\Passmark01\Desktop\imageUSB\imageUSB .exe -d")

[Thor]

Sorry, I missed the -d switch when I scanned the help...

10-May-2013 - 18:55:48:281 - PassMark Software - imageUSB v1.1.1011
10-May-2013 - 18:55:48:296 - Getting list of available removable USB drives...
10-May-2013 - 18:55:48:296 - Inspecting Drive C:\...
10-May-2013 - 18:55:48:312 - Debug: Thread (3204) Drive C:\: Not of USB Type Skipping. DriveType 3
10-May-2013 - 18:55:48:312 - Inspecting Drive D:\...
10-May-2013 - 18:55:48:312 - Debug: Thread (3204) Drive D:\: Not of USB Type Skipping. DriveType 3
10-May-2013 - 18:55:48:328 - Inspecting Drive E:\...
10-May-2013 - 18:55:48:328 - Debug: Thread (3204) Drive E:\ Size: 7797530624 Bytes Volume: LogiQuest APLs Other: NTFS


10-May-2013 - 18:55:48:328 - Inspecting Drive Z:\...
10-May-2013 - 18:55:48:343 - Debug: Thread (3204) Drive Z:\: Not of USB Type Skipping. DriveType 2
10-May-2013 - 18:55:48:343 - Ready...
10-May-2013 - 18:56:43:312 - Drive E:\ added to queue.
10-May-2013 - 18:56:43:312 - Debug: Thread (2840) Successfully opened image file D:\Transfer\ThumbDrive\ThumbDriveImage.bin (7802454528 bytes) for read acess.
10-May-2013 - 18:56:43:328 - Debug: Thread (2840) imageUSB image. Actual image size (w/o header) is 7802454016 bytes.
10-May-2013 - 18:56:43:343 - Debug: Thread (2840) UFD \\.\PhysicalDrive1 size 7803174912 bytes.
10-May-2013 - 18:56:43:343 - Debug: Thread (2840) Unmounting UFD E:\ to prepare for write.
10-May-2013 - 18:56:43:500 - Writing image D:\Transfer\ThumbDrive\ThumbDriveImage.bin to E:\ (drive 1)
10-May-2013 - 18:56:44:453 - Debug: Thread (2840) Writing...0.1%
10-May-2013 - 18:56:44:984 - Debug: Thread (2840) Writing...0.3%
10-May-2013 - 18:56:45:484 - Debug: Thread (2840) Writing...0.4%
.
.
.
10-May-2013 - 19:30:34:015 - Debug: Thread (2840) Writing...99.7%
10-May-2013 - 19:30:37:187 - Debug: Thread (2840) Writing...99.9%
10-May-2013 - 19:30:39:453 - Debug: Thread (2840) Writing...100.0%
10-May-2013 - 19:30:39:453 - Drive E:\ write completed.
10-May-2013 - 19:30:39:453 - Debug: Thread (2840) Copying first block from image to UFD...
10-May-2013 - 19:30:39:515 - Debug: Thread (2840) Verifying image written...
10-May-2013 - 19:30:40:390 - Debug: Thread (2840) Verifying...0.1%
10-May-2013 - 19:30:41:171 - Debug: Thread (2840) Verifying...0.3%
10-May-2013 - 19:30:41:937 - Debug: Thread (2840) Verifying...0.4%
10-May-2013 - 19:30:42:687 - Debug: Thread (2840) Verifying...0.5%
10-May-2013 - 19:30:43:468 - Debug: Thread (2840) Verifying...0.7%
10-May-2013 - 19:30:44:218 - Debug: Thread (2840) Verifying...0.8%
10-May-2013 - 19:30:44:953 - Debug: Thread (2840) Verifying...0.9%
10-May-2013 - 19:30:45:703 - Debug: Thread (2840) Verifying...1.1%
.
.
.
10-May-2013 - 19:34:56:343 - Debug: Thread (2840) Verifying...40.3%
10-May-2013 - 19:34:57:203 - Debug: Thread (2840) Verifying...40.5%
10-May-2013 - 19:34:58:046 - Debug: Thread (2840) Verifying...40.6%
10-May-2013 - 19:34:58:906 - Debug: Thread (2840) Verifying...40.7%
10-May-2013 - 19:34:59:171 - Debug: Thread (2840) Error: Contents at offset 3179282433 are different
10-May-2013 - 19:34:59:781 - Debug: Thread (2840) Verifying...40.9%
10-May-2013 - 19:35:00:625 - Debug: Thread (2840) Verifying...41.0%
10-May-2013 - 19:35:01:484 - Debug: Thread (2840) Verifying...41.1%
10-May-2013 - 19:35:02:328 - Debug: Thread (2840) Verifying...41.3%
10-May-2013 - 19:35:02:593 - Debug: Thread (2840) Error: Contents at offset 3221225473 are different
10-May-2013 - 19:35:03:187 - Debug: Thread (2840) Verifying...41.4%
10-May-2013 - 19:35:04:046 - Debug: Thread (2840) Verifying...41.5%
.
.
.
10-May-2013 - 19:41:16:609 - Debug: Thread (2840) Verifying...99.7%
10-May-2013 - 19:41:17:468 - Debug: Thread (2840) Verifying...99.9%
10-May-2013 - 19:41:18:312 - Debug: Thread (2840) Verifying...100.0%
10-May-2013 - 19:41:18:390 - Drive E:\ checksum completed.
10-May-2013 - 19:41:18:390 - Drive E:\ - MD5 Checksum: E7435E3D9D76A9BA3281808486DF3EC4
10-May-2013 - 19:41:18:390 - VERIFICATION FAILED!!! MD5 Checksum does not match (E:\). Image MD5 Checksum: 46E32625BC7AD46A7FB85A4D48AA6DE2
10-May-2013 - 19:41:18:390 - Drive E:\ - SHA1 Checksum: ACC382E3A2432BE81010D24E66EB1FB5B2CFDFAF
10-May-2013 - 19:41:18:406 - VERIFICATION FAILED!!! SHA1 Checksum does not match (E:\). Image SHA1 Checksum: 84971417F7AFB7AAC0D47715FDB2B736551133D4
10-May-2013 - 19:44:32:250 - Exiting...

[Richard (PassMark)]

The debug log indicates that the contents at the offset 3179282433 and 3221225473 are different from the source image. This could be because of various reasons including possible NTFS manipulation (which we were unable to replicate). It could also be that the USB drive contains bad sector or is defective as well.

Even if Windiff reports the same directory structure, those offset could be in unallocated space. You could possibly use OSForensics' Drive Preparation module or BurnInTest Disk test to quickly test the health of the flash drive.

We will consider adding an option not to dismount the drive before verifying in a future release, but there is currently no time table of when it will be released.

[David (PassMark)]

You can also use the OSForensics "Raw disk viewer" module on the source and destination drive, to determine what the hex differences actually are, and also see what file this sector on the drive was part of.

[imageman]

Hi,

I am having the same problem with image verification failed.

I checked the usb drives with OSForensics' Drive Preparation module and BurnInTest Disk test

Both tests passed.

The file size is obviously wrong when I compare the image to the disk size or use windiff.

sometimes imageusb will verify successfully 1 or 2 usb. Also, I have the latest version

Does anyone have any other pointers. Ive tried different OS but that does not work. Ive run imageusb in compatibility mode and as admin always still not working. I ran debug mode.

The funny thing is, imageusb worked a few times for me and verified 5 drives at one time success a few times and then it started failing. I cant figure out why.

[David (PassMark)]

What was the output from debug mode?
What file(s) had different sizes?

Maybe there is some other process running on the machine that is making changes on the USB drive (in parallel between the write and verify stages).
This might be malware replicating itself on to the USB drive, it might be some background file backup task that flags files on the USB drive as backed up, it might be some anti-virus program scanning the drive then making changes in NTFS to indicate which files have been scanned.

[imageman]

I guess I just wont use Verify any more!

I was also hoping maybe passmark could add in a write speed status on each usb for imgusb software during imaging