Issue with sdelete and guest Windows 2012 deduplication

[m.novelli]

Hi guys, would like to share an issue with sdelete on a Windows 2012 guest VM with deduplication enabled

One user copied by mistake 700 GB of data on the fileserver of one of my customer. That VM had Windows 2012 deduplication enabled. Veeam backups increased and exploded as well...

After removing the 700 GB of data the Windows volume still showed the space as allocated. With those two commands I've manually invoked the Windows 2012 Deduplication Garbage Collection and Scrub:

%systemroot%\system32\ddpcli.exe enqueue /gc /scheduled /vol * /priority normal /memory 50 /backoff
%systemroot%\system32\ddpcli.exe enqueue /scrub /scheduled /vol * /priority normal /memory 50 /backoff

At this point Windows showed a normal space allocation on the data partition, but Veeam backups where still too big and slow and didn't reflect actual size of real data. I've launched sdelete as normal to zero unused space but I've noticed an odd behavior on the deduplicated volume: Windows begin to show an increase in space used as sdelete was processing blocks. At 50% of sdelete processing I decided that sdelete was not deduplication-friendly.

Long story short, I've solved my issue running again the Windows 2012 Deduplication Garbage Collection and Scrub commands and then resetting CBT on the specific VM: http://www.veeam.com/kb1113

Marco

[tsightler]

I've launched sdelete as normal to zero unused space but I've noticed an odd behavior on the deduplicated volume: Windows begin to show an increase in space used as sdelete was processing blocks.

As far as I know this is just the way sdelete works. It basically creates a huge file that contains nothing but zeros until the disk is full (or almost full), effectively writing zeros in any place where there was previously data, then it just deletes this file leaving all the now zeroed blocks.

[dellock6]

Marco, you always need to remember the two different layers you are working with:
- win2012 dedup is enabled "inside" the guest OS, and optimises the free space inside the windows partition
- Veeam does not know nothing about the dedup, it works with the underlying VMDK file and reads CBT informations.

So it can happen vmdk blocks are not marked as free even if inside the guest those area of the partition are now free. One interesting solution for these problems would be in the future the extensive use of se (space efficient)-sparse disks, with a possibility to reclaim CBT blocks when marked unused by the Guest OS; but VMware itself is in the early stages of this technology.

Luca.

[m.novelli]

As far as I know this is just the way sdelete works. It basically creates a huge file that contains nothing but zeros until the disk is full (or almost full), effectively writing zeros in any place where there was previously data, then it just deletes this file leaving all the now zeroed blocks.

Sorry, you are right, I didn't explained correctly my issue with sdelete: I've stopped it at 10% to see if Windows could see the correct free space on the deduplicated volume, but it showed it as allocated. Then I've run sdelete up to 50% and stopped: again Windows was showing the zeroed space as allocated. I didn't have heart to run it up to 100%
I suspect some odd interaction between sdelete and Windows 2012 deduplication features
I've also noted that sdelete was superslow on that deduplicated volume: after 20 hours it zeroed about 50% of an 1.5 TB VMDK located on a brand new Equallogic (SATA disks with RAID10 configuration)

Marco

[dellock6]

My suspect is that on a deduplicated volume is better to use the native reclaim tools you quoted in the opening posts. Zeroing activities on deduped devices have usually lead to weird results, I think Win2012 is not different.

Luca.

[tsightler]

As far as I not native reclaim tools on 2012 only reclaim data from the dedupe pool, not from the rest of the disk. Since Windows 2012 is a post-process dedupe if you copy 700GB to it, then immediately delete it, the data won't even be in the dedupe pool so the native tools won't do anything.

[VladV]

Regarding speed it depends on what parameter and version of sdelete combination you are using. For version 1.61 using -z only zeros out the free space and -c cleans it by zeroing, writing 1s, then randoms, thus taking more time to finish.

Another thing is that if you filled the partition with 700gb and then deleted it I don't think that scrubbing fills the free space with zeroes. On a normal ntfs partition filling it and then deleting the data without zeroing, even after resseting the CBT, i believe that your full backups size will increase with the amount of data previously written (deduped and compressed if activated in veeam). Because of how NTFS manages deletions the data still remains in the VMDK and will be backed up even if deleted although the garbage collection process may add highly compressible data in the free space and so the full backup may not increase that much. Filling it with zeroes makes it highly compressible.