VMware HolePunch and Chains

[Zew]

So I love playing around with VMware, and particularly storage and networking and it got me wondering, I'm trying to recovery storage space on VM's turns out its a little more complicated than I first thought.

So turns out I haven't been able to get my empty disk space back (I've been testing with a VM I know registers 40 gigs in the VM, but shows over 60 gigs used on the vSphere console) by simply svmotioning the thin provisioned disc to another datastore.

I've been following https://pubs.vmware.com/vsphere-55/topi ... 43902.html and it apparently turns our a couple things;

1) I had to use sdelete from sysinternals on the VM since VMtools removed it's GUI, and the CLI simply reported that the disc was not shrinkable.

2) you can't run the holepunch vmkfstools option against a NFS based datastore (I'm currently svmotioning back to local store to attempt this)

3) If the cmdlet fails I'll be forced to build a VMFS datastore with a different block size to recover VM empty space.

so finally my question. How will this affect my backup chain?

[Vitaliy S.]

If virtual disk size (max capacity) doesn't change, then you will only have large incremental file size on the job run. Zeroing out blocks makes sense when you do full job pass, see this topic for further reading > Unusually large VBK size

[PTide]

So turns out I haven't been able to get my empty disk space back by simply svmotioning the thin provisioned disc to another datastore.

I'm curious if you have tried running sdelete prior to your first svmotion?

[Zew]

I'm curious if you have tried running sdelete prior to your first svmotion?

Hahaha check out 1) in my initial post

[Zew]

If virtual disk size (max capacity) doesn't change, then you will only have large incremental file size on the job run. Zeroing out blocks makes sense when you do full job pass, see this topic for further reading > Unusually large VBK size

But what about bitlooker? Crap thanks for the link, I noticed you mentioned defrag before sdelete. I did not do this, it totally slipped my mind... another thing to test to bring teh size down. My question now is can I easily see this change in vSphere under used space?

[Vitaliy S.]

But what about bitlooker?

Ah, good point, sdelete is no longer required with v9, thanks to BitLooker.

My question now is can I easily see this change in vSphere under used space?

Yes, I do believe it should be visible through vSphere Client, otherwise how would you understand that all steps (manipulation with virtual disk) have been performed correctly?

[Zew]

I'll have to check my backup sizes to see for sure. While having bit looker enabled will help shrink backups from veeam perspective, it doesn't on vmwares persecptive.

My goal is to reclaim storage or my hypervisors datastores, in hopes it also cleans up my backup sets. but if bit-looker is doing as you say it is then my backups should be affected none.

So question now is, does bitlooker auto defrag to ensure best results from sdelete (or whatever veeam is using.. if its using vmware tools I wonder if it even indeed works at all) and the only reason I wonder that was cause I did intially try to shrink my disk using vmware tools shrink cmdlet vs sysinternals sdelete. However found that it reported back in the cli with "This disk is not able to be shrunk". After which I found out the disk was initially thick provisioned vs thin. The tumbling rabbit hole I tell ya. So now I'm doing the following.

1) Converted my disc to thin via svmotion. No size change seem to be shown.

2) Defrag (currently in progress, reported around 39% fragmented)

3) sdelete -c and then -z

4) shut vm down, run vmkfstools -K location of VMDK (if on localstorage or iSCSI) else svmotion from Datastore (1MB block size) to new datastore (8MB block size)

5) gonna check reported size on vsphere, at this point I'm hoping to see used space equal to that of the quest OS.

After this, would I be forced to do an active full, or will CBT and bitlooker properly pick up on these changes?

Even on just a defrag part... if you defrag a VM... would the VIB on the next run just be massive as CBT sees all the blocks being moved as changed data to copy??!

[Vitaliy S.]

While having bit looker enabled will help shrink backups from veeam perspective, it doesn't on vmwares persecptive.

Yes, that's expected, since bitooker doesn't change source VM disk.

So question now is, does bitlooker auto defrag to ensure best results from sdelete (or whatever veeam is using.. if its using vmware tools I wonder if it even indeed works at all) and the only reason I wonder that was cause I did intially try to shrink my disk using vmware tools shrink cmdlet vs sysinternals sdelete.

Bitlooker is not using VMware tools or any other VMware technique. Please see this link for more info > Deleted File Blocks (BitLooker)

After this, would I be forced to do an active full, or will CBT and bitlooker properly pick up on these changes?

After doing manipulations with the virtual disk, you will need to run active full backup job pass.

Even on just a defrag part... if you defrag a VM... would the VIB on the next run just be massive as CBT sees all the blocks being moved as changed data to copy??!

CBT will report on all virtual disk blocks that have changed, so you incremental job pass will take this in account too.

[Zew]

Veeam Backup & Replication accesses the MFT file on the VM guest OS to identify deleted file blocks, and zeros out these blocks.

The question is still not answered... HOW does it zero out the blocks. Yes I'm always a curious nelly, and I don't like vague answers.

Veeam Backup & Replication processes and transports data blocks of the VM image in the following manner:
If a data block of the VM image contains only the deleted file blocks, Veeam Backup & Replication does not read this data block from the source datastore.
If a data block of the VM image contains zeroed out blocks and other data, Veeam Backup & Replication copies this block to the target. Due to data compression, data blocks that are marked as deleted are compressed, and the size of the resulting backup or replica file reduces.

The rest makes sense, however would have been nice to have some details on block sizes at the VMFS level, such as the difference between a VMFS3 datastore with a 2 MB block size, vs lets say a VMFS3 8MB block datstore, or a VMFS5 NFS based datastore, etc.

Lets get nerdy here...

[PTide]

HOW does it zero out the blocks

Please take a look at this extensive topic on the subject.

Thanks

[skrause]

1) Converted my disc to thin via svmotion. No size change seem to be shown.

2) Defrag (currently in progress, reported around 39% fragmented)

3) sdelete -c and then -z

4) shut vm down, run vmkfstools -K location of VMDK (if on localstorage or iSCSI) else svmotion from Datastore (1MB block size) to new datastore (8MB block size)

5) gonna check reported size on vsphere, at this point I'm hoping to see used space equal to that of the quest OS.

If you are trying to reclaim provisioned space in VMware datastores, converting to Thin before running defrag and then sdelete will actually do the opposite of what you want. VMware thin provisioning doesn't actually throw out zeros it just doesn't write the allocated blocks until the guest says "write this block", it will write out zeros to disk (do a non-quick format of a disk in windows on a thin VMware volume to see this in action). If you are wanting to reclaim space from what is shown as provisioned in the datastore you want to have it start out as thick, run your defrag etc, then convert it to thin via storage vmotion. If you want to actually get back over-allocated free space you can also do that easiest when the disks are thick by editing the VMDK file via the CLI.

[Zew]

Interesting that you say that, when the above steps worked for me. I had to svmotion my vmdks to a datastore of different block size but I did mange to reclaim all my unused datastore space.

[Zew]

Please take a look at this extensive topic on the subject.

Thanks

Ahhhh you guys zero out the blocks on the backup side, no the actual guest.