v9: BitLooker Zeroing

[NTmatter]

I was just reading through the BitLooker documentation. It states that BitLooker will "zero out dirty blocks on the VM guest OS before it copies VM data to the target location." Does this actually issue any I/O to the production storage, or is it cleverly ignoring/substituting the dirty blocks as it sends them to the repository?

The situation is that I'm about to switch this on for a well-churned multi-terabyte VM, prior to an Active Full backup. I'd like to figure out if I'm about to zero out a large chunk of data, extending the backup window while consuming ginormous amounts of bandwidth, giving my users a bad day.

To recap, I'm asking if BitLooker will:
a) Write zeroes to the VM's disks (and thereby production storage), snapshot the VM, then compress the blocks during transport.
b) Snapshot the VM, write zeroes to an ephemeral snapshot (which would also wind up on production storage), then compress the blocks during transport.
c) Snapshot the VM, skip (or do inline replacement of) the changed dirty blocks during transport.
d) Something completely different?

Thanks for any clarification that you can provide!

[PTide]

Hi,

c) Snapshot the VM, skip (or do inline replacement of) the changed dirty blocks during transport.

This. Dirty blocks are just skipped and don't get transported anywhere. Blocks are skipped based on data from MFT. So there is almost no impact on production storage (only small amount of read operations to obtain MFT data).

Thank you.

[Gostev]

Correct, and you should see the results (amount of data read and transferred, and smaller backup file size) instantly - as long as you have a lot of deleted data on virtual disks, which is however not so usual for server workloads.

But, keep in mind that during the incremental runs, Veeam will only process changed blocks, which are thus unlikely to belong to deleted files (otherwise, why would they be changing). So, material impact from this feature can only be seen during full backups.

[NTmatter]

Hooray! Sounds like it should be a big win for our environment. Would you be able to clarify this statement a little further:

So, material impact from this feature can only be seen during full backups.

Do you mean should only be seen? Or is BitLooker mutually exclusive to Changed Block Tracking?

As some background, we actually have a fair bit of churn from regular user activity (Photoshop + Atomic Save), DFS Replication staging, and Windows' Deduplication. All of this shuffling results in huge amounts of dirty blocks.

[Gostev]

Yes, they are mutually exclusive, simply because CBT already does not "pick up" blocks belonging to deleted files. Remember that deleting a file does not change contents of any of its blocks, they remain on disk as is. Instead, only the file allocation table is updated to mark the file as deleted. This is exactly what makes undelete possible

[mkaec]

Hyper-V generation 2 VMs support SCSI unmap and I think VSS already causes those blocks to be skipped during backup. I'm thinking this is a great option to enable for generation 1 VMs, but won't have a big impact on generation 2 VMs (on Hyper-V).

However, I'm a bit nervous about turning it on. What is the risk of MFT corruption in the VM causing incorrect blocks to be skipped?

[dellock6]

I think that if you have MFT corruption, the skipped blocks is not the worst problem you have...

[mkaec]

Not necessarily. I've seen chkdsk /f report, with regularity, that it is fixing minor things. I don't know how much, or if at all, that kind of stuff would impact bitlooker. But, if there were more serious MFT problems, would trying to make use of a backup that was damaged by MFT corruption make matters worse? Is MFT corruption detected during backup or could backups silently go on for a while before it is noticed?

I'm not trying to be negative. I am sure all this was considered during development and I'm just interested in the details.

[Gostev]

Since we will not be able to parse corrupted MFT, BitLooker processing will simply be skipped.

[NTmatter]

they are mutually exclusive

Apologies for the nitpicking - I'm still a bit fuzzy on this. I'm looking at the sequence of Full Backup > create file > delete file > Incremental Backup. The final MFT will have no entry for the churned blocks from the short-lived file, but the dirtied blocks will be copied into the incremental backup?

It's sounding like BitLooker is only used for Full Backups, not for Incrementals?

Thanks for all the help on this one

[PTide]

The final MFT will have no entry for the churned blocks from the short-lived file

And that's why those blocks won't be included into incremental backup with BitLooker enabled. If you disable BitLooker then CBT will pick up dirty blocks even if the files were deleted. In next run those blocks, unless you change them again, won't be copied since they are already present in previous backup. With both options enabled you will get all your changed blocks in your incremental except for those that do not belong to existing files (based on MFT data)

[mkaec]

Apologies for the nitpicking - I'm still a bit fuzzy on this.

Let's say a backup runs and backs up a 100 MB file. Then the next day you delete the file. Because the blocks were already backed up, the next incremental will not back it up again. This is CBT at work.

Now let's say you create a 100 MB file and then delete it the same day. Normally, it would get backed up, but Bitlooker will cause it to be skipped.

[NTmatter]

Thanks, PTide and mkaec!

Crystal clear now. BitLooker + CBT + High Churn = Minimal Bloat.

[larry]

Trying to understand what Veeam is doing with BitLooker
In the manual I read “Veeam Backup & Replication accesses the MFT file on the VM guest OS to identify deleted file blocks, and zeros out these blocks.” Does this mean Veeam is wiping my VM deleted area with zeros like a secure delete? If it does, then the first run I should expect a lot of SAN traffic on the next mirror job for that disk. I just need the first run at night if this is the case.

[chrisdearden]

no - we just write zero in the backup for it.

[lp@albersdruck.de]

With Bitlooker enabled, if I activate shadow copies on a VM disk ("previous versions"), will Veeam backup the blocks of files that were deleted but still exist as a shadow copy?

[PTide]

Hi,

With Bitlooker enabled, if I activate shadow copies on a VM disk ("previous versions"), will Veeam backup the blocks of files that were deleted but still exist as a shadow copy?

Datablocks that are included in a shadow copy are treated just as regular datablocks so if shadow copy session occurs between backup sessions all new blocks will go to the incremental backup.

Thank you.

[ober72]

Correct, and you should see the results (amount of data read and transferred, and smaller backup file size) instantly - as long as you have a lot of deleted data on virtual disks, which is however not so usual for server workloads.

But, keep in mind that during the incremental runs, Veeam will only process changed blocks, which are thus unlikely to belong to deleted files (otherwise, why would they be changing). So, material impact from this feature can only be seen during full backups.

In the case of an upgrade from Veeam 8 to Veeam 9 and then turning on this feature with existing jobs it will though have an immediate effect on the next incremental right?

thanks

[PTide]

will though have an immediate effect on the next incremental right?

Only if you've changed and deleted files between incremental runs. If only change or deletion occurred between incremental sessions then BitLooker will give no impact.

Thank you.

[pauliem]

I just ran a full backup against a VM that hosts a lot of files and has a high churn rate. The size of the backup dropped from 1.3Tb to 730Gb. Very impressive.

[ober72]

Would it make sense then in some circumstances to run active fulls to get the full space savings?

thanks

[skrause]

If you want to get the best space savings, using sdelete or some other method to write zeroes in all the free space and then running an active full will usually be your best solution to shrink your backup size.

Bitlooker is really nice for situations where there is high churn on the server (2012R2 servers with Deduplication enabled for example) so that you don't need to do the zero-out stuff as often (or at all).

[mkaec]

I don't think there would be much benefit to running SDelete. That's the point of BitLooker in that it knows to exclude the non-zeroed deleted files.

And I don't think there is a benefit to running an active full. BitLooker works fine at skipping deleted files in future backups even if previous backups did include the deleted files.

Now if you are backing up to a deduplicating device, changing the BitLooker setting may cause the next active full to not dedup as well as it would have. That's something to keep in mind.

[PTide]

Now if you are backing up to a deduplicating device, changing the BitLooker setting may cause the next active full to not dedup as well as it would have. That's something to keep in mind.

I'd say that enabling BitLooker will leave less stuff for appliance to dedupe - zeroes will be eliminated from backup by Veeam before they'll make it to appliance.

Also please keep in mind that in some cases you might get confusing results with BitLooker enabled for reverse incremental chains.

Thank you.

[mkaec]

I'd say that enabling BitLooker will leave less stuff for appliance to dedupe - zeroes will be eliminated from backup by Veeam before they'll make it to appliance.

That's true. But, it also changes the block arrangement. The matching algorithms may have a more difficult time matching up to the data already in the repository. I have tried this with one volume so far and the subsequent active full only deduped 2:1 instead of the more typical 10:1.

Before BitLooker - Veeam: 1.8 TB; Appliance: 100 GB
After BitLooker - Veeam: 1.6 TB; Appliance: 780 GB
Second Active Full After BitLooker - Veeam: 1.6 TB; Appliance 142 GB

Now a sample size of 1 is really small and I did have Windows dedup enabled on the volume. So, YMMV. I'm planning to turn BitLooker on some more volumes in the future. I'll report back when I've done it on one that does not have dedup running.

[PTide]

First of all - I've read my previous post one more time and I have to say that this:

zeroes will be eliminated from backup by Veeam before they'll make it to appliance.

supposed to be this:

deleted files will be eliminated from backup by Veeam before they'll make it to appliance.

Sorry for confusion.

Before BitLooker - Veeam: 1.8 TB; Appliance: 100 GB
After BitLooker - Veeam: 1.6 TB; Appliance: 780 GB
Second Active Full After BitLooker - Veeam: 1.6 TB; Appliance 142 GB

Yes, there is some impact on a dedupe ratio if there were no BitLooker option enabled before. However, after a while, your dedupe appliance will get filled with a "bitlookered" blocks of data thus dedupe ratio will become better. As a sidenote I'd like to mention that deduplicating storage is generally not a good choice for primary backup storage.

So I'm looking forward to hear about the results of a new test from you.

Thank you!

[hyvokar]

But, keep in mind that during the incremental runs, Veeam will only process changed blocks, which are thus unlikely to belong to deleted files (otherwise, why would they be changing). So, material impact from this feature can only be seen during full backups.

Hi! This actually explains why bitlooker isnt working as expected for me. By not using bitlooker on incremental runs, it does not apply on data changed by windows data deduplication optimization jobs. Is there any change, bitlooker could be enabled for incremental runs?

[mkaec]

BitLooker is active during incremental backups. I think the text you quoted is just trying to communicate that full backups are more likely to show a more dramatic difference.

[hyvokar]

Hmmh... My problem is, that after a full optimization on a windows 2012r2 fileserver (4th saturday, each month) my backup size (transferred data) bumps up by 150-200GB. On every other saturday (after normal optimization), backup size is 70-80GB higher than normal. On a normal day, backup size is between 10-20GB.

So can someone confirm, should bitlooker work with windows dedupe and forever reverse incremental backups ?


E: I've ugraded from v8, and only thing I changed was to check bitlooker checkbox for the job. Do I need to run active full or something like that?

[mkaec]

I've also seen this with Windows dedup. I don't think there's really any way around it. If you look at the size of a deduped file, you'll actually see a really small "size on disk" value. The actual file data is stored in 1GB files in System Volume Information\Dedup\ChuknStore\<GUID>\Data. I think the incremental numbers you are seeing are related to garbage collection. When a file is deleted, its contents are left in the related CCC file. Then at some point in the future, Windows clears out the old data from the CCC file. My guess is that it creates a new CCC file with just the still-valid data and deletes the old one. But, I'm not sure of what the exact implementation is. Veeam will see the this CCC activity, rightly so, as changed blocks and back them up in the incremental.

I actually think Windows dedup and block level backup don't mix all that well. Windows dedup just causes extra changed blocks due to how it works.