Comprehensive data protection for all workloads
NTmatter
Influencer
Posts: 21
Liked: 8 times
Joined: Mar 14, 2014 11:16 am
Full Name: Thomas Johnson
Contact:

v9: BitLooker Zeroing

Post by NTmatter »

I was just reading through the BitLooker documentation. It states that BitLooker will "zero out dirty blocks on the VM guest OS before it copies VM data to the target location." Does this actually issue any I/O to the production storage, or is it cleverly ignoring/substituting the dirty blocks as it sends them to the repository?

The situation is that I'm about to switch this on for a well-churned multi-terabyte VM, prior to an Active Full backup. I'd like to figure out if I'm about to zero out a large chunk of data, extending the backup window while consuming ginormous amounts of bandwidth, giving my users a bad day.

To recap, I'm asking if BitLooker will:
a) Write zeroes to the VM's disks (and thereby production storage), snapshot the VM, then compress the blocks during transport.
b) Snapshot the VM, write zeroes to an ephemeral snapshot (which would also wind up on production storage), then compress the blocks during transport.
c) Snapshot the VM, skip (or do inline replacement of) the changed dirty blocks during transport.
d) Something completely different?

Thanks for any clarification that you can provide!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: v9: BitLooker Zeroing

Post by PTide »

Hi,
c) Snapshot the VM, skip (or do inline replacement of) the changed dirty blocks during transport.
This. Dirty blocks are just skipped and don't get transported anywhere. Blocks are skipped based on data from MFT. So there is almost no impact on production storage (only small amount of read operations to obtain MFT data).

Thank you.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v9: BitLooker Zeroing

Post by Gostev »

Correct, and you should see the results (amount of data read and transferred, and smaller backup file size) instantly - as long as you have a lot of deleted data on virtual disks, which is however not so usual for server workloads.

But, keep in mind that during the incremental runs, Veeam will only process changed blocks, which are thus unlikely to belong to deleted files (otherwise, why would they be changing). So, material impact from this feature can only be seen during full backups.
NTmatter
Influencer
Posts: 21
Liked: 8 times
Joined: Mar 14, 2014 11:16 am
Full Name: Thomas Johnson
Contact:

Re: v9: BitLooker Zeroing

Post by NTmatter »

Hooray! Sounds like it should be a big win for our environment. Would you be able to clarify this statement a little further:
So, material impact from this feature can only be seen during full backups.
Do you mean should only be seen? Or is BitLooker mutually exclusive to Changed Block Tracking?

As some background, we actually have a fair bit of churn from regular user activity (Photoshop + Atomic Save), DFS Replication staging, and Windows' Deduplication. All of this shuffling results in huge amounts of dirty blocks.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v9: BitLooker Zeroing

Post by Gostev »

Yes, they are mutually exclusive, simply because CBT already does not "pick up" blocks belonging to deleted files. Remember that deleting a file does not change contents of any of its blocks, they remain on disk as is. Instead, only the file allocation table is updated to mark the file as deleted. This is exactly what makes undelete possible ;)
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec »

Hyper-V generation 2 VMs support SCSI unmap and I think VSS already causes those blocks to be skipped during backup. I'm thinking this is a great option to enable for generation 1 VMs, but won't have a big impact on generation 2 VMs (on Hyper-V).

However, I'm a bit nervous about turning it on. What is the risk of MFT corruption in the VM causing incorrect blocks to be skipped?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: v9: BitLooker Zeroing

Post by dellock6 » 3 people like this post

I think that if you have MFT corruption, the skipped blocks is not the worst problem you have...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec »

Not necessarily. I've seen chkdsk /f report, with regularity, that it is fixing minor things. I don't know how much, or if at all, that kind of stuff would impact bitlooker. But, if there were more serious MFT problems, would trying to make use of a backup that was damaged by MFT corruption make matters worse? Is MFT corruption detected during backup or could backups silently go on for a while before it is noticed?

I'm not trying to be negative. I am sure all this was considered during development and I'm just interested in the details.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v9: BitLooker Zeroing

Post by Gostev »

Since we will not be able to parse corrupted MFT, BitLooker processing will simply be skipped.
NTmatter
Influencer
Posts: 21
Liked: 8 times
Joined: Mar 14, 2014 11:16 am
Full Name: Thomas Johnson
Contact:

Re: v9: BitLooker Zeroing

Post by NTmatter »

Gostev wrote:they are mutually exclusive
Apologies for the nitpicking - I'm still a bit fuzzy on this. I'm looking at the sequence of Full Backup > create file > delete file > Incremental Backup. The final MFT will have no entry for the churned blocks from the short-lived file, but the dirtied blocks will be copied into the incremental backup?

It's sounding like BitLooker is only used for Full Backups, not for Incrementals?

Thanks for all the help on this one :)
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: v9: BitLooker Zeroing

Post by PTide » 1 person likes this post

The final MFT will have no entry for the churned blocks from the short-lived file
And that's why those blocks won't be included into incremental backup with BitLooker enabled. If you disable BitLooker then CBT will pick up dirty blocks even if the files were deleted. In next run those blocks, unless you change them again, won't be copied since they are already present in previous backup. With both options enabled you will get all your changed blocks in your incremental except for those that do not belong to existing files (based on MFT data)
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec » 3 people like this post

NTmatter wrote:Apologies for the nitpicking - I'm still a bit fuzzy on this.
Let's say a backup runs and backs up a 100 MB file. Then the next day you delete the file. Because the blocks were already backed up, the next incremental will not back it up again. This is CBT at work.

Now let's say you create a 100 MB file and then delete it the same day. Normally, it would get backed up, but Bitlooker will cause it to be skipped.
NTmatter
Influencer
Posts: 21
Liked: 8 times
Joined: Mar 14, 2014 11:16 am
Full Name: Thomas Johnson
Contact:

Re: v9: BitLooker Zeroing

Post by NTmatter »

Thanks, PTide and mkaec!

Crystal clear now. BitLooker + CBT + High Churn = Minimal Bloat.
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

[MERGED]: Understanding BitLooker

Post by larry »

Trying to understand what Veeam is doing with BitLooker
In the manual I read “Veeam Backup & Replication accesses the MFT file on the VM guest OS to identify deleted file blocks, and zeros out these blocks.” Does this mean Veeam is wiping my VM deleted area with zeros like a secure delete? If it does, then the first run I should expect a lot of SAN traffic on the next mirror job for that disk. I just need the first run at night if this is the case.
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: v9: BitLooker Zeroing

Post by chrisdearden »

no - we just write zero in the backup for it.
lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 33 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: v9: BitLooker Zeroing

Post by lp@albersdruck.de »

With Bitlooker enabled, if I activate shadow copies on a VM disk ("previous versions"), will Veeam backup the blocks of files that were deleted but still exist as a shadow copy?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: v9: BitLooker Zeroing

Post by PTide »

Hi,
With Bitlooker enabled, if I activate shadow copies on a VM disk ("previous versions"), will Veeam backup the blocks of files that were deleted but still exist as a shadow copy?
Datablocks that are included in a shadow copy are treated just as regular datablocks so if shadow copy session occurs between backup sessions all new blocks will go to the incremental backup.

Thank you.
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: v9: BitLooker Zeroing

Post by ober72 »

Gostev wrote:Correct, and you should see the results (amount of data read and transferred, and smaller backup file size) instantly - as long as you have a lot of deleted data on virtual disks, which is however not so usual for server workloads.

But, keep in mind that during the incremental runs, Veeam will only process changed blocks, which are thus unlikely to belong to deleted files (otherwise, why would they be changing). So, material impact from this feature can only be seen during full backups.
In the case of an upgrade from Veeam 8 to Veeam 9 and then turning on this feature with existing jobs it will though have an immediate effect on the next incremental right?

thanks
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: v9: BitLooker Zeroing

Post by PTide »

will though have an immediate effect on the next incremental right?
Only if you've changed and deleted files between incremental runs. If only change or deletion occurred between incremental sessions then BitLooker will give no impact.

Thank you.
pauliem
Enthusiast
Posts: 26
Liked: 3 times
Joined: Oct 28, 2010 11:31 am
Contact:

Re: v9: BitLooker Zeroing

Post by pauliem » 2 people like this post

I just ran a full backup against a VM that hosts a lot of files and has a high churn rate. The size of the backup dropped from 1.3Tb to 730Gb. Very impressive.
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: v9: BitLooker Zeroing

Post by ober72 »

Would it make sense then in some circumstances to run active fulls to get the full space savings?

thanks
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: v9: BitLooker Zeroing

Post by skrause »

If you want to get the best space savings, using sdelete or some other method to write zeroes in all the free space and then running an active full will usually be your best solution to shrink your backup size.

Bitlooker is really nice for situations where there is high churn on the server (2012R2 servers with Deduplication enabled for example) so that you don't need to do the zero-out stuff as often (or at all).
Steve Krause
Veeam Certified Architect
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec » 1 person likes this post

I don't think there would be much benefit to running SDelete. That's the point of BitLooker in that it knows to exclude the non-zeroed deleted files.

And I don't think there is a benefit to running an active full. BitLooker works fine at skipping deleted files in future backups even if previous backups did include the deleted files.

Now if you are backing up to a deduplicating device, changing the BitLooker setting may cause the next active full to not dedup as well as it would have. That's something to keep in mind.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: v9: BitLooker Zeroing

Post by PTide »

Now if you are backing up to a deduplicating device, changing the BitLooker setting may cause the next active full to not dedup as well as it would have. That's something to keep in mind.
I'd say that enabling BitLooker will leave less stuff for appliance to dedupe :) - zeroes will be eliminated from backup by Veeam before they'll make it to appliance.

Also please keep in mind that in some cases you might get confusing results with BitLooker enabled for reverse incremental chains.

Thank you.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec » 1 person likes this post

PTide wrote:I'd say that enabling BitLooker will leave less stuff for appliance to dedupe :) - zeroes will be eliminated from backup by Veeam before they'll make it to appliance.
That's true. But, it also changes the block arrangement. The matching algorithms may have a more difficult time matching up to the data already in the repository. I have tried this with one volume so far and the subsequent active full only deduped 2:1 instead of the more typical 10:1.

Before BitLooker - Veeam: 1.8 TB; Appliance: 100 GB
After BitLooker - Veeam: 1.6 TB; Appliance: 780 GB
Second Active Full After BitLooker - Veeam: 1.6 TB; Appliance 142 GB

Now a sample size of 1 is really small and I did have Windows dedup enabled on the volume. So, YMMV. I'm planning to turn BitLooker on some more volumes in the future. I'll report back when I've done it on one that does not have dedup running.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: v9: BitLooker Zeroing

Post by PTide »

First of all - I've read my previous post one more time and I have to say that this:
zeroes will be eliminated from backup by Veeam before they'll make it to appliance.
supposed to be this:
deleted files will be eliminated from backup by Veeam before they'll make it to appliance.
Sorry for confusion.
Before BitLooker - Veeam: 1.8 TB; Appliance: 100 GB
After BitLooker - Veeam: 1.6 TB; Appliance: 780 GB
Second Active Full After BitLooker - Veeam: 1.6 TB; Appliance 142 GB
Yes, there is some impact on a dedupe ratio if there were no BitLooker option enabled before. However, after a while, your dedupe appliance will get filled with a "bitlookered" blocks of data thus dedupe ratio will become better. As a sidenote I'd like to mention that deduplicating storage is generally not a good choice for primary backup storage.

So I'm looking forward to hear about the results of a new test from you.

Thank you!
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: v9: BitLooker Zeroing

Post by hyvokar »

Gostev wrote:But, keep in mind that during the incremental runs, Veeam will only process changed blocks, which are thus unlikely to belong to deleted files (otherwise, why would they be changing). So, material impact from this feature can only be seen during full backups.
Hi! This actually explains why bitlooker isnt working as expected for me. By not using bitlooker on incremental runs, it does not apply on data changed by windows data deduplication optimization jobs. Is there any change, bitlooker could be enabled for incremental runs?
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec »

BitLooker is active during incremental backups. I think the text you quoted is just trying to communicate that full backups are more likely to show a more dramatic difference.
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: v9: BitLooker Zeroing

Post by hyvokar »

Hmmh... My problem is, that after a full optimization on a windows 2012r2 fileserver (4th saturday, each month) my backup size (transferred data) bumps up by 150-200GB. On every other saturday (after normal optimization), backup size is 70-80GB higher than normal. On a normal day, backup size is between 10-20GB.

So can someone confirm, should bitlooker work with windows dedupe and forever reverse incremental backups ?


E: I've ugraded from v8, and only thing I changed was to check bitlooker checkbox for the job. Do I need to run active full or something like that?
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9: BitLooker Zeroing

Post by mkaec »

I've also seen this with Windows dedup. I don't think there's really any way around it. If you look at the size of a deduped file, you'll actually see a really small "size on disk" value. The actual file data is stored in 1GB files in System Volume Information\Dedup\ChuknStore\<GUID>\Data. I think the incremental numbers you are seeing are related to garbage collection. When a file is deleted, its contents are left in the related CCC file. Then at some point in the future, Windows clears out the old data from the CCC file. My guess is that it creates a new CCC file with just the still-valid data and deletes the old one. But, I'm not sure of what the exact implementation is. Veeam will see the this CCC activity, rightly so, as changed blocks and back them up in the incremental.

I actually think Windows dedup and block level backup don't mix all that well. Windows dedup just causes extra changed blocks due to how it works.
Post Reply

Who is online

Users browsing this forum: ante_704 and 203 guests