Comprehensive data protection for all workloads
Post Reply
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Morning everyone. Curiosity, so in v9 I see our backup files have shrunk considerably - some by at least a good 50% - awesome! However, if it was writing zeroed blocks to storage - dedupe in our case - would the deduplication simply have been omitting them from final storage at the time? I'm wondering if once our v8 files expire/delete and we're down to all much smaller v9 files, will we suddenly see 50% or whatever less final consumption on our dedupes?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by tsightler »

rreed wrote:However, if it was writing zeroed blocks to storage...
I think the important thing to remember is that these wouldn't have been zero blocks. Veeam already skipped sending zero blocks in prior versions. Dirty blocks are blocks that contain data from a disk perspective, but simply are no longer used by the OS because,for example, the file has been deleted. Imagine the simple scenario where you copy a 4GB ISO file to a server and then immediately delete it. Prior to v9 that would be 4GB of data written to your dedupe appliance, now suddenly it will be zero bytes. While I wouldn't anticipate saving 50% of your space, especially on the dedupe appliance itself, I would exact to reduce the space some, but hard to predict how much since it's very dependent on whether the dirty blocks themselves were "dedupe-able" (is that really a word?).
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Crap, that's what I meant, sorry. Yes, dirty blocks from an OS perspective. That's one thing on our list here, is our Win2k12R2 servers are not zeroing out/returning space to the SAN (unmap) when we delete OS level files on Thin Provisioned VM's. What I think we were seeing in our environment, is say, a thin provisioned 200GB VM that sits on around 15GB or so just OS. VMware sees 200GB provisioned VM but only 15GB consumed, fine. Backup seems to write 15GB, fine. We copy 150GB worth of files, the VM is suddenly 165GB, shows up in VMware, and backs up accordingly, fine. We delete the 150GB files, VMware (and this should probably get moved to vSphere at this point) still sees it's a 165GB VM. Backup file also doesn't change in our v8, but I do need to go back and confirm this. It seems that in v9 the written backup files do reflect the actual OS in-use data size shrinking - awesome.

So I was curious if underneath it all if we would see a reflective savings on our dedupe devices but yes also good point - in v8 it likely would have written the dirty blocks the first time, then deduplicated them hence forth. So in v9, they will never get written in the first place?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed » 1 person likes this post

Glancing through, looks like our incrementals haven't changed much if any at all, which is probably expected, but some of our fulls have shrunk to about 1/4-1/3 of their v8 size! Wow. If this was raw storage, that would be an amazing reduction in file size. Well done, Veeam.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by dellock6 » 1 person likes this post

Incremental backups can be reduced by bitlooker if the dirty blocks are created in the time between the previous backup and the new incremental. Probably your dirty blocks were old ones, still there in the Guest VM, but already backed up in a previous full. Incremental backup would skip them not because of bitlooker but just because they were not changed after the full, and this also explains why another full would again save it.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Yep, it does appear to be circumstantial. Looking through further I see some that didn't change much (probably already "clean"), some that shrank by quite a bit (clearly we need to look at those VM's). In all, massive win, Veeam. Well done.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Another question, in job stats and disk file properties I've also noticed the dedupe numbers seem to have shot up quite a lot for each full backup run. Are we seeing bitlooker at work here in these numbers, or just better dedupe in v9?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by foggy »

This might be a result of a smaller block size, haven't you changed storage optimization settings?
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

An excellent question, but no. We were using 16TB+ in v8 and continue to do so in v9. But, didn't you guys change that block size underneath us?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by foggy »

Yes, we did, hence the question. Judging on another thread, your v9 is a completely new installation, right? Then newly created jobs will use 4MB block in case Local target (16TB+) is selected, instead of 8MB in v8. However, if you've imported v8 configuration, the block doesn't change for existing jobs until you do that manually.
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Fresh install. Ah, so that might help contribute to some of our smaller file sizes? And to set level expectations w/ my users and especially management, do I need to prepare them for longer restores due to the smaller block size pulling from dedupe devices?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by foggy »

Generally, yes, since that might require more IOPS (however, less data to read). So some types of restore can take longer.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by foggy »

Well, regarding the restore times, this was just my theory. Getting deeper into this, and in practice smaller block often gives performance increase, specifically for those types of restore that require random I/O (IR, FLR). Combined with other under-the-hood optimizations v9 has, this should in fact result in better restore times from dedupe devices.
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Very good. So, through v8's larger block size, and now v9's smaller block size (per 16TB+ setting), how much actual research and testing has been done? Results?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by foggy »

I'm not ready to share any actual tests results, however this decision definitely had some background behind it.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by tsightler »

rreed wrote:Very good. So, through v8's larger block size, and now v9's smaller block size (per 16TB+ setting), how much actual research and testing has been done? Results?
There has been tons of research and testing, and not just in the lab, but also during beta with customer environments side-by-side with v8. If you want me to go into gory technical details I can certainly do so, but so far my experience in the field has been nothing but major improvements for both backup and restore. I certainly wouldn't expect you to see any performance decrease. In mulitple cases I've seen double-digit increases in performance for file and item level restore operations on the order of 10-15x faster. The multiple streams from per-VM, also makes a huge performance increase. The only possible place where the smaller block size could impact performance is on things like full VM restores or tape-out, but 4MB blocks are still plenty big for this use case as well and I haven't seen a measurable impact on the performance of these operations, while all other operations are significantly faster.
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by rreed »

Very good, guys. Besides, not like there's anything I could do to go back to 8MB blocks being a fresh install anyways! :wink: Excellent work overall, I do need to do some test restores next week after our backups have been baking in a week or so here. I'll report back any perceived improvements w/ some of my same test restore jobs I used to run in v8. Many thanks as always, Veeam Team.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Bitlooker's Zeroed Blocks and Dedupe Devices

Post by tsightler »

Definitely looking forward to your results!
Post Reply

Who is online

Users browsing this forum: ilisousou123, xzvf and 190 guests