Comprehensive data protection for all workloads
Post Reply
conradblack
Enthusiast
Posts: 41
Liked: 2 times
Joined: Mar 19, 2015 1:54 pm
Contact:

[MERGED] Using instant VM recovery from dedupe vol in Server

Post by conradblack »

Heya,

I'm looking to setup Veeam on a Windows 2012 Server with Windows Server 2012 deduplication on the backup repository volume. I'm curious about the guidelines/limitations when it comes to using instant VM recovery and SureBackup from a dedupe'd volume. Is this supported/recommended by Veeam? If so, are there any things that I would need to be aware of before setting it up?

Thanks
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Shestakov »

Hi,
Windows server 2012 deduplication is supported.
Please review the topic for to achive more information about enabling deduplication on the repository.
Thanks.
conradblack
Enthusiast
Posts: 41
Liked: 2 times
Joined: Mar 19, 2015 1:54 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by conradblack »

Just to confirm it is supported for both Instant VM Recovery and SureBackup?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Gostev »

Yes, however performance of those may not be acceptable for obvious reasons.
Generally, we do not recommend deduplication on a primary backup storage.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Shestakov » 1 person likes this post

Confirmed.
Note, that to improve performance, you can set OS to dedupe only files that are older than a specified interval.
For instance, if you want to make instant recovery from the last daily backup files, set to dedupe files older than 7 days only.
Thanks.
onthax
Service Provider
Posts: 40
Liked: 1 time
Joined: May 13, 2013 2:32 am
Location: Brisbane
Contact:

[MERGED] : Server 2012 R2 Deduplicated Backup Repository

Post by onthax »

Been following the changes with server 2012 R2 deduplication over the last year with interest and noted that microsoft have a stack of new best practices for deduplicated backup stores (for DPM)

https://technet.microsoft.com/en-us/lib ... 91438.aspx

Was wondering if there were any Veeam recommendations on wether to use "General Purpose file server" or "Hyper-V" as the -usage type when provisioning a new deduplicated volume?

Also wether to use "LAN Target" for 512KB blocks or "WAN Target" for 256KB Blocks.

MS recommendation is to format the volume with -AllocationUnitSize 64KB
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by veremin »

Kindly, search through the posts provided above for more information regarding Windows 2012 deduplicated repository. Thanks.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by lando_uk »

2012 Dedupe didn't worth for us with copy jobs, as once retention is met the vbk transforms caused a NTFS error and the jobs failed, causing the copies to increase and never get pruned.
We went through Dell and Veeam support with this and got nowhere, so now dedupe is disable for copy jobs.
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by namiko78 » 1 person likes this post

If it is complaining about file system limitations, you have to format the drive with the /L switch for large file size records. this is a MS recommendation. also keeping the drive defragmented helps avoid this issue.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by lando_uk »

Hi

This issue I has a year ago now (ticket 00533881) - maybe things have changed, I've not really kept track of the issue and just accepted the limitation.

I have no hope of formatting this volume as its full, but maybe we'll buy a new repository soon so I can try again with the /l option. But I'm not holding my breath that it fixes it, I just don't think it works with large (2TB+) vbk's.
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by namiko78 »

They say that files above 1TB are not good candidates for deduplication, but i've had some success with it, assuming the files are defragged properly. (formatting with /L may not be required, it apparently helps if fragmentation is bad) I'm only doing testing with large files over the past month or 2, and it's crapped out several times, but i started over with /L and a good defragmenter, and it's working now.

Going to give it a few months of telling the copy job to make it's monthly full's and see how things work. My uncompressed VBK is 5.1 TB, and it reports 74% deduplication ratio. Once i get a few more of these from the GFS settings, the savings should go way up.
onthax
Service Provider
Posts: 40
Liked: 1 time
Joined: May 13, 2013 2:32 am
Location: Brisbane
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by onthax »

v.Eremin wrote:Kindly, search through the posts provided above for more information regarding Windows 2012 deduplicated repository. Thanks.
Hi Eremin,

I read all of this thread before but it is looking at server 2012, where as there were a number of changes in windows dedup with 2012 R2 so i was after R2 specific information.

also this information requested isn't in the thread so i was hoping it would get attention, instead of in a 9 page thread on 2012.

Are there any veeam recommendations on -usagetype or allocation size?
Brad.Morris
Enthusiast
Posts: 43
Liked: 2 times
Joined: Sep 01, 2014 10:29 pm
Full Name: Brad Morris
Location: Melbourne, Australia
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Brad.Morris »

I have just built a Windows 2012 R2 Repository server and seeing great results on Windows Dedup. Seems to be a lot quicker doing the deduping. But I have to say I am not backing up to the repo yet. So far it just holds old backup copy data that I cannot mount back into B&R due to the bug.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by veremin »

So far it just holds old backup copy data that I cannot mount back into B&R due to the bug.
Does your backup copy job chain contain GFS restore point, and thus you can't map a job to it? If so, please, be aware that the issue is addressed in the upcoming update 2. Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Vitaliy S. »

onthax wrote:Are there any veeam recommendations on -usagetype or allocation size?
I don't believe we have any recommendations for this parameter, but let me check it with our QC team. If I have any reply on this, I will post back.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by lightsout » 16 people like this post

I'll give my feedback on 2012 dedup best practices.
  1. Format the disk using the command line "/L" for "large size file records".
  2. Also format using 64KB cluster size.
  3. Use Windows 2012 R2. Apply all patches as some rollups have improvements to dedup.
  4. Use Active full jobs with incrementals.
  5. Turn Veeam's compression off and use the "LAN" block size. Veeam's deduplication can stay on. This gave best overall space savings for me.
  6. If possible, spread your active full backups over the entire week. I have a script do it if you're interested.
  7. Modify the garbage collection schedule to run daily rather than weekly.
  8. Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted. I've had 4TB VBK process fine, it just take a long time!
  9. Use multiple volumes, where possible. Windows dedup is single threaded, but it can process multiple volumes at once. Although bigger volumes mean better dedup ratios!
  10. Configure your dedup process to run once a day, and for as long as possible.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by tsightler » 2 people like this post

The above recommendations from lightsout are excellent and line up perfectly with my own recommendations for Windows dedupe volumes. Windows dedupe can be quite effective, but you must be very careful that you follow the guidelines set forth above.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by jadams159 »

Lightsout, killer post, thanks!
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Mike Resseler »

Lightsout,

Killer recommendations. I have one comment on number 8. I fully agree with the 1TB specification but since 2012 R2 it is actually supported for 2TB :-). But out of experience, keep it with the 1TB is indeed the best recommendation

Thanks

Mike
alanbolte
Veteran
Posts: 635
Liked: 174 times
Joined: Jun 18, 2012 8:58 pm
Full Name: Alan Bolte
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by alanbolte »

Mike,

Do you have a source for the 2TB limit? I was never able to find one.
GlebGo
Lurker
Posts: 2
Liked: never
Joined: May 14, 2013 9:15 am
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by GlebGo »

Very useful for sure.

But I'm agree with Alan, it's a challenge to find the official statement from MS about 1TB/2TB limits. I will really appreciate if anybody could share the official source link.
GlebGo
Lurker
Posts: 2
Liked: never
Joined: May 14, 2013 9:15 am
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by GlebGo »

alanbolte
Veteran
Posts: 635
Liked: 174 times
Joined: Jun 18, 2012 8:58 pm
Full Name: Alan Bolte
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by alanbolte »

I've seen that before for the 1TB file size limit, I just haven't seen any statement about 2TB, or clarifying whether maximum supported file size depends on OS or whether you're using /L etc. Clearly you get best results by following the best practices in this thread, but official support statements are always nice.
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by push3r »

lightsout wrote:I'll give my feedback on 2012 dedup best practices.
  1. Format the disk using the command line "/L" for "large size file records".
  2. Also format using 64KB cluster size.
  3. Use Windows 2012 R2. Apply all patches as some rollups have improvements to dedup.
  4. Use Active full jobs with incrementals.
  5. Turn Veeam's compression off and use the "LAN" block size. Veeam's deduplication can stay on. This gave best overall space savings for me.
  6. If possible, spread your active full backups over the entire week. I have a script do it if you're interested.
  7. Modify the garbage collection schedule to run daily rather than weekly.
  8. Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted. I've had 4TB VBK process fine, it just take a long time!
  9. Use multiple volumes, where possible. Windows dedup is single threaded, but it can process multiple volumes at once. Although bigger volumes mean better dedup ratios!
  10. Configure your dedup process to run once a day, and for as long as possible.
Thank you very much for this best practice.

One question though, what's the logic for using the Incrementals with weekly Active Full? Wouldn't the Weekly Active Full be resource intensive and time consuming on both the production SAN/NAS and the Veeam repository? Why not just use Reverse Incremental? Also, why not Incrementals with Monthly Active Full?

Thank you.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by veremin »

Why not just use Reverse Incremental?
Because reversed incremental mode produces random I/O (not recommended for dedupe targets) in contrast to sequential write of active full/incremental.
Also, why not Incrementals with Monthly Active Full?
Even though incremental chain will be longer in that case, it might be also a way to go.

Thanks.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by lightsout »

What he said. :)

Monthly fulls would work well, but it is just a long chain which if corrupted can lose you a lot of backups.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by Shestakov »

Push3r,
The reason why reversed incremental backup method is marked as "slower" one is explained here.
And as for me, you are correct, monthly full is enough in lots of cases. The weakness of longer chains is a bit longer RTO rather than lower reliability. Usually 30-points long chain is pretty reliable, especially if you use Surebackup job.
Thanks.
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by push3r »

Thank you guys for the reply.

I get it now with regards to Reverse Incremental having random IO and not good for DeDup. I will use regular incremental instead.

As for the best practice recommendation of Weekly Full, I think I can live with Monthly Full because weekly would requires 4 Full Backup files and the IOPs strain on the production SAN/NAS.

From my testings using this best practice, Windows 2012 R2 DeDup is reporting around 50% DeDup Rate. Really awesome technologies from Microsoft and Veeam with minimal cost and pain.
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by push3r »

On a slightly different note, what settings are optimal for pushing Backup Copy Jobs to DeDup Target? I am pushing these to a Co-Location Site with 100Mbps WAN Site-to-Site VPN.

From my testing, I have these Veeam Settings:

For Target:
- "Enable Inline data dedup" box: Checked
- Compression Level: Dedup-friendly

Should I use "None" for Backup Copy Job too?

For Data Transfer:
- Through built-in WAN accelerators.

I am wondering, with 100Mbps WAN connection, Direct would be better? But Bandwidth saving is not as good as through WAN Accelerators? I don't have any Bandwidth Quotas limitation; so, it seems Direct would be better?

With those test settings, I am seeing around 55% DeDup rate. Just wandering if I can even improve on this.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: Best Practice for MS Server 2012 DeDup Repo

Post by lightsout »

I turned off compression for my backup copy jobs too, as that goes to a HW dedup device.
Post Reply

Who is online

Users browsing this forum: ante_704 and 226 guests