Best Practice for MS Server 2012 DeDup Repo

Availability for the Always-On Enterprise

Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby jfarr2008 » Mon Oct 29, 2012 6:57 pm

Are there any "best practices" when it comes to using MS Server 2012 as a backup target/repo with the new Server 2012 DeDup featured in the Veeam Blog post? Or do the same principles apply that would with a dedup appliance?
jfarr2008
Novice
 
Posts: 8
Liked: never
Joined: Thu May 06, 2010 7:04 pm
Location: Anniston, AL
Full Name: Jeremy Farr

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby tsightler » Mon Oct 29, 2012 7:25 pm 5 people like this post

Pretty much the same principles apply, however, it's important to note that the Windows 2012 dedupe is "post-process", this means you must always have enough space on the storage to contain at least one full, uncompressed backup. This is an important design consideration. Also, I believe that the default schedule for Windows dedupe process to run is at 10PM and run for 6 hours, which is likely to conflict with the backup window. Also, the Microsoft recommendation is to design around the fact that Windows dedupe will process ~100GB/hr for a single volume (you can scale to multiple volumes).

Basically, this just really hasn't been out long enough to develop a "best practices" around yet, however, here's the things to think about:

1. Remember that you will always need at least enough space to store a single, uncompressed full backup
2. If using sure backup and for faster restores, you may want to configure Windows 2012 dedupe to only dedupe files older than a given date (the default is 7 days I believe).
3. Schedule your dedupe window so that it runs when backups are not running.
4. Remember that Windows 2012 dedupe will process only about 100GB of new data per hour. This is probably not a big deal for incremental backups, but can be impactful for full backups.
5. You must use forward incremental, and disable Veeam compression to get the best results.

Because of the "post-process" requirement to have enough space to store a full, uncompressed backup, this will work against the feature for moderate term retentions (i.e. < 30 days) when compared to simply using Veeam compression/dedupe. This is especially for reverse incremental backups which are generally very space efficient anyway. For customers that are looking for longer term retention there can be significant benefits however.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby jfarr2008 » Mon Oct 29, 2012 7:46 pm

Thanks Tom. With regard to the schedule of the dedup job and the veeam backup window - any reason we couldn't kick off the dedup process with a post veeam job script ? Also, is there any reason NOT to use server 2012 dedup in conjunction with my local reverse incremental backups? Any space savings I can get are a plus.
jfarr2008
Novice
 
Posts: 8
Liked: never
Joined: Thu May 06, 2010 7:04 pm
Location: Anniston, AL
Full Name: Jeremy Farr

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby tsightler » Mon Oct 29, 2012 7:52 pm

I'm assuming you could schedule the dedup job with a post job script if you really wanted to.

First, assuming you are using Veeam compression with reverse incremental, you are unlikely to see much savings for your effort. First, unless you configure the Windows 2012 dedupe to process files that are "0 days" old, it will never actually dedupe the VBK since it's being changed every night, but if you do this, and happen to get good dedupe, you will likely see significant performance degradation of your backups. Dedupe is not recommended for files that see lots of I/O such as a VBK file in reverse incremental mode.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

[MERGED] Windows 2012 Dedup Issues...

Veeam Logoby mongie » Tue Nov 13, 2012 8:36 pm

I've seen everyone saying how well Windows 2012 dedup works with Veeam backups, and I've seen hints in my own environment, but I can't seem to get it just right.

I have two repository servers at the moment. Both Dell servers with direct-attached disk. One ~30TB and one ~60TB. Both servers have 48GB of ram, and aren't really doing much apart from running Veeam.

I find that both background processing jobs, and scheduled jobs take FOREVER. On the smaller server, I am seeing 3.5TB saved (13%) which is pretty good, but on the bigger server only ~400GB (0%).

I've also noticed that I'm getting an event log entry that says "Optimization job on volume D: (\\?\Volume{be0c51af-5496-4c14-8df8-bfd76532924f}\) was configured with insufficient memory." I believe Windows uses 25% of available ram for background processing, and 50% for manual jobs by default. Is there a way to increase this?

In cases where you have seen great reductions, do you notice dedup jobs running along fairly quickly? or do they take forever? How do you have your schedule set? How many days? for how many hours? (do you use 2 jobs?)

Also, what are the specs of your hardware? How much RAM?

Any suggestions would be appreciated.
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Tue Nov 13, 2012 8:37 pm 1 person likes this post

Oh, and also - Has anyone seen a difference between using Dedup Friendly compression in Veeam and No Compression?
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby jpeake » Tue Nov 13, 2012 9:28 pm

I just started using Win 2012 dedupe also. When I was running it in lab, with fewer machines in a backup job the results were great. Now in the real world the results are not. I think the problem is that I had all my servers in a single backup job, and the resulting files were too large for Win Dedupe engine to ever get through in it's scheduled window.

I have a group of Linux server, much smaller, and see about 80% savings there.

On my Windows server job, it showed 0% savings

So just yesterday I split my Windows machines up into 4 different jobs. Will see how that goes. Trying to keep each full backup file under 1TB. I know I will lose a little Veeam dedupe there

You can run manual jobs, using power shell, or just modify the scheduled tasks that are setup when you enable dedupe. See the cmdlets here:
http://technet.microsoft.com/en-us/libr ... 48450.aspx

My Veeam server has 8 X 2.5Ghz cores and 44GB of RAM. Storage is a Dell MD3600i. I was running dedupe as a backgound task (Windows marks this as low priority) and as a normal job from 12AM - 4PM each week day, and all day on Sunday. It will use up to 50% of RAM during those times.

The fastest rate I would see was about 100 GB/hr for dedupe processing. After a week, it still hadn't got through three 3TB+ files. So that really screws the stats. The % saved in the GUI is based on the size of the volume, even files that have not fully processed yet.

If you run:
Get-DedupeStatus | fl
you can see more accurate stats. Look at the "Optimized..." counters to see how well it is really performing

Resource monitor will show you that it's working also. Look under the "DISK" tab at the system and fsdmhost process. It will also show reads and writes to the ChuckStore folders
jpeake
Enthusiast
 
Posts: 88
Liked: 25 times
Joined: Tue Sep 25, 2012 7:57 pm

Re: Windows 2012 Dedup Issues...

Veeam Logoby jpeake » Tue Nov 13, 2012 9:33 pm

I tried dedupe friendly compression and saw about 10-15% compression rate from Veeam. But that was when all my Win Servers were in a massive job, so dedupe never finished.

Now that I have split to separate jobs, I turned compression off.
jpeake
Enthusiast
 
Posts: 88
Liked: 25 times
Joined: Tue Sep 25, 2012 7:57 pm

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Tue Nov 13, 2012 9:35 pm

Thanks for the reply...

How did you get the stat of 100GB/hr?

I've read up on most of the powershell commands, but that "fl" command is very helpful.
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby jpeake » Tue Nov 13, 2012 9:40 pm

that rate was just an average. My Linux servers job is 49GB for a full and it took about a half hour to finish processing each of them
jpeake
Enthusiast
 
Posts: 88
Liked: 25 times
Joined: Tue Sep 25, 2012 7:57 pm

Re: Windows 2012 Dedup Issues...

Veeam Logoby jpeake » Tue Nov 13, 2012 9:52 pm

dedupe just finished a pass on my Linux Servers folder.

OptimizedFileSize is 177GB
SavedSpace is 157GB
OptimzedFileSavingsRate is 88%

I have mine set to 5 days, so will take another week before I can tell how it works with smaller files on the Windows Servers jobs. And another week after that for it to really shine with another set of fulls. I am hoping for big things. It seems to be pretty solid, just maybe needs smaller (under 1TB) files. Or I just need to be more patient.

It's great that you can pretty much just set-it-and-forget-it though.
jpeake
Enthusiast
 
Posts: 88
Liked: 25 times
Joined: Tue Sep 25, 2012 7:57 pm

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Tue Nov 13, 2012 9:56 pm

It looks like it has finished at least one full pass of one of my servers, because all data has been "optimised", but the problem is, its only showing an optimised savings rate of 16% (on 27TB of data.)

If I run another job now, it will take hours to finish - I basically lose track of jobs because I can't tell when one finishes and one starts.
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby jpeake » Tue Nov 13, 2012 10:36 pm

that seems pretty low. Is that with compression off in Veeam? And LAN target? How many full backups are in that 27TB?
jpeake
Enthusiast
 
Posts: 88
Liked: 25 times
Joined: Tue Sep 25, 2012 7:57 pm

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Tue Nov 13, 2012 11:00 pm

23 in total, I think there are 2-3 for each backup job. Some are pretty big too (7.8TB).
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby jpeake » Tue Nov 13, 2012 11:52 pm

if you look at the file properties on one of those big files, what attributes does it show? I think if the dedupe process has finished, the file attributes will have PL, (sparse and sym link)

Just curious if those big monsters have actually been completed. I couldn;t find any docs from MS regarding this, or if there is a file size limit for dedupe.
jpeake
Enthusiast
 
Posts: 88
Liked: 25 times
Joined: Tue Sep 25, 2012 7:57 pm

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], jeremi, Majestic-12 [Bot], PaulJGamma, Yahoo [Bot] and 63 guests