Best Practice for MS Server 2012 DeDup Repo

Availability for the Always-On Enterprise

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby foggy » Tue Jun 04, 2013 3:55 pm

baatch wrote:On the compression level, is there any reason not to do Optimal or Extreme instead of dedupe friendly for Server 2012 dedup?

The only reason I guess is worse dedupe ratio you will get with these levels comparing to dedupe-friendly.
foggy
Veeam Software
 
Posts: 15272
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby tsightler » Tue Jun 04, 2013 9:38 pm

You will get very little dedupe if you perform full compression level. "Dedupe friendly" compression is a very simple compression algorithm that reduces the data set size using a very simple algorithm this can still benefit at least partially from dedupe, however, advanced compression algorithms typically reduce dedupe down to near nothing unless you are storing the same compressed pattern multiple times.
tsightler
Veeam Software
 
Posts: 4872
Liked: 1819 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby norelco » Wed Jun 05, 2013 6:15 pm

So... if we're best off using forward incremental then we're not having to do the disk IO associated with reverse incremental backups. It's recommended to use RAID10 arrays for backup repository (presumably so the disk IO is there to do reverse incrementals). If we're not going to use reverse incrementals, then would a RAID 5 array be considered acceptable?

n
norelco
Influencer
 
Posts: 14
Liked: never
Joined: Fri Feb 08, 2013 9:42 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby foggy » Thu Jun 06, 2013 3:20 pm

norelco wrote:If we're not going to use reverse incrementals, then would a RAID 5 array be considered acceptable?

A couple of topics FYI regarding RAID recommendations, might be helpful:
RAID6 or RAID10 for backup storage on local disk array?
NAS RAID level
foggy
Veeam Software
 
Posts: 15272
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby haribos » Thu Aug 01, 2013 8:54 am

I'm testing with Server 2012 Dedup, the only 'issue' i experience is that the Synthetic Fulls takes extreme long time, because the required information is from the dedup data this is slower. What is the best whay staying on Synthetic Full or Active Full backup?
haribos
Novice
 
Posts: 3
Liked: never
Joined: Wed Jul 31, 2013 8:15 am

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Vitaliy S. » Thu Aug 01, 2013 10:24 am

If you can run your production VMs on a snapshot for the extended period of time, then I would suggest active full, in all other cases synthetic full should be a preferable choice because of the no impact on production storage/VMs.
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby depps » Tue Aug 13, 2013 4:29 pm

Has anyone used Server 2012 de-dupe attached to a QNAP?

I'm thinking of getting a 64TB QNAP and attaching it via iscsi or nfs to a physical or virtual machine and using this to store 5 years of monthly's and bi-weekly sql backups.
depps
Influencer
 
Posts: 20
Liked: never
Joined: Mon Jan 24, 2011 10:16 pm
Full Name: Daniel Epps

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby depps » Tue Aug 13, 2013 5:31 pm

Possibly looking at a Dell NX3200 or HP 1630 running Storage Server 2012 as well.
depps
Influencer
 
Posts: 20
Liked: never
Joined: Mon Jan 24, 2011 10:16 pm
Full Name: Daniel Epps

[MERGED] Windows 2012 dedupe design best practices

Veeam Logoby luckyinfil » Tue Oct 01, 2013 3:13 pm

I've been using windows 2012 dedupe along with veeam to backup our VMs, but what I'm noticing is that the dedupe fails to keep up with the data creation. For example, whenever my synthetics are created, there could be up to 5 TB of data created across a couple of volumes that day. Doing some research, it has been mentioned that dedupe only works at 100GB /hour MAX speed. Is there anyway to increase that rate provided we have more than sufficient resources?

What is the best practices to designing for large synthetic backups? (considering that best practices for local repositories are a maximum of 8TB for files)
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby v.Eremin » Tue Oct 01, 2013 3:46 pm

Doing some research, it has been mentioned that dedupe only works at 100GB /hour MAX speed. Is there anyway to increase that rate provided we have more than sufficient resources?

Hi, John.

As far as I know, this limitation exists on a volume level. Windows is able to deduplicate a single volume at a time and can process roughly about 100 GB of data per hour, or 2 terabytes (TB) per day. Even though Windows server with additional CPU and memory resources is able to deduplicate multiple volumes at the same time, the speed limitation of single deduplicated volume can’t be currently bypassed.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13701
Liked: 1020 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby luckyinfil » Tue Oct 01, 2013 3:51 pm

Do you have a source for that? Based on my observations, I see that there are multiple threads deduping on different volumes (i see multiple fsdmhost.exe processes working on different files in 2 different volumes in resource monitor).
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby v.Eremin » Tue Oct 01, 2013 3:54 pm

Yep, sure:

Data can be optimized at 20-35 MB/Sec within a single job, which comes out to about 100GB/hour for a single 2TB volume using a single CPU core and 1GB of free RAM. Multiple volumes can be processed in parallel if additional CPU, memory and disk resources are available.
v.Eremin
Veeam Software
 
Posts: 13701
Liked: 1020 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby lxzndr » Wed Oct 02, 2013 6:57 pm 2 people like this post

I've been using 2012 dedup for about 6 months now (also using windows storage spaces).
relatively small shop. 24 VMs being backed up in veeam, 14 daily, 10 once a week. (Veeam 6.5, VMware 5.1)
While for many of you this is a tiny shop, for some (even large shops) it may provide some insight, particularly when trying to both keep a lot of restore points, and deal with the 2012 ingest rate issue. Do your backups to a non-dedup volume, and then copy them to the dedup volume for archival. you can continue your forward/reverse incrementals at normal speed, and get your longer term storage at the same time. You do lose the immediate access to the archived jobs from the Veeam GUI, but they are really there for infrequent use anyway. think of them as your old Tape copies. With this process you could even do a G/F/S process in dedup.

Here is my process:

All Veeam backups go to a non-dedup enabled volume on the server. I basically have one job per VM for backup, except for weekend, then the VMs that have minimally changed data are in one job.
I do a full once per week, which ones are done rotate. so VM 1+2 on Monday, 3+4 on tuesday, etc. that splits the volume down for the dedup.
Each day the jobs are finished by midnight. At 1am I kick off a script that does a full robocopy of the non-dedup volume to a external 4TB NAS device. When that finishes, it does a robocopy of the VBK files to the dedup volume, when that finishes it fires off a manual dedup job. (deduce file age=0 - immediate deduplication, disabled built in dedup schedule)
I get an e-mail after each robocopy, and then configured triggered task to e-mail dedup status after dedup jobs complete.

I have also created a script that will remove the archived jobs after they exceed a specified retention period. (haven't removed any yet)

I end up with 1 to 2 weeks (depending on VM) of live backup files - full + forward incremental (duplicated to external NAS) VBK sizes from 3GB up to 280GB. (~1.9TB of VBK files)
Then I have (currently) 6 months of VBK files in the dedup volume. I have expanded the capacity about 3 times in the 6 months. started at 3TB volume. FreeSpace fluctuates a bit. I only need about 400GB free due to staggering the full backups through the week.

Volume : E:
VolumeId : \\?\Volume
Capacity : 6.5 TB
FreeSpace : 607.24 GB
UsedSpace : 5.91 TB
UnoptimizedSize : 53.49 TB
SavedSpace : 47.58 TB
SavingsRate : 88 %
OptimizedFilesCount : 757
OptimizedFilesSize : 53.42 TB
OptimizedFilesSavingsRate : 89 %
InPolicyFilesCount : 757
InPolicyFilesSize : 53.42 TB
lxzndr
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri Jun 24, 2011 3:26 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby v.Eremin » Thu Oct 03, 2013 11:18 am

@lxzndr

Have you considered an update to the latest product version (7.0) that has a new type of job called “backup copy job”?

I’m asking, since your deployment seems like a perfect use case of backup copy job. Not only will it handle for you the process of copying backup data to deduplicated volume, but also it will deal with retention period and GFS rotation scheme, as well.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13701
Liked: 1020 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby VladV » Thu Oct 03, 2013 1:14 pm

We have a similar setup as lxzndr and was hoping that Backup copy will help us discard the scripting and robocopying solution.

Because of how Backup Copy is designed, you will always have the transformation processes which modify the VBK because of the minimum of 2 retention points and the weekly archives. This, in comparison to having only weekly VBKs on the dedupe volumes, has a lower dedupe performance . If Backup copy would have permitted the use of weekly backups (or other GFS periods) without the use of a permanent VBK/VIB combo it would have been ideal.
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: DaveWatkins, Google Feedfetcher, sg1 and 5 guests