Best Practice for MS Server 2012 DeDup Repo

Availability for the Always-On Enterprise

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby v.Eremin » Tue May 03, 2016 1:23 pm

Doesn't your tape device itself provide compression? Thanks.
v.Eremin
Veeam Software
 
Posts: 13734
Liked: 1027 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby lightsout » Tue May 03, 2016 1:24 pm

I do the same thing, so I just made sure hardware compression was turned on for the tape jobs.
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby HussainMahfood » Tue May 03, 2016 1:26 pm

@lightsout

I do the same but veeam compression is better. I would stick with hardware compression till veeam provide a feature for this.

@Vladimir Eremin
Thanks I do have it enabled and was thinking to utilize veeam compression instead
HussainMahfood
Enthusiast
 
Posts: 35
Liked: 7 times
Joined: Mon Jun 24, 2013 9:43 am
Full Name: Hussain Mahfood

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby v.Eremin » Tue May 03, 2016 1:33 pm

I think by disabling software compression in source job and enabling hardware one in secondary (tape) job you would get both the best deduplication ratio in primary repository and decent compression on tapes.

So, to me it sounds like the best approach.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13734
Liked: 1027 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

[MERGED]: Tape & Windows 2012 R2 Dedupe

Veeam Logoby dalbjerg » Tue May 10, 2016 9:30 am

Hi

we have a dedicated windows 2012 R2 server, with veeam install on it.
It have a 40GB diskdrive to temporally store Backup on, and then it move the backups to Tape.

Could we enable dedupe on the 40GB disk drive, to save some space. Or will this brake the tape backups?

Regards Kenneth Dalbjerg
dalbjerg
Lurker
 
Posts: 1
Liked: never
Joined: Mon Feb 29, 2016 7:08 pm
Full Name: Kenneth Dalbjerg

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby PTide » Tue May 10, 2016 10:08 am

Hi,

Could we enable dedupe on the 40GB disk drive, to save some space. Or will this brake the tape backups?
You can do that, it won't harm, however it may slowdown the backup process. You should configure the jobs for active full backups plus incremental backups - since jobs with transformation will require block "de-hydration" and then "re-hydration" on the storage. These operations might require significant time. The same applies to Backup to Tape job - prior to writing to tape your backups will have to be "re-hydrated" first. Please review this thread for more details in regards to Win2012R2 deduplication.

Thank you.
PTide
Veeam Software
 
Posts: 3252
Liked: 273 times
Joined: Tue May 19, 2015 1:46 pm

Re: [MERGED]: Tape & Windows 2012 R2 Dedupe

Veeam Logoby nunciate » Tue May 10, 2016 12:44 pm

dalbjerg wrote:we have a dedicated windows 2012 R2 server, with veeam install on it.
It have a 40GB diskdrive to temporally store Backup on, and then it move the backups to Tape.

Could we enable dedupe on the 40GB disk drive, to save some space. Or will this brake the tape backups?


You will be fine with this. We have a scale-out repository with 10 very large extents assigned to it adding up to over 100Tb of space.
I have Windows 2012 Dedupe running on all of my extents and have no problems.

Remember Dedupe works best when you are running active full backups. I set my volume deduplication settings to dedupe files older than 2 days. That way they get written to disk and then to tape before the get deduped out. Doing it this way you will have no disk resource issues.

I will give you an example. I have a single drive with 18Tb of space. I have 3.14Tb of Free Space on the drive currently but I am storing 31.9Tb worth of data on that drive. I have deduped out 52% of that used space. Not Bad!
nunciate
Expert
 
Posts: 140
Liked: 23 times
Joined: Tue May 21, 2013 9:08 pm
Full Name: Alan Wells

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby rnt-guy » Tue May 24, 2016 6:02 pm

Hi everyone. been going through these recommendations and I wonder if our situation is slightly different. We have several customers where we backup their VMs to our own Veeam host that has 21TB of usable space after raid, hotspare, etc, before compression & deduplication. After we back up their data to our host we send that data to a Veeam cloud host. They use their own storage and don't use Windows Server 2012.

Our goals are this:

1. Meet customer backup windows. Restore windows are reasonable and not typically a sticking point.
2. Use as little space as possible on our veeam host on prem at customer since we pay for this unit out of pocket and adding more space is not free.
3. use as little space as possible with the hosting provider. we get charged and thus the customer gets charged. the less expensive it is the more likely the customer will choose our backup solution.

As of now we're doing incrementals with a weekly synthetic to help with the backup window issue. We don't schedule active fulls.

We have deduplication running on our repository server on our host. we have a wan accelerator as does the cloud host which helps reduce the copy job time.

because cost is usually the first sticking point we're trying to figure out how to keep the size of the backups at the cloud provider as small as possible. unfortunately we have to assume they'd be using cheap disk that has no compression/deduplication built in so it's up to us to keep the files small.

The backup jobs are set to dedupe-friendly compression witht local target for storage optimization. The copy job is set to high or extreme compression.

we have win2012r2 deduplication set to 0 days so that it dedupes it right away keeping our size on disk low. if we have a big enough buffer I could see us letting it go 2 days so that the copy job isn't recompacting the data on the way out. However the copy jobs start within minutes or hours of the backup jobs finishing and this is at night whereas the deduplication doesn't start until 10am so it's pretty unlikely it would be unpacking deduped data just to copy offsite.

hopefully this dump of info for you wasn't too much but easy to navigate. My apologies if not.

What I'm wondering is

a. what is the combination that results in the smallest files at the cloud provider assuming they don't do their own compression & deduplication?
b. what speeds up the backups the best given our goals?
c. will meeting a&b necessitate using more space on our onsite host?

* side note: our host is dedicated to veeam with internal storage so there's plenty of cpu and memory. disk is the limited space without buying another set of repository space.
rnt-guy
Service Provider
 
Posts: 61
Liked: 1 time
Joined: Thu Feb 04, 2016 12:58 pm
Full Name: RNT-Guy

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby albertwt » Tue May 31, 2016 7:20 am 1 person likes this post

lightsout wrote:I'll give my feedback on 2012 dedup best practices.

  1. Format the disk using the command line "/L" for "large size file records".
  2. Also format using 64KB cluster size.
  3. Use Windows 2012 R2. Apply all patches as some rollups have improvements to dedup.
  4. Use Active full jobs with incrementals.
  5. Turn Veeam's compression off and use the "LAN" block size. Veeam's deduplication can stay on. This gave best overall space savings for me.
  6. If possible, spread your active full backups over the entire week. I have a script do it if you're interested.
  7. Modify the garbage collection schedule to run daily rather than weekly.
  8. Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted. I've had 4TB VBK process fine, it just take a long time!
  9. Use multiple volumes, where possible. Windows dedup is single threaded, but it can process multiple volumes at once. Although bigger volumes mean better dedup ratios!
  10. Configure your dedup process to run once a day, and for as long as possible.


Cool, thansk for sharing this tips in concise format.
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 621
Liked: 20 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby foggy » Tue May 31, 2016 5:22 pm 1 person likes this post

@rnt-guy
a. Overall, your setup looks good in terms of meeting your goals. One suggestion could be to use the 'WAN target' setting that will allow to decrease the backup files size due to the smaller block size it uses.
b. Depends on the job bottleneck stats.
foggy
Veeam Software
 
Posts: 15303
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby albertwt » Wed Aug 31, 2016 10:43 pm

Hi LightsOut,

[*]If possible, spread your active full backups over the entire week. I have a script do it if you're interested.

I'm interested of the script, may I know what is that script for ?

[*]Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted. I've had 4TB VBK process fine, it just take a long time!


I've got some of my Exchange Servers & SQL Servers backup to be larger than 8 TB .VBK, so do I just skip this backup volume ?

[*]Configure your dedup process to run once a day, and for as long as possible.

If the backup copy job is running continuously does it still possible to run the deduplication ?
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 621
Liked: 20 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby albertwt » Wed Aug 31, 2016 10:45 pm

foggy wrote:@rnt-guy
a. Overall, your setup looks good in terms of meeting your goals. One suggestion could be to use the 'WAN target' setting that will allow to decrease the backup files size due to the smaller block size it uses.
b. Depends on the job bottleneck stats.


Hi Alex,

Does it means the smaller than or equal WAN target SAN Stripe / Block size means the backup size can also be smaller in size ?
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 621
Liked: 20 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby foggy » Thu Sep 01, 2016 1:55 pm 1 person likes this post

This relates to the fact that with smaller block size you would not need to copy, say, entire 1MB block if only 1KB has changed in it, but 256KB only (if you switch to WAN target).
foggy
Veeam Software
 
Posts: 15303
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby lightsout » Mon Sep 12, 2016 12:22 am 1 person likes this post

albertwt wrote:Hi LightsOut,

I'm interested of the script, may I know what is that script for ?

I've got some of my Exchange Servers & SQL Servers backup to be larger than 8 TB .VBK, so do I just skip this backup volume ?

If the backup copy job is running continuously does it still possible to run the deduplication ?


    1. So here is my code. This will cycle the full backups on a day of a week.
    Code: Select all
    $days=@("Friday", "Saturday", "Sunday", "Monday", "Tuesday", "Wednesday", "Thursday")

    $i=0
    $jobs=get-vbrjob | ? {$_.JobType -eq "Backup" }
    foreach ($job in $jobs) {
      $job.Name

      $job | Set-VBRJobAdvancedBackupOptions -EnableFullBackup $true -FullBackupDays $days[$i] -FullBackupScheduleKind Daily -DayOfWeek $days[$i] | out-null
      $i++
      if ($i -ge $days.count) { $i = 0 }
    }

    2. I'd suggest skipping those, until Windows 2016. They will take a long time to dedup.
    3. Yes, you can run both together, just make sure you there is enough I/O for the dedup process to run and your jobs to complete in their windows.
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby albertwt » Mon Sep 12, 2016 12:39 am

Cool, many thanks for the clarification. :D
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 621
Liked: 20 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], Yahoo [Bot] and 1 guest