Difference in dedupe between Backup and Backup Copy jobs

VMware specific discussions

Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby wgrixti » Tue Jan 19, 2016 6:42 am

Hi,

I have hopefully a quick question.

I am using VBR 8 Patch 3. The backup jobs are run using Direct SAN mode and the repository is a Windows Server 2012 R2 with Dedupe enabled on the volume.
We are currently using Forever Forward as the backup methodology and have a retention period set to 2.
We then have a Backup Copy process that is a 1 to 1 of all the backup jobs. The backup copy goes over the LAN is stored on another repository running Windows Server 2012 R2 with Dedupe enabled on the volume too. We are keeping 30 restore points for the Backup Copy.

What I am seeing is massive dedupe savings on the Backup Copy Repository and significantly less on the primary backup repository.
Primary: 12% Dedupe Rate - Current number of restore points: 2 - Disk Used: 8TB
Backup Copy: 65% Dedupe Rate - Current number of restore points: 10 - Disk Used:4.4TB

Am I missing something in relation to the dedupe rates and disk space being used? Is this normal?
I have tried to follow all the best practice in relation to dedupe on Windows Server 2012 R2 to be found on the forums and other resources.

Sorry if this comes across as a silly question but if someone could give some guidance I would appreciate it.
Let me know if I need to provide more information.

Wayne
wgrixti
Influencer
 
Posts: 11
Liked: never
Joined: Fri Oct 30, 2015 1:02 am
Full Name: Wayne Grixti

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby foggy » Tue Jan 19, 2016 10:49 am

Wayne, what are the compression settings for both jobs? Do you have similar Data Deduplication settings on both repository servers? What is the 'Duplicate Files Older Than' setting for both?
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby wgrixti » Tue Jan 19, 2016 10:03 pm

Hi Alex,

Compression is turned off on both jobs to ensure better dedupe settings as per most recommendations I have found.
Veeam inline dedupe is enabled.
The Duplicate Files Older Than is se to 1 day on both repositories.
The dedupe settings for the servers/repositories are identical. Dedupe garbage collection is set to run daily.
wgrixti
Influencer
 
Posts: 11
Liked: never
Joined: Fri Oct 30, 2015 1:02 am
Full Name: Wayne Grixti

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby foggy » Wed Jan 20, 2016 12:42 pm

Could you please check the VBK size on both repositories?
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby wgrixti » Thu Jan 21, 2016 6:30 am

The VBKs are significantly smaller on the Backup Copy repository.
For example for one job on the main repository the size is 444GB and in the backup copy repository it is only 22GB.
I am struggling to understand the difference in size between the two?
wgrixti
Influencer
 
Posts: 11
Liked: never
Joined: Fri Oct 30, 2015 1:02 am
Full Name: Wayne Grixti

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby foggy » Thu Jan 21, 2016 10:35 am

Backup copy job does not copy files but synthetically creates restore points copying required blocks that correspond to the latest VM state from the source repository. Most likely backup file in your primary location contains a lot of dirty blocks and need to be compacted. With v8 this can be done with the help of an active full that will read data from the source storage and create brand new file. In v9 there's an ability to schedule compact operation.
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby wgrixti » Thu Jan 21, 2016 10:21 pm

Hi,

For your information the backup jobs have only be running for less than 2 weeks in the Forever Forward state, all previous backup data had been deleted as this is a relatively new B&R install, migrating away from Commvault. Do you think they would have that much in the way of dirty blocks? The size difference still seems quite extreme to me.

I am looking to upgrade to V9 next week at which point I can try the compact operation but I am not expecting the size of the data to be reduced so dramatically.
wgrixti
Influencer
 
Posts: 11
Liked: never
Joined: Fri Oct 30, 2015 1:02 am
Full Name: Wayne Grixti

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby foggy » Fri Jan 22, 2016 11:57 am

Were some VMs probably removed from the backup job during this time? Btw, how backup copy job is configured in terms of source VMs selection?
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby wgrixti » Fri Jan 22, 2016 11:35 pm

No VMs have been removed. This is a new environment.
We have 9 backup jobs, each job has its own backup copy configured.
wgrixti
Influencer
 
Posts: 11
Liked: never
Joined: Fri Oct 30, 2015 1:02 am
Full Name: Wayne Grixti

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby foggy » Mon Jan 25, 2016 9:28 am

What I meant was how VMs were added into the backup copy jobs: from backups, jobs, or infrastructure?
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby wgrixti » Mon Jan 25, 2016 10:22 pm

From jobs. Sorry for the misunderstanding.
wgrixti
Influencer
 
Posts: 11
Liked: never
Joined: Fri Oct 30, 2015 1:02 am
Full Name: Wayne Grixti

Re: Difference in dedupe between Backup and Backup Copy jobs

Veeam Logoby foggy » Tue Jan 26, 2016 2:34 pm

Re-reading the thread just noticed you're using forever incremental in the primary job. That means that the full backup is daily updated (merged with the latest increment file), so does not get deduplicated. While on the secondary storage you have higher retention that is not reached yet, so merge doesn't take place and full backup file is deduplicated. This explains difference in dedupe rates.
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to VMware vSphere



Who is online

Users browsing this forum: perrylee and 33 guests