Difference in dedupe between Backup and Backup Copy jobs

[wgrixti]

Hi,

I have hopefully a quick question.

I am using VBR 8 Patch 3. The backup jobs are run using Direct SAN mode and the repository is a Windows Server 2012 R2 with Dedupe enabled on the volume.
We are currently using Forever Forward as the backup methodology and have a retention period set to 2.
We then have a Backup Copy process that is a 1 to 1 of all the backup jobs. The backup copy goes over the LAN is stored on another repository running Windows Server 2012 R2 with Dedupe enabled on the volume too. We are keeping 30 restore points for the Backup Copy.

What I am seeing is massive dedupe savings on the Backup Copy Repository and significantly less on the primary backup repository.
Primary: 12% Dedupe Rate - Current number of restore points: 2 - Disk Used: 8TB
Backup Copy: 65% Dedupe Rate - Current number of restore points: 10 - Disk Used:4.4TB

Am I missing something in relation to the dedupe rates and disk space being used? Is this normal?
I have tried to follow all the best practice in relation to dedupe on Windows Server 2012 R2 to be found on the forums and other resources.

Sorry if this comes across as a silly question but if someone could give some guidance I would appreciate it.
Let me know if I need to provide more information.

Wayne

[foggy]

Wayne, what are the compression settings for both jobs? Do you have similar Data Deduplication settings on both repository servers? What is the 'Duplicate Files Older Than' setting for both?

[wgrixti]

Hi Alex,

Compression is turned off on both jobs to ensure better dedupe settings as per most recommendations I have found.
Veeam inline dedupe is enabled.
The Duplicate Files Older Than is se to 1 day on both repositories.
The dedupe settings for the servers/repositories are identical. Dedupe garbage collection is set to run daily.

[foggy]

Could you please check the VBK size on both repositories?

[wgrixti]

The VBKs are significantly smaller on the Backup Copy repository.
For example for one job on the main repository the size is 444GB and in the backup copy repository it is only 22GB.
I am struggling to understand the difference in size between the two?

[foggy]

Backup copy job does not copy files but synthetically creates restore points copying required blocks that correspond to the latest VM state from the source repository. Most likely backup file in your primary location contains a lot of dirty blocks and need to be compacted. With v8 this can be done with the help of an active full that will read data from the source storage and create brand new file. In v9 there's an ability to schedule compact operation.

[wgrixti]

Hi,

For your information the backup jobs have only be running for less than 2 weeks in the Forever Forward state, all previous backup data had been deleted as this is a relatively new B&R install, migrating away from Commvault. Do you think they would have that much in the way of dirty blocks? The size difference still seems quite extreme to me.

I am looking to upgrade to V9 next week at which point I can try the compact operation but I am not expecting the size of the data to be reduced so dramatically.

[foggy]

Were some VMs probably removed from the backup job during this time? Btw, how backup copy job is configured in terms of source VMs selection?

[wgrixti]

No VMs have been removed. This is a new environment.
We have 9 backup jobs, each job has its own backup copy configured.

[foggy]

What I meant was how VMs were added into the backup copy jobs: from backups, jobs, or infrastructure?

[wgrixti]

From jobs. Sorry for the misunderstanding.

[foggy]

Re-reading the thread just noticed you're using forever incremental in the primary job. That means that the full backup is daily updated (merged with the latest increment file), so does not get deduplicated. While on the secondary storage you have higher retention that is not reached yet, so merge doesn't take place and full backup file is deduplicated. This explains difference in dedupe rates.