Backup copies taking too long, looking for a better strategy?

[preecyp]

We have a head office and a branch/DR office. The two sites are connected to one another with a 100mbit link. At each site we have a a physical Windows 2012 R2 server which is dedicated for backups (Veeam B&R 9.5 Enterprise Plus installed on both). These servers are also used as storage repositories and we've enabled Windows dedupe. As well as the Veeam servers we've a couple of vSphere ESXi 6.0 hosts at each site. The majority of our production vm's at situated at our head office.

On our head office Veeam server we have a backup job which occurs daily (incremental) with full active jobs running each Friday. The backup is stored locally on the same server and restore points are retained for 28 days. The size of the protected vm's is roughly 3.5tb and a full backup usually takes 7 hours, incremental take less than 2 hrs. We have enabled copy backups of this job to the Veeam server at our DR site.

The issue we are facing is the full copy backup job is not completing within the 24 hour window. Only half the vm's complete and informed the Copy interval has expired. Initial copy did not complete, and will resume on the next copy interval'. The problem is, the copy job does not resume, instead it starts processing the local incremental job instead. This results in the majority of our offsite backups showing as 'incomplete'.

Am i better off disabling the copy backup and defining a new backup job with the DR Veeam server set as the repository?

Any tips or suggestions on how i can trim the offsite backups? I can disable Windows dedupe on the DR server if this will allow us to make better use of compression, etc?

I have logged a case with tech support case: 03349141 but any advise in the mean time would be appreciated.

[PTide]

Hi,

I wonder what are your bottleneck stats for the backup copy job, whether you use WAN accelerators, and how much data does the copy job usually manage to transfer by the time 24 hour window ends?

Thanks!

[Andreas Neufert]

Please change the Job run from 1 day to multiple days (maybe 99) and allow backup copy job to complete the initial run. Then change it back to daily processing or the value that you want to use.

This should fix your issue.

[preecyp]

We originally enabled the WAN accelerators, however, tech support (a previous case/issue) advised the accelerators would not be effective in our environment. I believe this was down to the size of our backups and the speed of our link. This was a while ago but the reasons was 'accelerators are only effective on slower links'. I could have misunderstood and we can re-enable the accelerators if it will help?

I can temporarily change the job to 99 days to give the initial full active backup time to complete. However, once i change it back to daily, the copy backup will fail to complete the following Friday as we currently perform full backups each Friday.

If i reconfigure the copy backup to perform full synthetics, rather than active fulls, would this reduce the data/time required? If so, i understand the use of dedupe is not recommended when using synthetic full backups?

I have a case open and provided tech support the support logs this morning so they can identify the bottleneck.

Thanks for the advise so far.

[PTide]

I could have misunderstood and we can re-enable the accelerators if it will help?

No, the support engineer is spot on - WAN accelerators won't help in this case. It was me who overlooked another "zero" so I thought it was 10Mbit link, sorry for confusion.

If i reconfigure the copy backup to perform full synthetics, rather than active fulls, would this reduce the data/time required?

If you are talking about switching the source backup job to synthetic, then it won't make much difference since BCJ transfers only new/changed blocks from the source job. If you meant "switching BCJ from active fulls to synthetic" then please elaborate a little bit, as BCJ has only "active full" option, not synthetic full".

If so, i understand the use of dedupe is not recommended when using synthetic full backups?

Considering the data that you've provided, yes. Since you use Windows deduplication, and your backup file is clearly way over 1Tb in size, this article is worth checking.

Also it is not quite clear whether the backup copy job fails to synchronize every Friday, or is it just the first run that cannot complete

P.S. Bottleneck stats can be found in the job session.

Thanks!

[preecyp]

Ah ok, i hadn't actually checked the BCJ settings. I assumed synthetic full was available within the BCJ options too.

Our .vbk files are around 2.5tb in size whereas the incremental .vib's are only 200gb. So going forward i guess i could disable Windows dedupe and re-enable the native dedupe & compression settings within Veeam?

I've checked the BCJ stats and its reporting 'network' as the bottleneck. We;re getting a processing rate of around 11MB/s.

[PTide]

Since the bottleneck is the network, not the target, I'd suggest to leave Windows deduplication in place and change the job to 99 days, as Andreas's proposed, in order to have initial backup copy sync completed. After that, let BCJ to run incremental session and see if the bottleneck is still the network. As I've mentioned, BCJ should normally transfer only amounts of data that is more or less equal to your average increment.

Thanks!

[preecyp]

Ok thanks i will give that a go, however, am i going to be back in the same situation again next friday when the active backup run for a second time?

Case # 03349141

[PTide]

No, even if the source backup job generates active full, BCJ is supposed to transfer only changed blocks since the last run. Looking forward for the results from you.

Thanks!

[spookydoor]

I have two copy jobs that will not complete. They either fail or finish with the warning: Initial copy did not complete, and will resume on the next copy interval. Some VMS were not processed during the copy interval.

I cannot get these jobs to complete. They have been set at a 24 hour interval. I have tried a 3 hour interval and a 10 minute interval, but nothing works.

These jobs copy data from multiple backup jobs. One backup job is daily, one is weekly, and two are set to run every 2 hours.

We recently had a RAID array fail and had to get that fixed and put a loaner server in place. Things continued to function as normal with the loaner. These two copy jobs are both pointing to external drives that get rotated out. I have turned on the setting in the job to indicate rotating drives.

We have had maybe 1 successful run of this job in the past 4 or 5 days.

[Natalia Lupacheva]

Hi Bryan,

Thank you for the detailed description of your issue and the provided case id!
I've moved your post to the existing thread with the similar problem, please take a look at the discussions above.
Might help you to find a bottleneck.

Thanks!

[spookydoor]

I set the interval on both copy jobs to 99 days, earlier this morning. I started a manual sync on both jobs and they just stopped at 21% and entered an idle: waiting on new restore points status. I kicked off a manual backup of one of the jobs and I was going to kick off a manual back up of the weekly job, afterward, to see if that would get the copy jobs to finish, but before I could kick off the manual backup for the weekly job, both copy jobs failed. They look like they're starting over on their own. They were at 0% when I checked them 2 minutes ago.

[foggy]

Looks like it is the fact that the source backup jobs have different cadence ("One backup job is daily, one is weekly, and two are set to run every 2 hours.") what confuses you while observing the backup copy behavior. Anyway, please continue working with our support engineers - they will be able to review your setup and confirm if what's happening is expected or not. Thanks!