Discussions specific to the VMware vSphere hypervisor
Post Reply
preecyp
Novice
Posts: 6
Liked: never
Joined: Jan 07, 2019 5:18 pm
Full Name: phil preece
Contact:

Backup copies taking too long, looking for a better strategy?

Post by preecyp » Jan 07, 2019 5:55 pm

We have a head office and a branch/DR office. The two sites are connected to one another with a 100mbit link. At each site we have a a physical Windows 2012 R2 server which is dedicated for backups (Veeam B&R 9.5 Enterprise Plus installed on both). These servers are also used as storage repositories and we've enabled Windows dedupe. As well as the Veeam servers we've a couple of vSphere ESXi 6.0 hosts at each site. The majority of our production vm's at situated at our head office.

On our head office Veeam server we have a backup job which occurs daily (incremental) with full active jobs running each Friday. The backup is stored locally on the same server and restore points are retained for 28 days. The size of the protected vm's is roughly 3.5tb and a full backup usually takes 7 hours, incremental take less than 2 hrs. We have enabled copy backups of this job to the Veeam server at our DR site.

The issue we are facing is the full copy backup job is not completing within the 24 hour window. Only half the vm's complete and informed the Copy interval has expired. Initial copy did not complete, and will resume on the next copy interval'. The problem is, the copy job does not resume, instead it starts processing the local incremental job instead. This results in the majority of our offsite backups showing as 'incomplete'.

Am i better off disabling the copy backup and defining a new backup job with the DR Veeam server set as the repository?

Any tips or suggestions on how i can trim the offsite backups? I can disable Windows dedupe on the DR server if this will allow us to make better use of compression, etc?

I have logged a case with tech support case: 03349141 but any advise in the mean time would be appreciated.

P.Tide
Product Manager
Posts: 5264
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by P.Tide » Jan 07, 2019 6:49 pm

Hi,

I wonder what are your bottleneck stats for the backup copy job, whether you use WAN accelerators, and how much data does the copy job usually manage to transfer by the time 24 hour window ends?

Thanks!

Andreas Neufert
Veeam Software
Posts: 3821
Liked: 687 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by Andreas Neufert » Jan 08, 2019 9:30 am

Please change the Job run from 1 day to multiple days (maybe 99) and allow backup copy job to complete the initial run. Then change it back to daily processing or the value that you want to use.

This should fix your issue.

preecyp
Novice
Posts: 6
Liked: never
Joined: Jan 07, 2019 5:18 pm
Full Name: phil preece
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by preecyp » Jan 08, 2019 11:31 am

We originally enabled the WAN accelerators, however, tech support (a previous case/issue) advised the accelerators would not be effective in our environment. I believe this was down to the size of our backups and the speed of our link. This was a while ago but the reasons was 'accelerators are only effective on slower links'. I could have misunderstood and we can re-enable the accelerators if it will help?

I can temporarily change the job to 99 days to give the initial full active backup time to complete. However, once i change it back to daily, the copy backup will fail to complete the following Friday as we currently perform full backups each Friday.

If i reconfigure the copy backup to perform full synthetics, rather than active fulls, would this reduce the data/time required? If so, i understand the use of dedupe is not recommended when using synthetic full backups?

I have a case open and provided tech support the support logs this morning so they can identify the bottleneck.

Thanks for the advise so far.

P.Tide
Product Manager
Posts: 5264
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by P.Tide » Jan 08, 2019 2:25 pm

I could have misunderstood and we can re-enable the accelerators if it will help?
No, the support engineer is spot on - WAN accelerators won't help in this case. It was me who overlooked another "zero" so I thought it was 10Mbit link, sorry for confusion.

If i reconfigure the copy backup to perform full synthetics, rather than active fulls, would this reduce the data/time required?
If you are talking about switching the source backup job to synthetic, then it won't make much difference since BCJ transfers only new/changed blocks from the source job. If you meant "switching BCJ from active fulls to synthetic" then please elaborate a little bit, as BCJ has only "active full" option, not synthetic full".

If so, i understand the use of dedupe is not recommended when using synthetic full backups?
Considering the data that you've provided, yes. Since you use Windows deduplication, and your backup file is clearly way over 1Tb in size, this article is worth checking.

Also it is not quite clear whether the backup copy job fails to synchronize every Friday, or is it just the first run that cannot complete

P.S. Bottleneck stats can be found in the job session.

Thanks!

preecyp
Novice
Posts: 6
Liked: never
Joined: Jan 07, 2019 5:18 pm
Full Name: phil preece
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by preecyp » Jan 08, 2019 4:22 pm

Ah ok, i hadn't actually checked the BCJ settings. I assumed synthetic full was available within the BCJ options too.

Our .vbk files are around 2.5tb in size whereas the incremental .vib's are only 200gb. So going forward i guess i could disable Windows dedupe and re-enable the native dedupe & compression settings within Veeam?

I've checked the BCJ stats and its reporting 'network' as the bottleneck. We;re getting a processing rate of around 11MB/s.

P.Tide
Product Manager
Posts: 5264
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by P.Tide » Jan 08, 2019 7:40 pm 1 person likes this post

Since the bottleneck is the network, not the target, I'd suggest to leave Windows deduplication in place and change the job to 99 days, as Andreas's proposed, in order to have initial backup copy sync completed. After that, let BCJ to run incremental session and see if the bottleneck is still the network. As I've mentioned, BCJ should normally transfer only amounts of data that is more or less equal to your average increment.

Thanks!

preecyp
Novice
Posts: 6
Liked: never
Joined: Jan 07, 2019 5:18 pm
Full Name: phil preece
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by preecyp » Jan 09, 2019 10:38 am

Ok thanks i will give that a go, however, am i going to be back in the same situation again next friday when the active backup run for a second time?

Case # 03349141

P.Tide
Product Manager
Posts: 5264
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup copies taking too long, looking for a better strategy?

Post by P.Tide » Jan 09, 2019 12:40 pm

No, even if the source backup job generates active full, BCJ is supposed to transfer only changed blocks since the last run. Looking forward for the results from you.

Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests