Host-based backup of VMware vSphere VMs.
Post Reply
fafa24
Enthusiast
Posts: 70
Liked: 6 times
Joined: Jun 14, 2015 4:48 pm
Contact:

Backup copy offsite storage

Post by fafa24 »

Dear All,

We have a backup copy job for one bigger server. The VM is about 1.5GB.

Interval for this job: 1 day
Restore point: 7
weekly: 1
monthly: 5
WAN accelerator: yes
Allowed window time: business days night only, weekends hours extended

This is a copy job which runs now for almost a year. Initial the file was seeded. However since approximately beginning of October the copy job always fails. Meaning the backup windows is too short to complete. I think it has happened when I deleted manually the last monthly backup because offsite we were out of space. I did this for other jobs as well and they still complete fine. I mention this, but not sure it is related. I also amended all jobs to keep less monthly backups.

What can I do that this job completes again? Could it be that the interval for one day is too short? My assumption is even the interval starts every day from the beginning with the WAN accelerator on, the job should continue where it was.

Thanks,
Edy
ACC
Enthusiast
Posts: 25
Liked: 5 times
Joined: Apr 29, 2015 10:38 am
Full Name: ACC
Contact:

Re: Backup copy offsite storage

Post by ACC »

If its simple a matter of to much changes for one day, the solution is temporary upping the copy interval to several days to allow the copy job to continue copying the data. The problem is that 1 day interval also limits the copy job to use restore points within that same period. If a disk isnt copied within that period on the next cycle it will start copying the changes from the beginning again because the copy job is forced to use the new restore point and discards the partial transferred data. The new cycle failes also because it cant copy the new data in time and in essence the gets more behind every day. The only way to let a straggler VM catch up is to increase the interval so the copy can use the older restore point and continue were its left off.

Personally i find this bit of a design oversight of Veeam, that still the only way to let it catch up is to stop of the offsite copy for several days until the straggler is caught up. I would love Veeam would implement a configurable time window to allow the copy job to use older restore points in order to be able to complete that restore point for straggler VMs, while the other VMs are continue their regular cycle. This way the offsite copy across wan connections has a "autoheal" feature.
fafa24
Enthusiast
Posts: 70
Liked: 6 times
Joined: Jun 14, 2015 4:48 pm
Contact:

Re: Backup copy offsite storage

Post by fafa24 »

Thanks for your reply. I thought the copy interval could be the problem, but I was not sure. I have set the copy interval now to 9 days temporary. This gives me the whole week and more important all next weekend. I will monitor the job and if the job will finish early I can lower the copy interval.

We still have version 8.02. Do you think with version 9 it has improved?

Thanks,
Edy
ACC
Enthusiast
Posts: 25
Liked: 5 times
Joined: Apr 29, 2015 10:38 am
Full Name: ACC
Contact:

Re: Backup copy offsite storage

Post by ACC »

I havent noticed any improvements with Veeam 9 on this matter.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy offsite storage

Post by foggy »

ACC wrote:If a disk isnt copied within that period on the next cycle it will start copying the changes from the beginning again because the copy job is forced to use the new restore point and discards the partial transferred data.
It is not entirely correct, since data transfer will be resumed and only missing blocks (those that are required to build the new restore point corresponding to the VM latest state) will be transferred along with the changes occurred since the last source restore point. Previous restore point will be left in incomplete state.
ACC
Enthusiast
Posts: 25
Liked: 5 times
Joined: Apr 29, 2015 10:38 am
Full Name: ACC
Contact:

Re: Backup copy offsite storage

Post by ACC » 1 person likes this post

Ive battled many times with this issue and contacted Veeam Support several times about it and in my experience the fastest way to get straggler and offsite-backup in general back into sync is increasing interval until the restore point is complete and then reducing the interval (if necessary in steps) back to the original interval.

Personally would like to configure the behavior of the copy job, to be either the current Veeam default or Veeam continuing the incomplete restore point. The last behavior is preferable when dealing with offsite repositories, which usually comes with limited bandwith and transfer windows, for two reasons. The first, a several days old but complete restore point is much valuable to me then a chain of incomplete restore points. The second reason is data transferred, for example if the copy job needs 4 days to upload a restore point, when the copy job start a new restore point upload it only has to upload the delta between the new and previous restore point and dont have deal with all the data that has changed multiple times within that time frame.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy offsite storage

Post by foggy »

ACC wrote:Ive battled many times with this issue and contacted Veeam Support several times about it and in my experience the fastest way to get straggler and offsite-backup in general back into sync is increasing interval until the restore point is complete and then reducing the interval (if necessary in steps) back to the original interval.
I'm no way arguing this, those are correct steps in this situation. Was just a small technical correction.
fafa24
Enthusiast
Posts: 70
Liked: 6 times
Joined: Jun 14, 2015 4:48 pm
Contact:

Re: Backup copy offsite storage

Post by fafa24 »

After three days the copy job finished and I have now lowered the copy interval to one day again. The window is now fine and can ttransfer the data. However I ran into another problem with merging the restore points.

This is the error message:

Failed to merge full backup file. Terminated at 29.10.2016 22:29:21 Failed to generate points Error: Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup. Please wait, and try again later. Error information: The DELETE statement conflicted with the SAME TABLE REFERENCE constraint "FK_OIBs_OIBs". The conflict occurred in database "VeeamBackup", table "dbo.Backup.Model.OIBs", column 'link_id'

We have MS SQL 2012 installed locally on the same B&R server. There are other backup copy job which runs locally and to the offsite location without a problem. So I doubt there is a problem with the SQL server. I have opened now a case 01962309 with Veeam.

Maybe the community has an idea what went wrong.

Thanks,
Edy
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy offsite storage

Post by foggy »

I'd let our engineers review this issue more closely, since the error contains the exact statement that caused it, so they will be able to verify. Unfortunately, I cannot check the case notes, since it is in German.
fafa24
Enthusiast
Posts: 70
Liked: 6 times
Joined: Jun 14, 2015 4:48 pm
Contact:

Re: Backup copy offsite storage

Post by fafa24 »

I was in contact with an engineer exactly a week ago. The suggestion was to create a new backup copy job and move the existing backup chain on the offsite store to a new folder. Which means to build up a new chain. Since the speed is not really fast the backup copy has not finished yet, but is almost through. We use a backup window. The engineer didn't really tell me what the problem is. I wanted to use the existing chain as a backup seed, but he recommend not to use it. I set the copy interval to 4 days for the first run, it couldn't finish the job during this interval. Now I set it to 1 day. However since new restore points will be added daily from the normal backup job I'm not sure it was a wise decision to change it back to 1 day. Should I let it let the initial job finish first?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy offsite storage

Post by foggy »

It is recommended to extend the copy interval initially, to let the job finish the first sync.
Post Reply

Who is online

Users browsing this forum: randerson and 88 guests