v9 Problem - Daily Full to Tape with Copy Job as Source

Everything about backing up to tape

v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby FloG » Mon Nov 21, 2016 2:52 pm

Hello everyone,

we're currently experiencing major problems with daily full backup to tape with a backup copy job as source.

We currently have the following setup:
- Three vSphere 6 Hosts (two in primary site, one in remote office)
- One physical Veeam v9 Update 2 Server in primary site with iSCSI Repository (RepA) and SAS TapeLibrary
- One VM in remote office acting as Veeam Backup Proxy, Guest Interaction Proxy, etc. and using a CIFS NAS as Backup Repository (RepB)
- Several reverse incremental backup jobs to backup VMs from remote office to RepB (all running monday-saturday at 19:00)
- Several reverse incremental backup jobs to backup VMs from primary site to RepA (all starting monday-saturday at 20:00)
- One backup copy job to copy VM backups from remote Office RepB to primary site RepA (Interval 1 day, starting 18:55, BackupCopyLookForward RegKey enabled, no WAN acceleration, finishing normally at around 21:45)
- One daily backup to tape job starting monday-friday at 22:30 with linked jobs: all backup jobs from primary site, backup copy job remote to primary (write to Daily Media Pool, only Full, synthetic on monday-friday , no incremental)
- One weekly backup to tape job starting saturday at 22:30 with linked jobs: all backup jobs from primary site, backup copy job remote to primary (write to Weekly Media Pool, only Full, synthetic on saturday, no incremental)

We want only full backups to tape in order to be able to restore everything from one single media set. This works for all the latest restore points of the reverse incremental backup jobs (regardless whether weekly or daily backup to tape job is running). But the restore point of the backup copy job is only sometimes written to tape. To be specific: the restore points from the backup copy job are only written to tape on saturdays (weekly tape job) and on mondays, wednesdays and fridays (daily tape job).

I suspect this is caused by the virtual full interval (https://helpcenter.veeam.com/backup/vsp ... rvals.html) not finding a new restore point to synthesize when the last tape session ended after midnight thus running into the next virtual full day.

What can we do to get a complete full backup to tape every day including the restore points from the backup copy job ?

Also I haven't found a way to get the tape job to write the backup copy job's restore point of the actual day to tape. We always end up with e.g. a wednesday media set containing the wednesday backups from the primary site but the tuesday backup from the remote office (backup copy job) even though the backup copy job finishes copying the data about one hour before we even start writing to tape. Is there any way to get this resolved ?

Thanks in advance !
FloG
Influencer
 
Posts: 17
Liked: never
Joined: Mon Nov 21, 2016 1:27 pm
Full Name: Florian Guenzel

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby zpocrnjic » Wed Dec 14, 2016 3:04 pm

Hi,
Did you find solution for this? After upgrade to 9.5 I have exactly same problem. Veeam support recommendation was to setup backup jobs on the way that, Backup, Backup Copy and Backup to Tape jobs are created and completed same day. I setup like that but it is still writing data to tape every second day. Most frustrating is that it is reporting status "Success" even files are not written to tape.
Thanks
Zack
zpocrnjic
Lurker
 
Posts: 2
Liked: never
Joined: Thu Nov 10, 2016 3:20 pm
Full Name: Zack Pocrnjic

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby lyapkost » Fri Dec 16, 2016 5:45 pm

Hi! There is a speciality with backup copy jobs as a source for tape jobs
If the primary job is backup copy job, keep in mind that the last restore point of the backup copy job stays active until the next restore point is created. The tape job does not copy such active points, because they may be updated. For this reason, the backup chain on tape will be always one restore point shorter than on disk. see user guide

That's why the latest restore point copied by backup copy job becomes available for tape job only when the next backup copy interval begins (by defaiult it is in 1 day).

In v. 9.0 behaviour which Florian's described is expected because backup to tape job finishes after midnight and it's monday restore point counts as tuesday's one so the tape job doesn't copy another point on tuesday. It is not expected in v. 9.5 though.
lyapkost
Veeam Software
 
Posts: 76
Liked: 13 times
Joined: Fri Nov 27, 2015 2:26 pm
Location: Saint Petersburg
Full Name: Konstantin

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby FloG » Mon Dec 19, 2016 9:17 am

Hi Konstantin,

thanks for the reply. Since we're actually talking about to different problems I'll split this post accordingly.

Problem 1: One-day shift of backup copy job data on tape

So the only way to get the reverse incremental backups and the backup copy job from the same day written to one media set is to start the tape job after the copy job's interval ended. Is that correct ? If so, I will definitely give that a try.

Problem 2: Daily virtual fulls to tape

Just like in Zack's case I can confirm that even with v9.5 the behavior of daily fulls to tape hasn't changed. They're only written to tape every second day with the following message on missed days:
Synthesized full backup was not created for 15.12.2016: there are no new restore points between 15.12.2016 and 15.12.2016

According to the User Guide (v9.0 and v9.5) the Virtual Full Interval for selecting restore points as virtual full candidates hasn't changed. Could you please elaborate on why the same behavior is not expected any more in v9.5.

So, do I need to configure all my jobs (backup jobs, backup copy job including it's interval, daily backup to tape job) to finish before midnight of the same day to make this work ? This would be a problem because backup, copy and tape writing take about 7 hours at the moment. To finish all that before midnight I would have to start my backup jobs during production hours which is a no-go.

Any other ideas/advice on how to get this working ?

Thanks
Florian

PS: There are I few things I don't get with backup copy jobs when it comes to why things are done a certain way ... this seems to be one of those. Why does a restore point of a backup copy job have to stay active when all the linked backup jobs were successfully copied? As soon as one backup for every linked job is copied the restore point won't change any more until the interval ends and a new restore point is created. So why is it still active and not freed for further processing (e.g. writing to tape, creating a virtual full)? Probably there's a perfectly good reason for this that I just don't see but I have to ask ...
FloG
Influencer
 
Posts: 17
Liked: never
Joined: Mon Nov 21, 2016 1:27 pm
Full Name: Florian Guenzel

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby lyapkost » Mon Dec 19, 2016 10:28 am

Problem 1. You're correct, try to reconfig jobs and one-day shift should disappear. The reason why the last restore point is active till the next interval starts is quite simple: backup copy job knows nothing about source jobs' schedule (it may include several jobs with different schedules as source ones and moreover source jobs may be started manually which is completely unpredictable).

Problem 2. Here is one thing which is changed in 9.5 as compared with 9.0: in 9.0 virtual full backup got date/time of the moment it was created (last modified) on tape, in 9.5 it gets date/time 00:00 AM the day it was scheduled. This is made exactly to fix such issues as you described (tape job starts in the evening and continues after midnight, and if virtual full gets next day's time there will be no another virtual full later the same day). It is strange to hear that nothing's changed. I'll check it in my lab.

Thank you.
lyapkost
Veeam Software
 
Posts: 76
Liked: 13 times
Joined: Fri Nov 27, 2015 2:26 pm
Location: Saint Petersburg
Full Name: Konstantin

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby FloG » Mon Dec 19, 2016 1:02 pm

Thanks again for the quick reply.

Problem 1: I will reconfigure the jobs accordingly and see if the shift disappears.

While I'm aware that backup copy jobs don't (and probably don't need to) know about source job schedules I think it should be easy for the backup copy job to keep track about whether it already copied the restore point of a source job within the current interval or not. Since (as far as I understand) backup copy jobs only copy one restore point per interval for each source job this information should suffice to know when all restore points for all linked source jobs have been copied and then have the restore point of the backup copy job freed for further processing. Even the addition of another source job to the backup copy job wouldn't break this since after the modification the backup copy job would start a new interval anyway. Maybe this can be forwarded as a feature/modification request.

Problem 2: Maybe I'll try to recreate the daily tape job in v9.5 and see if this changes anything. But still it would be nice to hear if you can recreate the issue in your lab.

Thanks
Florian
FloG
Influencer
 
Posts: 17
Liked: never
Joined: Mon Nov 21, 2016 1:27 pm
Full Name: Florian Guenzel

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby lyapkost » Mon Dec 19, 2016 3:38 pm

Ok, I've just confirmed "problem 2", investigating it, I can't give any estimates though. Thanks for pointing it out.

Returning to the "problem 1" - I understand your point, well, staying in 'idle' status even if all backups are already copied is by design in backup copy job and affects interaction with tape job. It's unlikely to be changed because of too much logic connected here, not just tapes.
lyapkost
Veeam Software
 
Posts: 76
Liked: 13 times
Joined: Fri Nov 27, 2015 2:26 pm
Location: Saint Petersburg
Full Name: Konstantin

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby FloG » Mon Dec 19, 2016 3:53 pm

Problem 1: OK, I was just trying to understand the logic behind it. But if we can circumvent the problem by simply changing the schedules of some backup copy and tape jobs I'll happily take that. I already changed the schedules of our jobs and will see if it's working in the next few days.

Problem 2: Thanks for investigating the problem. I'll wait for a fix or workaround.

Thanks
Florian
FloG
Influencer
 
Posts: 17
Liked: never
Joined: Mon Nov 21, 2016 1:27 pm
Full Name: Florian Guenzel

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby nathano » Thu Dec 22, 2016 8:48 pm

lyapkost wrote:Ok, I've just confirmed "problem 2", investigating it, I can't give any estimates though. Thanks for pointing it out.


lyapkost have you made any progress with this? I have updated to 9.5 and I am still seeing this issue.

"22/12/2016 8:30:28 PM :: Synthesized full backup was not created for 22/12/2016: there are no new restore points between 22/12/2016 and 22/12/2016 "

I was told the midnight issue would be resolved with 9.5, is that not the case?

OR is this happening due to problem 1? as this is happening on the tape job "parts" that are coming from a backup copy job? and the backup copy job is set to run once a day at 7pm and run continuously, the copy job is finished within 45min. the source job for the copy job runs once a day.

the overall objective here is to get a fullbackup of all VMs to tape every night. This was stage 1 of the implementation as I replaced the previous solution. I ultimately want to have replication of the VMs from the remote site to the datacentre and then working with the replica as the source for backups might make the tape side of things easier rather than using a backup copy job as source. However it would be great if this worked as I was expecting :-)

Thanks
Nathan
nathano
Influencer
 
Posts: 15
Liked: never
Joined: Mon Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby nathano » Thu Dec 22, 2016 8:56 pm

FloG wrote:Problem 1: OK, I was just trying to understand the logic behind it. But if we can circumvent the problem by simply changing the schedules of some backup copy and tape jobs I'll happily take that. I already changed the schedules of our jobs and will see if it's working in the next few days.


FloG did you have any success with changing you copy job schedules? I'm not quite sure if I'm suffering from problem 1 or 2 but either way I'm skipping fulls to tape on the parts that are using a copy job as the source.

Any further info appreciated.

Thanks
Nathan
nathano
Influencer
 
Posts: 15
Liked: never
Joined: Mon Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby FloG » Fri Dec 23, 2016 10:06 am

Hi Nathan,

I changed the interval start/end time of my backup copy job and the schedule of my tape job so the tape job starts writing AFTER the backup copy job's interval ended. This way I solved problem 1 and now have the current restore point of the backup copy job on tape (not the previous one from the day before).

Unfortunately this had no effect in regards to problem 2. The restore point of the backup copy job is still only written to the daily tapes every second day.

I'm afraid we'll have to wait for the Veeam guys to further investigate this issue.

Regards,
Florian
FloG
Influencer
 
Posts: 17
Liked: never
Joined: Mon Nov 21, 2016 1:27 pm
Full Name: Florian Guenzel

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby nathano » Wed Jan 11, 2017 1:19 am

I just logged a ticket (02035131) about problem 2 as it's still happening in 9.5 (even thought I was previously told it would be fixed in 9.5) and I was sent a link to download update 1 for 9.5 and to install that and see how it goes. I have installed the update 1 and tonight the backup should be OK as it skipped last night, so tomorrow night I should know if it's working or not.

I'm also thinking that it's not possible to run 2 full backups to tape on the same day. say I want to run another full backup to tape for some reason, maybe to take offsite and test restore on a separate system, or perhaps for a special archive or something. If there's some rule that says only 1 full backup of a VM to tape each day then I'd like to understand what the reasoning for that is.
nathano
Influencer
 
Posts: 15
Liked: never
Joined: Mon Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby lyapkost » Wed Jan 11, 2017 9:50 am

Hi, Nathan. The "Problem 2" exists in v. 9.5 as well as in upd1, we're about to investigate it and fix in future versions.

To copy 2 same full backups to tape you need to create another backup to tape job (you may point it to the different media pool to separate tapes but that's not obligatorily). There are no other limitations.
lyapkost
Veeam Software
 
Posts: 76
Liked: 13 times
Joined: Fri Nov 27, 2015 2:26 pm
Location: Saint Petersburg
Full Name: Konstantin

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby nathano » Wed Jan 11, 2017 9:32 pm

So if I understand correctly, I will need to create one backup to tape job for Monday and Wednesday and a separate job for Tuesday and Thursday (they all use the same medial pool). I already have a separate job for Friday.
nathano
Influencer
 
Posts: 15
Liked: never
Joined: Mon Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield

Re: v9 Problem - Daily Full to Tape with Copy Job as Source

Veeam Logoby lyapkost » Thu Jan 12, 2017 4:00 pm

I didn't check it but it looks like creating couple separate jobs (mo-we-fr & tu-th-sa) is a valid workaround.
lyapkost
Veeam Software
 
Posts: 76
Liked: 13 times
Joined: Fri Nov 27, 2015 2:26 pm
Location: Saint Petersburg
Full Name: Konstantin

Next

Return to Tape



Who is online

Users browsing this forum: No registered users and 2 guests