v9 Problem - Daily Full to Tape with Copy Job as Source

[FloG]

Hello everyone,

we're currently experiencing major problems with daily full backup to tape with a backup copy job as source.

We currently have the following setup:
- Three vSphere 6 Hosts (two in primary site, one in remote office)
- One physical Veeam v9 Update 2 Server in primary site with iSCSI Repository (RepA) and SAS TapeLibrary
- One VM in remote office acting as Veeam Backup Proxy, Guest Interaction Proxy, etc. and using a CIFS NAS as Backup Repository (RepB)
- Several reverse incremental backup jobs to backup VMs from remote office to RepB (all running monday-saturday at 19:00)
- Several reverse incremental backup jobs to backup VMs from primary site to RepA (all starting monday-saturday at 20:00)
- One backup copy job to copy VM backups from remote Office RepB to primary site RepA (Interval 1 day, starting 18:55, BackupCopyLookForward RegKey enabled, no WAN acceleration, finishing normally at around 21:45)
- One daily backup to tape job starting monday-friday at 22:30 with linked jobs: all backup jobs from primary site, backup copy job remote to primary (write to Daily Media Pool, only Full, synthetic on monday-friday , no incremental)
- One weekly backup to tape job starting saturday at 22:30 with linked jobs: all backup jobs from primary site, backup copy job remote to primary (write to Weekly Media Pool, only Full, synthetic on saturday, no incremental)

We want only full backups to tape in order to be able to restore everything from one single media set. This works for all the latest restore points of the reverse incremental backup jobs (regardless whether weekly or daily backup to tape job is running). But the restore point of the backup copy job is only sometimes written to tape. To be specific: the restore points from the backup copy job are only written to tape on saturdays (weekly tape job) and on mondays, wednesdays and fridays (daily tape job).

I suspect this is caused by the virtual full interval (https://helpcenter.veeam.com/backup/vsp ... rvals.html) not finding a new restore point to synthesize when the last tape session ended after midnight thus running into the next virtual full day.

What can we do to get a complete full backup to tape every day including the restore points from the backup copy job ?

Also I haven't found a way to get the tape job to write the backup copy job's restore point of the actual day to tape. We always end up with e.g. a wednesday media set containing the wednesday backups from the primary site but the tuesday backup from the remote office (backup copy job) even though the backup copy job finishes copying the data about one hour before we even start writing to tape. Is there any way to get this resolved ?

Thanks in advance !

[zpocrnjic]

Hi,
Did you find solution for this? After upgrade to 9.5 I have exactly same problem. Veeam support recommendation was to setup backup jobs on the way that, Backup, Backup Copy and Backup to Tape jobs are created and completed same day. I setup like that but it is still writing data to tape every second day. Most frustrating is that it is reporting status "Success" even files are not written to tape.
Thanks
Zack

[lyapkost]

Hi! There is a speciality with backup copy jobs as a source for tape jobs

If the primary job is backup copy job, keep in mind that the last restore point of the backup copy job stays active until the next restore point is created. The tape job does not copy such active points, because they may be updated. For this reason, the backup chain on tape will be always one restore point shorter than on disk. see user guide

That's why the latest restore point copied by backup copy job becomes available for tape job only when the next backup copy interval begins (by defaiult it is in 1 day).

In v. 9.0 behaviour which Florian's described is expected because backup to tape job finishes after midnight and it's monday restore point counts as tuesday's one so the tape job doesn't copy another point on tuesday. It is not expected in v. 9.5 though.

[FloG]

Hi Konstantin,

thanks for the reply. Since we're actually talking about to different problems I'll split this post accordingly.

Problem 1: One-day shift of backup copy job data on tape

So the only way to get the reverse incremental backups and the backup copy job from the same day written to one media set is to start the tape job after the copy job's interval ended. Is that correct ? If so, I will definitely give that a try.

Problem 2: Daily virtual fulls to tape

Just like in Zack's case I can confirm that even with v9.5 the behavior of daily fulls to tape hasn't changed. They're only written to tape every second day with the following message on missed days:

Synthesized full backup was not created for 15.12.2016: there are no new restore points between 15.12.2016 and 15.12.2016

According to the User Guide (v9.0 and v9.5) the Virtual Full Interval for selecting restore points as virtual full candidates hasn't changed. Could you please elaborate on why the same behavior is not expected any more in v9.5.

So, do I need to configure all my jobs (backup jobs, backup copy job including it's interval, daily backup to tape job) to finish before midnight of the same day to make this work ? This would be a problem because backup, copy and tape writing take about 7 hours at the moment. To finish all that before midnight I would have to start my backup jobs during production hours which is a no-go.

Any other ideas/advice on how to get this working ?

Thanks
Florian

PS: There are I few things I don't get with backup copy jobs when it comes to why things are done a certain way ... this seems to be one of those. Why does a restore point of a backup copy job have to stay active when all the linked backup jobs were successfully copied? As soon as one backup for every linked job is copied the restore point won't change any more until the interval ends and a new restore point is created. So why is it still active and not freed for further processing (e.g. writing to tape, creating a virtual full)? Probably there's a perfectly good reason for this that I just don't see but I have to ask ...

[lyapkost]

Problem 1. You're correct, try to reconfig jobs and one-day shift should disappear. The reason why the last restore point is active till the next interval starts is quite simple: backup copy job knows nothing about source jobs' schedule (it may include several jobs with different schedules as source ones and moreover source jobs may be started manually which is completely unpredictable).

Problem 2. Here is one thing which is changed in 9.5 as compared with 9.0: in 9.0 virtual full backup got date/time of the moment it was created (last modified) on tape, in 9.5 it gets date/time 00:00 AM the day it was scheduled. This is made exactly to fix such issues as you described (tape job starts in the evening and continues after midnight, and if virtual full gets next day's time there will be no another virtual full later the same day). It is strange to hear that nothing's changed. I'll check it in my lab.

Thank you.

[FloG]

Thanks again for the quick reply.

Problem 1: I will reconfigure the jobs accordingly and see if the shift disappears.

While I'm aware that backup copy jobs don't (and probably don't need to) know about source job schedules I think it should be easy for the backup copy job to keep track about whether it already copied the restore point of a source job within the current interval or not. Since (as far as I understand) backup copy jobs only copy one restore point per interval for each source job this information should suffice to know when all restore points for all linked source jobs have been copied and then have the restore point of the backup copy job freed for further processing. Even the addition of another source job to the backup copy job wouldn't break this since after the modification the backup copy job would start a new interval anyway. Maybe this can be forwarded as a feature/modification request.

Problem 2: Maybe I'll try to recreate the daily tape job in v9.5 and see if this changes anything. But still it would be nice to hear if you can recreate the issue in your lab.

Thanks
Florian

[lyapkost]

Ok, I've just confirmed "problem 2", investigating it, I can't give any estimates though. Thanks for pointing it out.

Returning to the "problem 1" - I understand your point, well, staying in 'idle' status even if all backups are already copied is by design in backup copy job and affects interaction with tape job. It's unlikely to be changed because of too much logic connected here, not just tapes.

[FloG]

Problem 1: OK, I was just trying to understand the logic behind it. But if we can circumvent the problem by simply changing the schedules of some backup copy and tape jobs I'll happily take that. I already changed the schedules of our jobs and will see if it's working in the next few days.

Problem 2: Thanks for investigating the problem. I'll wait for a fix or workaround.

Thanks
Florian

[nathano]

Ok, I've just confirmed "problem 2", investigating it, I can't give any estimates though. Thanks for pointing it out.

lyapkost have you made any progress with this? I have updated to 9.5 and I am still seeing this issue.

"22/12/2016 8:30:28 PM :: Synthesized full backup was not created for 22/12/2016: there are no new restore points between 22/12/2016 and 22/12/2016 "

I was told the midnight issue would be resolved with 9.5, is that not the case?

OR is this happening due to problem 1? as this is happening on the tape job "parts" that are coming from a backup copy job? and the backup copy job is set to run once a day at 7pm and run continuously, the copy job is finished within 45min. the source job for the copy job runs once a day.

the overall objective here is to get a fullbackup of all VMs to tape every night. This was stage 1 of the implementation as I replaced the previous solution. I ultimately want to have replication of the VMs from the remote site to the datacentre and then working with the replica as the source for backups might make the tape side of things easier rather than using a backup copy job as source. However it would be great if this worked as I was expecting

Thanks
Nathan

[nathano]

Problem 1: OK, I was just trying to understand the logic behind it. But if we can circumvent the problem by simply changing the schedules of some backup copy and tape jobs I'll happily take that. I already changed the schedules of our jobs and will see if it's working in the next few days.

FloG did you have any success with changing you copy job schedules? I'm not quite sure if I'm suffering from problem 1 or 2 but either way I'm skipping fulls to tape on the parts that are using a copy job as the source.

Any further info appreciated.

Thanks
Nathan

[FloG]

Hi Nathan,

I changed the interval start/end time of my backup copy job and the schedule of my tape job so the tape job starts writing AFTER the backup copy job's interval ended. This way I solved problem 1 and now have the current restore point of the backup copy job on tape (not the previous one from the day before).

Unfortunately this had no effect in regards to problem 2. The restore point of the backup copy job is still only written to the daily tapes every second day.

I'm afraid we'll have to wait for the Veeam guys to further investigate this issue.

Regards,
Florian

[nathano]

I just logged a ticket (02035131) about problem 2 as it's still happening in 9.5 (even thought I was previously told it would be fixed in 9.5) and I was sent a link to download update 1 for 9.5 and to install that and see how it goes. I have installed the update 1 and tonight the backup should be OK as it skipped last night, so tomorrow night I should know if it's working or not.

I'm also thinking that it's not possible to run 2 full backups to tape on the same day. say I want to run another full backup to tape for some reason, maybe to take offsite and test restore on a separate system, or perhaps for a special archive or something. If there's some rule that says only 1 full backup of a VM to tape each day then I'd like to understand what the reasoning for that is.

[lyapkost]

Hi, Nathan. The "Problem 2" exists in v. 9.5 as well as in upd1, we're about to investigate it and fix in future versions.

To copy 2 same full backups to tape you need to create another backup to tape job (you may point it to the different media pool to separate tapes but that's not obligatorily). There are no other limitations.

[nathano]

So if I understand correctly, I will need to create one backup to tape job for Monday and Wednesday and a separate job for Tuesday and Thursday (they all use the same medial pool). I already have a separate job for Friday.

[lyapkost]

I didn't check it but it looks like creating couple separate jobs (mo-we-fr & tu-th-sa) is a valid workaround.

[PariPari]

Did Update 1 solve your problem?
We are facing a similar issue, same config (backup, backup copy, to tape from copy).
Should be easy, seems like it isnt.

[FloG]

As far as I understood from Konstantin's posts this is a confirmed issue in all current versions/updates. So v9, v9 Update 1 (don't know about Update 2 but probably too) and v9.5 are all affected.

Until there's a fix for this the only workaround seems to be what Konstantin's last post suggested:

I didn't check it but it looks like creating couple separate jobs (mo-we-fr & tu-th-sa) is a valid workaround.

I'm planning to simply clone my daily tape job and change the scheduled days for these two jobs to MO-WE-FR and TU-TH. My weekly/monthly/yearly jobs that run on SA are not affected since they're separate jobs already (not using GFS jobs but a "manual" GFS rotation managed by a PowerShell script)

[nathano]

I had setup two different tape jobs, one Mon&Wed and one Tue&Thur, there was already a different Friday job..

Not sure what's going on here but it's Monday night and the tape backup is backing up restore points from today (23rd) for local backup jobs and restore points from 22nd for the backup copy jobs! aarrgghh!!

no tape jobs run over the weekend..


the processing of the backup copy job finished at 7:07pm
and when you look at the files in the repository for the copy job, there's a vbk file timestamped at 23rd at 7:10pm

from the tape job
23/01/2017 8:30:07 PM :: Building source backup files list started at 23/01/2017 8:30:07 PM
23/01/2017 8:30:08 PM :: No restore point for 23/01/2017 is available for virtual full backup, using previous restore point for 22/01/2017
23/01/2017 8:30:08 PM :: Synthetic full backup from xxxxxxxxxx-SQL to NextD2017-01-22T190000.vib (23/01/2017 12:00:00 AM) will be placed into the media set

so there must be some other reason for it skipping the current restore point, and using the previous.

[nathano]

I have just logged a new case (02046548) about this. It's quite important we have the latest restore points to tape each night!

[FloG]

Hi Nathan,

to get the current restore point of the copy job written to the same tape(s) as the restore point(s) from the backup jobs of the same day I had to make sure the copy job's interval ends befor the tape job starts. In my opinion this is almost impossible with the default behavior of the backup copy jobs (at interval start -> copy most current restore points). I had to set the following registry key on the B&R server and reboot it:

Code: Select all

Path:  HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Key:   BackupCopyLookForward
Type:  REG_DWORD
Value: 1

This registry key changes the behavior of !!! ALL !!! copy jobs from "at interval start -> instantly start copying the latest restore points" to "at interval start -> wait for the next restore points to be created and then start copying them".

This way I was able to configure the following scenario (example for MO):
1.) Copy job to copy backups from remote site B to main site A starts SU at 11:30 pm with an interval of 1 day (due to the mentioned registry key the job sits idle after start)
2.) Backup jobs in remote site B start MO at 7 pm (running until about 8 pm)
3.) As soon as restore points are being created by the jobs from 2.) the copy job from 1.) starts copying them to main site A (running until about 10 pm)
4.) Backup jobs in main site A start MO at 8 pm (running until about 10 pm)
5.) Interval of copy job from 1.) ends MO at 11:30 pm thus its restore point is freed for further processing
6.) Tape job starts MO at 11:45 pm and writes restore points from 3.) and 4.) to tape

This setup works for me but clearly has its disadvantages:
- Changing the bahaviour of the copy jobs by using the registry key affects all copy jobs and you might need to rethink other parts of your backup plan
- The copy job interval runs from 11:30 pm to 11:30 pm but only has an effective backup window from about 7:30 pm (first backup jobs in remote site B finishes and gets copied) to 11:30 pm (interval ends). This might not be enough time to copy everything. Of course you could set the interval start/stop time to some other time but you'll also have to adjust the start time for the tape job accordingly to have it start AFTER the interval ends.
Personally, I wanted the tape job to start at the same day as the backup jobs (at 11:45 pm) so the media set name variables %date% and/or %dayofweek% reflects the day the actual backups were taken. If you don't mind the MO backups being on tapes that have "Tuesday" and/or tuesday's date in their media set name, you can of course push the start/stop time of the copy job and the start time of the tape job further into the next day.

I hope this helps at least a little bit and wasn't too complicated ... it's hard to explain in writing and not having english as your native language doesn't help either!?

[nathano]

Thanks for that, makes sense...

my backups are remote site start at 6:00, 6:30 and 6:45 pm and I have them setup with the secondary storage target so it effectively triggers the copy job.

the backups take a matter of minutes to run and backup copy jobs have the latest restore points copied to the datacentre by about 7.30pm, the tape job starts at 8.30pm

on the schedule tab of the copy job I have it set to run any time (continuously) perhaps that is my problem. I will try defining time on the schedule tab so that it only runs from 6pm to 8pm and see how that goes. I don't like restricting the time on the copy job like that but if it gets it working for now then that's a good thing and I'll go from there.

I also have a reply from Veeam support and will work with them as well.. I just need to understand what the limitation is exactly so I can work around it and push Veeam to fix it

[FloG]

I don't think that the restriction of time slots on the Schedule tab will make any difference. The settings there are only to e.g. prevent the copy job from copying data during production hours. The important settings in regards to whether the copied restore points are free for tape writing is the interval and start time setting on the Job tab.

My copy job for example is configured to run every 1 day starting at 11:30 pm even if the backup jobs don't start before 7 pm the next day. But by setting the start time of my 1 day interval to 11:30 pm I can make sure that the current interval (that for example started on SU 11:30 pm) that copied my restore points from the backup jobs that started at MO 7 pm ends on MO 11:30 pm before my tape job starts at MO 11:45 pm. This way at MO 11:45 pm the copy job has already started a new copy interval and waits for new restore points to be copied and thus keeps his fingers off the restore points from MO 7 pm when they're written to tape.

But as mentioned before, this only works because I set the registry key to change the copy jobs behavior to copy the NEXT restore point that is created within it's interval instead of copying the MOST CURRENT one that already exists. I'm not sure how this would work with the default copy job behavior.

In my opinion the whole backup copy job thing is far from perfect but most of the time you can push it to accomplish what you want it to And I guess the Veeam guys will figure something out that fits your needs.

[nathano]

you're right, the scheduling made no difference!

I understand what you are saying with that regedit change. I will read it again when I'm not tired and make sure I really understand the implications of doing that and also see what the Veeam Support guys have to say.

With the split tape jobs (mon/wed and tue/thu) there should be no problem starting my tape job later and having it run past midnight so I can make sure there's plenty of time buffer with my settings...

BUT - I would have thought that it would "just work" as most Veeam stuff does

[FloG]

Yeah, you're right ... to me it seems a bit like backup copy jobs never work the way you would want/need them to and you always have to tweak everything around it to make the whole thing work somehow!?

I guess I'll just gather all the points that I don't understand/would like to work differently and post a feature/enhancement request when I have the time ...

[FloG]

Are there any news regarding a solution (not a workaround) for this problem ? Will this maybe be fixed in v9.5 Update 2 ?

[stoal76]

Hi guys, we are still experiencing problem 2, where Copy jobs that contains a full backup are not going to tape and we are running 9.5 update 3a.

My case number is 03409945

Can anyone confirm if this problem has been fixed? or if the work around discussed above resolves it by setting up a separate backup job for each day?

It seems to be very random as to when the tape backup sees a full even though my main jobs are set to Synthetic full every day minus the one day where we do an Active full.

I have two copy jobs that go to the same tape job and one can see a full and the other can't and it rotates, sometimes they both see a full other times either one doesn't.

I have noticed that the days it does backup the full it is from the previous day's full which is fine.
23/02/2019 11:30:13 p.m. :: No restore point for 23/02/2019 is available for virtual full backup, using previous restore point for 22/02/2019

But then days it doesn't backup the full it doesn't look like it is dropping back to the previous days full even though I have confirmed there is one.
23/02/2019 11:30:13 p.m. :: Synthesized full backup was not created for 23/02/2019: there are no new restore points between 23/02/2019 and 23/02/2019

Any help would be appreciated

[nathano]

I've just noticed the problem is happening again for me, just checking to make sure I haven't changed anything.. has all been working previously.. very frustrating..

[veremin]

Might be worth installing Update 4 first (basically, that's the first step that will be recommended by our support team). Thanks!

[nathano]

I did u4 as soon as it was rtm.

[Dima P.]

Nathan,

If you confirm the said behavior with the latest Update 4 please open the support case (as we would need the debug logs anyway). Please do not forget to share the case ID with us. Thanks!