Feature Request (Tape and Forever Incremental)

[newfirewallman]

I can't have 2 full backups transfered to tape and so the forever incremental modifies the vbk every day (merging oldest incremental), there are two vbk files copied to tape, the oldest point in the chain, and the synthesized one (plus the incremental files)
I've using reverse incremental to avoid this problem and copy to tape only the full backup, that is the newest one, but It stresses my backup target too much=slow.

Regards.

I also had to use reverse incremental to avoid that and also to work around the shortcomings of tape backups. But it also stressed my backup target to much with 5-6 jobs totaling 10-15TB. Forever incremental is what I need (still slow when merging, but better)

So will there be a fix to backup to tape with synthetic and still allow the daily incremental to disk?

[veremin]

Kindly, open a ticket with our support team and provide case number here. It's getting hard to understand through forum correspondence what goal you're trying to achieve and how. Thanks.

[newfirewallman]

New Case created.
00892770

[Dave338]

I opened a case some weeks ago and the final response was to use reverse incremental and wait for a patch.

Now the patch is released but this issue remains present.

Is simple, there's no way to have incrementals and a simple backup to tape of the most recent restore point, or I haven't found it. You get your full backup copied (which is from some days ago), or many full backups, or the full+synthesized, or also the vib's.... for me, used to other backup softwares ir nearly impossible to understand how it works xD

The option to backup only latest chain that appears if you have more than one full on the target seems the salvation, but I'm testing and seems only working on the first run of the job...

Is any difficult to have an option that makes the tape backup copy ONLY the latest full backup, independently of your backup strategy or type??

Regards
Dave.

[Dave338]

Mmmm... after thinking about it for a while... I'll try to resume the "problem"

Problem occurs basically when using Forever Forward. One your retention period is reached, the full vbk file gets modified every day (or backup), merging it the older incremental, so veeam thinks is a new vbk and copy it to tape with your synthesized full, so finally 2 fulls on the tape, plus the incrementals because you had to enable the "process incrementals". This last problem is solved in theory with patch 2 (any confirmation?), so now you don't have to enable the "process incrementals" option to enable synthesized. But the initial problem is solved? Or I will continue getting 2 fulls on tape every week because my "old" vbk is modified?

With reverse incremental, it works well until you make an active full, because also the vbk is modified every day, and when you make an active full, you have two new vbk to copy on tape, unless you make the tape backup between your last incremental and the active full, which is a very narrow windows. So again 2 full backups on tape.

The only way seems to have Forward+active fulls on friday and make the tape copy at weekend, after the active full is done, so it will copy only the new full backup because previous one is supposed to have been copied week before and veeam will dismiss it. I think this one should work and probably is the method I'm going to try, because I'm not comfortable with the reverse incremental method without doing any active fulls, or doing it once a year or so... (I can't be making manual tape backups every week/month). The trick is to start the strategy on friday so the first full backup will be copied to tape that weekend, and next week only new active full will be copied to tape. If you make your first full at Tuesday for example, and have another active full at Friday, on Sunday your tape job will copy again two full backups. Only at first run but is annoying everyway...

I expect it to be much more clear now...
Is all this stuff correct??? xDDD

Regards.
David.

[newfirewallman]

I had to use forever incremental with transform. As long as the my backup storage was fast enough it would be fine because the vbk wouldn't change, but once a week. It became a problem when my file server backups were large enough with enough changes that it wouldn't finish transforming in time for the backup to tape to run.

With Update 2 it doesn't seem to pull two VBK or process incremental's, which is very good as the correct data and amount of data is going to tape. The only issue that continues is the fact that the vbk is changed everyday once your at the retention period (when using forever incremental) and this WILL break the tape job when it is running.

[newfirewallman]

This is my recent update from my ticket:

This is intended behavior. The tape job will always fail if the backup job kicks off. There are a couple things we can do to mitigate this, though.

1. Continue to disable your standard jobs when the tape job is running.
I don't want to miss backups, nor do I want to have to manually monitor my backups with so much intervention. I have to wait for merging to finish and then start a job while disabling other jobs. It isn't consistent and should be expected.
2. Rethink the job schedules, so that, for example, the tape job runs on Saturdays, and the backup job does not.
If you are a 24/7 shop or even just a 7 day a week shop this isn't acceptable. This is the method i'm shooting for at the moment by just missing a backup on sunday night. It would also help and be a slight work around if i could have the daily backup start at different time on different days. For instance if i could start my saturday backup at 1pm instead of 9pm like a normal day? This would give it more time finish and merge, then i could start the tape job.
3. Change the backup method of the backup job to either Reverse Incremental, or Incremental with weekly Active Fulls. If you make this change, the tape job will not have to do a synthetic and thus will finish much faster.
Revere incremental is no good because it has the same issue, the VBK is constantly changing daily.
Incremental with transform work, until the IO become and issue and it takes DAYS to transform and get caught up and then i can run my tape job. In my instance it was taking 4-6 days to transform all my data to get caught up.
Active fulls would work if Veeam would truly grab only the latest VBK, but my experience hasn't been great with that. Might the first backup job, but it has been inconsistent and sometimes grabs both. Having 1 backup job that is 9.5TB takes a lot more space on disk than i would like when i have to keep two full VBK's and all the incrementals, plus the extra tapes it takes when veeam decides to backup both VBK's


Additionally, there were very significant improvement with merges in Update 2, so if you don't have that, yet, I would strongly recommend applying it to shorten the amount of time the synthetics take.

Thanks,
Greg Bean
Veeam Support

[michaelryancook]

I am in the same boat and believe there is an issue in the way the backup to tape is handled within Veeam. There needs to be a way to have tape jobs only write the most recent .vbk to tape. My expectation was that if I am using reverse incremental with monthly active fulls when my backup to tape job runs it should either copy my most recent .vbk produced by the reverse incremental job or the .vbk produced by the active full based on whichever was more recent. Not write both to tape which requires twice the number tapes. I'm thinking the backup job should have a checkbox saying active fulls only which would let it write just the most recent active full .vbk and/or a "copy every" interval like backup copy jobs do which would dictate how far back it should search for .vbks to write to tape and would only write one per interval. I think these options would address this issue for reverse incrementals but not other methods.

Thoughts?

[justind]

The Veeam way to address this is to re-create your tape job and choose Latest backup chain or something similar. Just do this before every backup cycle and you're sweet.

There is another thread that has been going on about this forever and that is the solution... or migrate to the forever incremental.

[michaelryancook]

Hmm, I'm wondering if the choose latest is selectable via the powershell snapin? If so I could write a script that removes the one tape job and creates a new job with that option selected then have it scheduled monthly.

[Dave338]

I have re-thinked my last post and maybe I have found a solution.

Actually I have an 8 point retention in my backup job and a 2 daily r.p., 1 weekly and 1 monthly on a GFS backup copy job, so If I do an active full, next week I would have two fulls on my folder, which is undesirable.

I'm going to move to 10 daily restore points on my GFS, eliminate the weekly (with 10 dailys it has no sense, because it would be redundant), and maintain my monthly, leaving only two rps in my backup job.
With that configuration, if I do the active full on Monday, on Friday I'll have only one full backup, and so my tape backup will have no problems.

I'll post my findings If I change again my strategy ,hjehjehe

Regards.

[Dave338]

Oh I've just remembered... If I do that, my monthly backup will be syntethised 10 days later than I want, after the retention date has been expired... another effect that I don't understand about the GFS backup copy policy... but I won't have the downside of having two full backups on the backup job chain when tape job kicks on, so I'll continue with this configuration.

David.

[Greg.Bean-DeFlumer]

What Blake is asking for, specifically, is this:

There would be an option that would cause the source backup job to check if the tape job is currently running when the source job kicks off. If the source job detects that the tape job is currently running, the source job would wait to actually start until the tape job is completes, therefore not causing the tape job to fail.

[Dima P.]

Hello guys,
Mind me joining back to the discussion I’ve previously abandoned, to sum up Update 2 synthesized full backup logic.

Schedule:
-The synthesized full backup schedule equals the day when the full backup from the source chain should be created. It’s not a backup job schedule trigger – it’s just an actual pointer to the daily increment from the repository, which will be taken into processing once the backup job is started.

How increments are selected for synthesized full backup:
- If there is no incremental restore point for the specified day – logic will pick the incremental restore point for synthesized full backup creation from the day before. If there is no incremental restore point again it will keep checking all previous days in the chain unless there are no unused incremental restore points, by unused I mean previously not written to the tape.
- If there are many increments for the set day – only the latest will be taken for synthesized full source.

Incremental backups processing:
- It’s possible to uncheck the incremental backup processing an leave synthesized full backup only.
- All incremental restore points are written to tape except the taken for synthesized full backup creation. All the further incremental restore points on tape are relinked to the last created synthesized full backup.

Limitations and tricks:
-Synthesized full backup will be created only for forever incremental backup jobs and backup copy jobs. Incremental with active/synthetic full and reversed incremental backup as a source will ignore this option and described logic.
-If you set backup to tape job to run once a week and check more than one day in the synthesized full backup schedule – only one full backup will be generated on tape, closest to the day of the backup to tape job run.
-If you want synthesized full backup to be created more frequently (i.e every day) you need to set backup to tape job schedule to run every day.

[Dima P.]

Now let’s get back to your issues and questions:

The option to backup only latest chain that appears if you have more than one full on the target seems the salvation, but I'm testing and seems only working on the first run of the job...

It is true, however, we recognize this functionality to be highly requested – so it’s now in the top priority list.

Problem occurs basically when using Forever Forward. One your retention period is reached, the full vbk file gets modified every day… But the initial problem is solved? Or I will continue getting 2 fulls on tape every week because my "old" vbk is modified

It should be solved in the Update2. If you see any duplicated .vbk it should be counted as bug and reported to support team

this is intended behavior. The tape job will always fail if the backup job kicks off…

Have to admit it’s totally looks like a bug. I’ve already forwarded all your findings to QA team for investigation. Will follow up on this one once I hear anything.

Hmm, I'm wondering if the choose latest is selectable via the powershell snapin?

Not possible. Only via hard links workaround provided in this thread

my monthly backup will be syntethised 10 days later than I want, after the retention date has been expired

It will check the synthesized full backup schedule every backup to tape job run, so if the restore point still persists on repository and was not previously written by this tape job – then yes, but, please, keep in mind the general recommendation to have the tape job’s retention longer than the disk job.

[Greg.Bean-DeFlumer]

Have to admit it’s totally looks like a bug. I’ve already forwarded all your findings to QA team for investigation. Will follow up on this one once I hear anything.

What makes you think this looks like a bug?

The tape job should be stopped as it has less priority in comparison with backup job (secondary vs primary destination). Thanks.

Blake (newfirewallman) isn't talking about tape synthetics. If a backup to disk job kicks off while a regular backup to tape job is running (not a synthetic), the tape job will stop. From my understanding, if the files we are reading are required to be locked for the tape job, the tape job will be stopped, because it has less priority than the backup to disk job.

[Greg.Bean-DeFlumer]

Speaking about workarounds, you might want to set a certain script as a pre-job command. The script will check whether the tape job is running and proceed to execution only if it is not. Thanks.

He wants to have the backup to disk job (forever incremental) stop before the merge, then allow the tape job to finish, and then complete the merge on the backup to disk job.

[Greg.Bean-DeFlumer]

Finally found the post I was looking for.

Can you elaborate on your request? You have two jobs (backup and backup to tape one), sometimes it happens that the backup to tape job takes longer than expected, and the subsequent run of the backup job fails due to files being locked, correct? And you want to have sort of timeout after which the backup job can proceed smoothly?

[newfirewallman]

Thank you for taking interest in my findings I truly appreciate the desire to fix an issue that has been frustrating and a limitation to software that is so reliable.

Ideal solution is when a synthesized tape job is running (that may take 2 days or more because of the size) it would be great if it can still run the backup to disk job (which is one of the sources for the tape job) without the synthesized tape job being affected. (create the new incremental, but wait to merge until after the tape job runs).

[newfirewallman]

"From my understanding, if the files we are reading are required to be locked for the tape job, the tape job will be stopped, because it has less priority than the backup to disk job."
Issue with the statement in quotes above. This is what i want to avoid (where tape job is synthetic or not) I love the regular backup to disk to still run as an incremental, wait for tape to finish, then commit changes to the regular disk backup job. Thus not affecting the tape job, AND not missing a regularly scheduled backup.

[Dima P.]

Blake,
Thanks! QA team is looking into this so, hopefully, we will have this behavior fixed in the close future. Cheers!

[richard.hare]

Hi,

Just wondered if there was any news on the "Postpone merge until tape job finishes" feature request? (Wondering if it appears in V9)

Thanks again!

[Dima P.]

For sure, we did not implement it as a feature request; however, we enhanced the logic under the hood – I hope v9 will address the described issues.

[richard.hare]

Hi,

Did we manage to get the "Delay merge until after tape job" function in V9 or am I missing the new logic/option in the gui?

[Dima P.]

Hello Richard,

We are still in discussions. As for the v9 behavior – the source disk job is still counted as a high priority and will stop the tape job. Just to clarify, you would like the source job to be postponed or tape job to be retried (continued) after merge of the source job completed? Thanks.

[richard.hare]

Hi and thanks for the quick reply.

I think the best solution would be to allow the tape job to run as normal, have the backup job create an incremental as normal (I am assuming both would just require read access to the existing backup files at this point, so locking shouldnt be an issue) and then have the disk job incremental merge occur after the tape job completes.

If this isn't a plausible solution or for jobs that do an active full that may clash with secondary backups, maybe just an option on the secondary backup job... "Delay primary job(s) until this job completes". That way, you could choose the behaviour you would like, per job. If you didn't select the option, the secondary target backup would stop, as it does now.

I may be over-simplifying it, and would welcome comments, but that would fit our scenarios pretty well.

Thanks!

[newfirewallman]

Had a recent ticket open with Veeam on a different tape issue and mentioned this lack of a feature (tech noticed all the scripting being done to ensure it was the correct day, the jobs were complete, kicking off Tape job and then when tape job complete it was re-enabling the other jobs). He thought it was a great idea and a real concern. Would be really nice to see in the next release.

[Dima P.]

Blake,

The option to give a priority to tape jobs and put on hold the source disks jobs is coming soon

[newfirewallman]

Sounds like a nice work around, I will take it over my scripting to accomplish the same thing. Although what would be ideal (thinking to where my environment will be going) is to be able to have the scheduled incremental run (daily backups) while the synthesized backup is also running and just not commit the incremental until the tape job is finished. This way when you have very large backups going to tape (weekly or monthly) and the requirement for daily backups is still in affect you won't lose backups during that time frame.

[Dima P.]

just not commit the incremental until the tape job is finished

That's the plan.