architecture with Virtual Synthesized Full Backup for tape

[wbauer84]

We have a environment with Veeam 8.0 Backup & Replication

Our intention in this environment is to have an actual full backup every day on tape.
We are going with a one time full backup and "forever incremental" backup to the repository, and from there we want to make a "Virtual Synthesized Full Backup to tape" from the full backup file and the following incremental backup files
For this, an co-worker of your company told us to put the settings as followed, mainly to REMOVE EXPLICITLY the “process incremental backup files” setting in the tape job. In the schedule area of the media pool/full backup section, we've selected already the days, which should be the option, when he makes this virtual synthesized full backup:




The result is, after he made in the main "backup to the repository" job, a successfull incremental backup, even if he merged because of the retention policy the OLDEST incremental backup file into the full backup file, as you can see on the screenshot: The Tape job doesn't recognize there is new data.





Any ideas??

[hoFFy]

Nice one, we've got exactly the same problem... "No new files to backup per job setting".

[foggy]

Guys, in order to enable synthesized backup, you need to enable processing of incremental files in the tape job.

[wbauer84]

As far I understand the "Veeam_backup_8_userguide_vmware.pdf, pages 695+696 (pdf page), you don't have to check this box:
"You can schedule creating a synthetic full backup. The synthetic full backup file is created directly on tape and requires no disk space. To schedule the synthetic full, click Schedule and select the necessary scheduling options."

This is the Schedule box beside the "Media pool for full backups" section in the tape job, I've put also in my screenshots!

Because when we check the "Proces incremental backup files", even we have enabled the days in the schedule box above, he stores us ALL incremental backup files on a tape, and doesn't make any kind of synthesized backup.
By the way a Veeam engineer which was helping us with presales support, was also telling us, that the checkbox "process incremental backup files" has NOT to be checked in order to process a "Virtual Synthesized Full Backup for tape"

[Dima P.]

Wolfgang,

I am sorry for the misunderstanding – the feature of the synthesized full backup to tape is a brand new (it was just released in v8) and it might be confusing. “Process incremental backup files” should be additionally checked in order to trigger the synthesized full creation in the current version. However, we are aware that users would like to have synthesized full back up without incremental backups processing, so we are currently working with the dev team to make it happen.

Once again I am sorry for any confusion caused, I will send a follow up to tech support team and tech writers team to update their internal and external docs – thank you for pointing this out.

[hoFFy]

I'm currently running a job with "processing incremental files" and can confirm that much more data is written to tape. The job hasn't completed yet, only 4%, but has already written 130GB to tape. Normally one tape backup uses about 800GB... It seems that there is much more done than wbauer84 and me expected

[hoFFy]

Wolfgang,

I am sorry for the misunderstanding – the feature of the synthesized full backup to tape is a brand new (it was just released in v8) and it might be confusing. “Process incremental backup files” should be additionally checked in order to trigger the synthesized full creation in the current version. However, we are aware that users would like to have synthesized full back up without incremental backups processing, so we are currently working with the dev team to make it happen.

Once again I am sorry for any confusion caused, I will send a follow up to tech support team and tech writers team to update their internal and external docs – thank you for pointing this out.

That said we have to continue using reversed incremental in the source backup job to only write a daily full backup to tape?

[foggy]

That said we have to continue using reversed incremental in the source backup job to only write a daily full backup to tape?

Yes, in case of reverse incremental job, only VBK is copied to tape during each job run.

[hoFFy]

Yes, in case of reverse incremental job, only VBK is copied to tape during each job run.

Of course, that has been the typical way to create a daily full backup to tape. I (we) thought, that it's possible with v8 to write daily full backups to tape without using the slower reversed incremental in the source backup job.

[foggy]

My response could sound a bit confusing. Actually, you can use either of the methods, in both cases (incremental forever with synthesized tape backup or reverse incremental) just a single full is copied to tape. The fact that you enable processing of increments doesn't mean that increments themselves will be copied to tape (in fact, increment will be copied if the day it is created on is not selected in the synthesized backup schedule).

[hoFFy]

Okay now I understand what you mean

The amount of data written to tape is in both cases the same, right?
Which files does Veeam read in order to create a synthesized backup to tape? All files from incremental forever? If thats the case, creation of a synthesized backup will take longer if there are many restore points, right?

[foggy]

The amount of data written to tape is in both cases the same, right?

Right.

Which files does Veeam read in order to create a synthesized backup to tape? All files from incremental forever?

Those files that contain data blocks that are required to synthesize a full backup.

If thats the case, creation of a synthesized backup will take longer if there are many restore points, right?

In fact, the difference should be almost unnoticeable (unless you're copying from a dedupe storage).

[wbauer84]

Hi foggy,

Well, we re-checked the option "process incremental backup files" and it seems now, that he processes the incremental files now in the synthesized vbk file on tape.

But it seems, that he still has a weird behaviour:

In both environments we make forever incremental backups to disk repository every 3 hours.
When now the tape job, belonging to this forever incremental job starts, and in the meanwhile another time the incremental job ran, he wants to backup in the running tape job again a full vbk file, and tells in the job: "Changes in source storage detected. Scan for changes"
That leads to, that in environments with tape libraries, he takes one tape after the other (eating that much cannot be healthy), and in the environments with a single tape drive, he doesn't finish the job, and wants another tape

Can you STOP this behaviour, that he just backup to tape ONCE and not scanning for changes again, when he backupped already a full vbk with synthesized vbi's?

[wbauer84]

Hi foggy again!

Now I was checking the jobs again,

In one job, where the settings are the same as in my screenshot, additionally everywhere of course "process incremental backup files" checked now, he was backing up two vbk-Files and one vbi-Files on ONE tape, where is in the repository just one vbk file and of course many following vbi files.
Then I wasn't really sure about what was happening, so I started in this job again a incremental following up backup job, let him finish, and startet then the corresponding tape job again.
Normally he should write FULL SYNTHESIZED VBK with the state of the last VBI-File to EVERY TAPE JOB, right?
This time for example he was just processing the LAST VBI-File (and JUST the vbi file to tape, so NO synthesizing), and the ejected the tape again.

Somehow really this function doesn't seem to work as well, as it should...
Or can you explain me this behaviour?
By the way, we were recreating the Backup to repository disk jobs 2 days ago completely new from scratch, that we have a fresh and clean VBK/VBI line, so were not talking about long time ago created jobs....

[veremin]

What schedule does a backup to tape job has? Is it "as new backup files appear" or "after this job" by any chance? Thanks.

[wbauer84]

None of both.
He's running once a (working) day, in the evening per time schedule.
In the case I described in my post today 10.49pm, we just were starting the tape job IN the day, so in the same time, when there were incremental jobs running at the tape-linked backup-to-repository job.
In the case I described in my post today 11.19pm a tape job was running by manual already in the afternoon and finished. Then to try out things I was starting in the evening again the incremental job by manual, let him finish (took him 10 minutes), and then started by manual the corresponding tape job.
Normally, he should find new vbi file(s) and synthesize completely a new vbk file and write this to tape.
In this weird case, he JUST was backing up the vbi file to tape, and wrote in this job:
26.11.2014 23:05:28 :: Processing full backup of ....SERVERS-BACKUP: nothing to backup
26.11.2014 23:05:28 :: Processing incremental backup of ....SERVERS-BACKUP
26.11.2014 23:12:16 :: 0 directories and 1 files backed up successfully

[veremin]

Synthesized backup is created once per day. So, it seems that in your case it had been created on that day already, and therefore the subsequent run copied only incremental files. Thanks.

[Dima P.]

Wolfgang,

You setup looks correct - we had a discussion with QA team and they asked you to open a support case, please, let support team know that QA is interested in your issue and post the case ID to this thread.

Right now it looks like bug: if the synthesized full was already but then a new increment appears tape job would be triggered to create another synthesized full. The workaround for now is to adjust backup to disk / backup to tape schedule so backup to tape would start only after the last backup to disk incremental run.

[wbauer84]

Hi d.popov and v.Eremin!

The case# under which THIS problem is reported to support is: Case # 00680258

Hmm, I was checking the logs now again.
So we still have the "process incremental files" checkbox checked in the tape job!
Now I can guarantee, that the tape jobs were started JUST once a day (with the appropiate days checked in the schedule box above "process incremental files"), and definitly started after the corresponding incremental job (which runs 3-4 times over the day) finished for this day making incremental backups.

Result in two different jobs, with the same settings in one environment:
Job-no 1:
It seems now, he writes one vbk-file, with the timestamp of the last incremental backup in this day, BUT ALSO writes ALL the corresponding vib-files from this day to tape!
And we thought, with this setting (process incremental files) he is just supposed to write one synthesized vbk-file (on these days selected in the schedule box above the "process incremental files" setting)?!

(please don't wonder in the first screenshot is the 27th and not 28th in the tape files properties, the tape job for 28th I paused to start until I made this screenshots for you, for testing purposes, with all this problems....

And as a poster before mentioned already too: when the box "process incremental files" is checked, he backups more files to tape than just ONE synthesized vbk, with all incremental of this day..., at least all the vib files too, and as we can see in job-no2, a little bit more....[/b]


So, what I tried out (because he didn't processed the files of 28th because of pausing the job, until now), I took away the "process incremental files" checkbox in the tape job STOR-SERVER, but left ALL days in the schedule box above checked, made again by manual a incremental backup (with date 29th - Saturday) with the result, as soon the incremental job ended and I started the tape job (manual), that he tells "no new files to backup per job setting" again.
So
27th the last vbk-file was written in the evening on tape, (even with all the vib-files, which should not be), but "process incremental files" checked
28th he was makeing incremental jobs too, as you can see in the screenshots, but I paused the tape job, which was just not running on the 28th
On 29th we made him a manual incremental backup by hand (becaus schedule just makes Mo-Fr), and when I took away the "process incremental files" job, he says "nothing to backup", when I start the tape job by manual, after the incremental finished.




Job-no 2:
There he is makeing something very weird. He writes everything like in Job-no 1, but ALSO produces another synthesized vbk-file (2014-11-21....vbk), with the OLDEST incremental file available (I was checking the job, it's not an old vbk he left on tape because he has enough space, NO, he was creating/synthesizing it, in the job, running on 28th!!...
There "process incremental files" was also checked!


The case# under which THIS problem is reported to support is: Case # 00680258
(Please don't wonder: The case was openend in the beginning because of a read error from USB disk, but this is solved already with http://www.veeam.com/kb1887, but we continued haveing problems with the vbk-synthesizing, so this problem is now the actual one in this case!)

[wbauer84]

RESULT FOR US
Leaving the "process incremental files" box unchecked, leads NOT to get synthesized vbk-files in 3 jobs (2 environments), EXCEPT in the meantime by HDD repository retention policy, he was migrating a vib into the vbk file on the HDD, because the number of allowed vib came to the end, then he recognizes a new vbk file, he backs up then. By the way, leave this checkbox unchecked to be the right method, is still the opinion from your co-worker, which supports us in this case ID!! So against your opinon
Checking the "process incremental files" box, leads to, that he makes syntesized VBK files, even with the right date, but also writes all the VIB files from this day to tape, and in the job-no 2 case, he writes an additional syntesized VBKfile (with the oldest VIB date).
At a second customer (case # 00689913), the "process incremental files" box checked (the same like here), leads that he never stops producing many syntesized vbk-files on tape, so the tape job never stops, even the job starts to run in the evening, when all incremental stopped running.
This seems to be a little bit the behaviour like in this screenshots job-no 2 here, but in this case we have a tape library and not a singular drive (maybe as long he has enough tapes with space online in the media set, he really produces syntesized vbk files, with each of the existing vib files?!?!)

Anyhow, your workaround of waiting to start the tape job (with process incremental checked) until the incremental jobs finished, works for the customer of case # 00680258 which has singular tape drives, even he writes the corresponding vib-files of the day too, and if he has still space on the tape, another vbk file with another vib state synthesized, which should not be the right behaviour, right?
But this workaround doesn't work for the second customer, where he sees a tape library, there he writes vbk files with syntesized vbi states, as long he has writeable tapes available (we never were waiting longer than 35 hours, what would happen, if he runs out of tapes, but maybe he will finish the job "happy)..

[florian.meier]

Hi

Im now a littlebit confused.

If i have a normal incremental veeam backup copy job, from which i save the data to tape i have to do this:

Uncheck the Box "Process incremental backup files" and veeam should always create a Virtual Synthesized Full Backup to tape.
So that means, only one full bkp file gets written to tape, no incremental files!

But this works also, if on my backup copy target is a chain with a full backup one week ago and some incremental files on it.
The tape job capture all the files (full and incrementals) and write one big full to tape, right?

Please correct me if I understand something wrong.

Regards.

[veremin]

It seems now, he writes one vbk-file, with the timestamp of the last incremental backup in this day, BUT ALSO writes ALL the corresponding vib-files from this day to tape!
And we thought, with this setting (process incremental files) he is just supposed to write one synthesized vbk-file (on these days selected in the schedule box above the "process incremental files" setting)?!

Yes, it's supposed. However, currently there are problems with how synthesized full is generated in cases when multiple increments are created daily. As mentioned above, QA team is aware of this issue and working on addressing it.

Uncheck the Box "Process incremental backup files" and veeam should always create a Virtual Synthesized Full Backup to tape.

In order for synthesized full to be created, process incremental backup files option should be enabled.

Thanks.

[wbauer84]

Hi v.Eremin,

Well, the problems with creating synthesized full backup we experienced also before, when we had just ONE incremental backup a day, it was the same behaviour, but in the meantime we switched already to more than one incremental a day, so we cannot proof it anymore...

So for synthesized full to be done (even there are problems now), the "process incremental backup files" has to be ENABLED now (all the time you're saying NO)?

Is there are time range we can expect the hotfix, because this bug doesn't allow us to make full backups every day, without backing up every day full from source (which was the reason we bought Veeam, because it's one of his main features)?

[veremin]

So for synthesized full to be done (even there are problems now), the "process incremental backup files" has to be ENABLED now (all the time you're saying NO)?

This requirement is by design. The synthesized logic will be applied only when the said option is enabled.

Is there are time range we can expect the hotfix, because this bug doesn't allow us to make full backups every day, without backing up every day full from source (which was the reason we bought Veeam, because it's one of his main features)?

The fix for this issue won't make it into the first patch. However, I've asked QA team to take a look at the mentioned cases. So, we'll create a private fix and you should be provided with it, as soon as it's implemented.

Thanks.

[veremin]

Case # 00680258

QA team's made internal requirement regarding that problem and confirmed that the private fix should provided to you, once it's ready. Thanks.

[PariPari]

Hey,

I have some questions about the to tape functions in V8
Currently we are backing up with Incremental forever, with a synthetic full on thursdays. We would like veeam to export a daily full to tape.
The Export Job is configured with a media pool for fulls (process incr is not checked). But i dont get how to configure the schedule next to that option so the to tape job will do a synthetic full for tape export... can someone please clarify this?

[hoFFy]

I'm adressing nearly the same behaviour in case #00685887 but I'm getting answers there very very slowly....

Source backup job creates forward incremental files once a day in the evening and creates active full backups weekly on mondays.
The backup to tape job runs in the morning on the next day (to be able to change tapes, if capacity of one tape is exceeded) with schedule for full backups set to every day (thought this will create a full backup on tape every day) and also checked "process incremental backup files".

It seems that the schedule for creating full backups on tape is ignored, because it only writes .vib files to tape:
Folder with source backup files contains a .vbk from 12/03 and .vib files from there on once a day

The tape job from 12/04 wrote the .vbk from 12/03 created at 12:00 and one .vib created the same day at 19:00.
Tape job from 12/05 wrote only a .vib file from the day before and this morning there have been written the three .vib files created between friday and sunday.

But no full backup has been created...!

This is very importand for us because we have no correct backup on tape, too

[foggy]

The Export Job is configured with a media pool for fulls (process incr is not checked). But i dont get how to configure the schedule next to that option so the to tape job will do a synthetic full for tape export... can someone please clarify this?

Basically, you need to specify the day you want synthesized backup to be created on and enable processing of incremental files. However there are currently issues related to synthesized full backup creation, please review the thread above for details.

[PariPari]

Alright thanks. I checked 'process incremental backup files' now and can see the same behaviour as mentioned already in this topic. According to the job log only vibs will be copied to tape.
"Processing full backup for Job xy: nothing to backup"
Its a pretty new installation, so we will tell the customer to wait for a fix.

[foggy]

If you open a case with our technical support, you will be promptly notified on its availability.