Backup to Tape Performing Unnecessary Extra Full Backup

[Princes]

Case ID: 00821217

Just to check I'm not going mad and this is not expected behaviour:

I have recently migrated our tape backup system from Backup Exec 2010 to using the Veeam 8 Tape Backup architecture. To that aim, I have created a Backup to Tape job for each of my existing Backup to Disk jobs, which is set to run after each Backup to Disk job completes. Each Backup to Disk job is of the Incremental Type, set to creating a synthetic Full every Friday PM and to Transform previous backup chains into rollbacks. This means that a new .VBK Full Backup file is created for each Backup to Disk job every weekend, and a new .VIB Incremental file is created Mon-Thur.

I have configured the Backup to Tape job to run after each Backup to Disk job, which on a Friday will pick up the modified .VBK and send it off to the Full Backup tape pool as desired. During the week (Mon-Thur), the latest new .VIB is picked up and sent off to the Incremental Backup pool instead. I have enabled the 'Process incremental backup files' option in the job.

Everything works fine until after the Friday full backup to tape has completed. The .VBK is sent off to tape as expected after the synthetic full and transform process has completed as part of the Backup to Disk job. However the next run of the tape job on the Mon PM results in both the latest new .VIB for that day being backed up, as well as yet another copy of the unchanged, unmodified .VBK that was already backed up successfully to tape when the job last run. Curiously, this only seems to happen for some of the tape jobs, all of which are configured the same way. Different jobs are also affected by the issue each week - others back up just the latest .VIB for the Mon as I would expect. I have checked to see whether the .VBK's have been modified in any way and all retain the timestamp from when the last Backup to Disk job completed and the Full Backup to Tape job ran afterwards. The file appears unmodified in any way - the byte count is also the same - since the Backup to Disk then to Tape.

If I allow this additional necessary backup of the .VBK to take place and finish successfully on the Mon, subsequent backups seem to work fine and the issue goes away, each job committing just the latest new daily .VIB to tape as expected. The unexpected behaviour then returns again for different jobs after the next weekend (Fri) full. The issue therefore only seems to affect the first Incremental tape job run after the last Full Backup to Tape. I would rather not have this behaviour since I cannot afford the backup window time and additional tapes for an extra unneeded Full backup to tape every Monday when I only need the new Incremental file committed to tape.

Am I going mad?

[vmff]

I also am leaving from Backup Exec 2010, after a failed migration to Backup Exec 2012, and just barely bought Veeam after the 30 day trial seemed to be working fine. I'm now experiencing the exact same issue, with a variation on the original job being reverse incremental, but I only want the most recent .vbk on the tape, which shockingly is seemingly not configurable?? I asked support and they said, nope, VB&R can't do that. Huh? You can't backup a full backup to tape each day without issue? Real big problem for me too.

Their suggestion is to try and get someone on the forums to write me a script, however, there are issues with that too, but guess that's my only option now that I've paid them for this.

[Princes]

With the Reverse Incremental backup mode, does the job not just modify the same .VBK full back file each day, injecting the incremental changes since last backup into the file? In which case, the backup to tape job should be identifying the .VBK file as modified each day and trying to back it up?

My issue is slightly different in that I am using Incremental with a weekly synthetic full created on Fri. The rest of the week (Mon-Thur) is just .VIB incremental files. However the backup to tape job seems to be identifying the .VBK as having been modified since its last successful full backup on Fri after the synthetic full job is completed - which it has not according to bytecount and timestamp....

[vmff]

That is how it works, except when an active full is run, then it starts the reverse incremental chain over again from the latest active full, but leaves the previous reverse incremental .vbk as one of the restore points for the retention policy.

VB&R doesn't seem to be identifying whether the full has been modified for me, I've seen people reference that it should, but instead (with incremental backups ignored on the tape copy job) all full backups in the repository are being copied out to tape each time. I'm only running an active full once a month, so I suppose this problem will go away when I reach my 15 day retention period, but that means half the month I'm backing up twice as much data as I'd like to.

This is a real problem for me, because our tapes are virtual and replicated offsite, so that process is killing our DR connection.

Still haven't had luck finding a script that will work for this, wish they would just add a check box to only back up most recent full backup or something...I might have to go back to Backup Exec to do it, but not really wanting to put a BE agent on the Veeam server.

[Princes]

Issue reoccurred again last night as expected (first attempted Incremental to tape after the last successful Full to tape), with only 3 jobs working correctly this time, the rest once again incorrectly identifying the last synthetic full .VBK as having been modified since the last successful back to tape, and thus adding it to the Incremental backup to tape job. I have once again checked the .VBK file to ensure that it was indeed successfully backed up to tape during the last backup to tape job and that the file itself has not been modified in any way since. It would seem possible that a bug exists in the internal logic that identifies whether the file has been modified, or perhaps the application is not correctly storing the metadata concerning the last successful full to tape and so is not aware that there has been one and tries to backup the same file once again.

Not getting anything from support who have missed their remote session appointment today as well.... Is there anyone else experiencing anything similar to this where backup to disk files are being incorrectly identified as modified by a tape job causing them to be backed up twice (though only seemingly after the last successful full to tape - if this second backup of the full .VBK file is allowed to complete, the issue corrects itself with the next backup)?

[Dima P.]

Hello James,
I am sorry for delay in response. Let me summarize your issue and setup:

- Source job is incremental with periodic synthetic Full backup (plus rollbacks)
- Backup to tape is scheduled to run ‘After the source job’
- This second unexpected .vbk is placed on tape only during the Friday backup to tape job run

Need to know was your VBR patched with patch1? In addition, how your disk and tape backup job retention was configured? Also, what media set creation option is used in target media pool?

Thank you in advance.

[trustblomme]

Hello,

is there any update on this problem? I have a simular problem and Veeam is eating all my expensive LTO tapes.

Case 00814113

The implemented dll fix worked for 1 week and then the problem returned.

The job dedects a change on file level and rewrites every vib to tape:

11/03/2015 21:33:25 :: Modified storage E:\Backup\XXXXX Backup\XXXX Backup2015-03-09T210057.vib on This server was added to backup list -> this entry returns for every VIB file.

Please advise,

Thanks,

Steven

[TommyB]

The implemented dll fix worked for 1 week and then the problem returned.

The job dedects a change on file level and rewrites every vib to tape:

Had the same issues, but in my case the DLL fix worked - so there seems to be some variation in whatever causes this...

[Princes]

Hi,

This issue has (likely) been rectified. I will know for sure after the first Incremental Tape backup has run on Monday 16th PM.

Veeam Support remoted into our Veeam B&R Server and identified the issue as one that they are aware of. A new "Veeam.Backup.Core.dll" was supplied with a build date of 15/02/2015, replacing the existing v8 Patch 1 build date of 19/12/2014 version. After all Veeam B&R services were restarted, the issue appears resolved when the tape jobs are rerun, with just the latest Incremental .VIB files now being selected for backup and the last, already backed up and unmodified synthetic Full .VBK no longer being flagged as modified and added to the job.

If this desired behaviour remains after the latest new synthetic Fulls have been compiled over the weekend, then I will be 100% sure the issue has been resolved.

I was told that this fix is in the forthcoming Patch 2 for v8. If you need the fix before then I would contact support and request the latest .dll.

Thanks.

[mfo]

Hello,
Issue: Unmodified full backup files are repeatedly written to tape unnecessarily.

We use Veeam Backup 8.0.0.917 with disk and tape backups.

Our backup scheme is Forward incremental Backup with syntetic full backup and rollback transformation on Friday. So on Saturday, when Full Tape Backup is scheduled, I have only the fresh *.vbk and *.vrb file in my repository.
These files are than written to one a half LTO5 tapes, which gives me just enought free space on the second tape to write the upcoming incremental backup files during the following week.
The first backup per week runs as expected and all *.vbk files are written to tape.

But lately, the *.vbk full backup files are repeatedly written to tape the following days, which consumes a lot of unnecessary tape space and I'll end up with 5 tapes instead of 2-3 for a weekly media set.
I create a fresh media set just before the full tape backup starts on Saturday in order to have one single media set per week with the full backup and incremental files.
From my point of view, the *.vbk files are unmodified, so there should be no reason to write them again.

Creation of the syntetic full backup and rollback transformation is finished 5h prior to the start of the full tape backup, so there should be no interference.
Full tape backup is only scheduled on Saturday, incremental tape backup is enabled. Tape backup job is scheduled from Monday to Saturday.

Any suggestions?

[foggy]

Christoph, this looks like a known issue, please contact support for the hotfix.

[trustblomme]

Hi,

Contact provided me with a new dll that seems to fix the problem for now.

Thanks Veeam Support!

Steven

[bertdhont]

We see kind of the same problem, in one specific case with forward incremental and active full backups.
Situation:

• Daily backup job to disk, forward incremental, active full once a week

• Tape job to process full and incremental backups of that backup to disk job

I start a first backup, active full.
After that one is finished, I start an incremental backup to disk
When this one is finished, I start the tape job. This one finds a full and an incremental backup to put on tape (2 different media pools).
During the tape offload, I start another incremental backup.
After the full is written to tape, Veeam notified that another .VIB was created, so it was added to the list of files to put on tape.
BUT, the strange thing is, it also notifies that the full (VBK) is modified and the VBK is written once again to tape....

I don' t understand why that full is modified, because we are using a forward incremental backup.

The log of the tape job below:

19/03/2015 10:28:56 :: New storage E:\VBRRepository\testBDHO\testBDHO2015-03-19T095624.vbk on This server was added to backup list
19/03/2015 10:28:56 :: New storage E:\VBRRepository\testBDHO\testBDHO2015-03-19T101058.vib on This server was added to backup list
19/03/2015 10:28:56 :: Queued for processing at 19/03/2015 10:28:56
19/03/2015 10:28:56 :: Required backup infrastructure resources have been assigned
19/03/2015 10:28:57 :: Media pool Media Pool test BDHO locked successfully
19/03/2015 10:28:57 :: Drive 2 (Drive ID: Tape1) locked successfully
19/03/2015 10:28:57 :: Loading tape ID0578L6 - Mei / Nov - B from Slot 18 to Drive 2 (Drive ID: Tape1)
19/03/2015 10:30:37 :: Current tape is ID0578L6 - Mei / Nov - B
19/03/2015 10:31:07 :: Starting new media set Media set 19/03/2015 10:31
19/03/2015 10:31:09 :: New tape backup session started, encryption type is off
19/03/2015 10:31:34 :: Processing full backups started at 03/19/2015 10:28:55
19/03/2015 10:36:48 :: Backup completed successfully
19/03/2015 10:36:49 :: New storage E:\VBRRepository\testBDHO\testBDHO2015-03-19T103215.vib on This server was added to backup list
19/03/2015 10:36:49 :: Modified storage E:\VBRRepository\testBDHO\testBDHO2015-03-19T095624.vbk on This server was added to backup list
19/03/2015 10:36:49 :: New storage E:\VBRRepository\testBDHO\testBDHO2015-03-19T101058.vib on This server was added to backup list
19/03/2015 10:36:49 :: Changes in source storage detected. Scan for changes
19/03/2015 10:45:19 :: Media pool Media Pool daily bdho locked successfully
19/03/2015 10:45:19 :: Loading tape ID0575L6 - Apr / Okt - A from Slot 15 to Drive 2 (Drive ID: Tape1)
19/03/2015 10:47:01 :: Current tape is ID0575L6 - Apr / Okt - A
19/03/2015 10:47:30 :: Starting new media set Media set 19/03/2015 10:47
19/03/2015 10:47:30 :: New tape backup session started, encryption type is off
19/03/2015 10:47:55 :: Processing incremental backups started at 03/19/2015 10:28:55
19/03/2015 10:48:08 :: Busy: Source 3% > Proxy 3% > Network 5% > Target 99%
19/03/2015 10:48:08 :: Primary bottleneck: Target
19/03/2015 10:48:08 :: Network traffic verification detected no corrupted blocks
19/03/2015 10:48:08 :: Processing finished at 19/03/2015 10:48:08
19/03/2015 10:48:08 :: 0 directories and 4 files backed up

[Dima P.]

Hello Bert,
Steven confirmed above that we have a working hotfix at the support side for a situation like yours. Could you please contact support team and let us know if the existing solution works for you. Thanks in advance.

[krogerss]

Can anyone confirm if this fix is included in Update 2? I believe I'm experiencing this very issue and am looking to attain this hotfix. Thanks.

[trustblomme]

HI,

We still have this problem in Update 2. Support is looking for a solution.

Steven

[krogerss]

That sucks . Thanks.

[florian.meier]

we also using p2 and veeam v8. when we creating a virtualized syntetic full tape backup, veeam always takes the full of the increment forever full backup and also the syntetic full vbk of the choosen scale to tape. so we have on every tapte two vbk files.
thats maybe the same issue than yours.

[veremin]

So, you're running backup to tape job with virtual full backup enabled against forward incremental chain with regular synthetic full backup, not against forward forever incremental chain, right?

[florian.meier]

no i run virtual full backup enabled against forward forever incremental chain!

[florian.meier]

same issue here: http://forums.veeam.com/tape-f29/foreve ... ml#p148644

[trustblomme]

Any update on this problem?

Thanks,

Steven

[florian.meier]

Im also interested. Is this a known problem for veeam?
Are you planning to bring a hotfix or patch in next minor?

Thank you

[Dima P.]

Hello guys,

In the existing version with Update 2 you may recognize the following behavior while backing up to tape forever incremental backup chains. Once you configured the backup to tape job it picks up the whole backup chain starting from the first backup. In addition, it will create one virtual full back up on tape for the specified date (synthesized full backup) to cut off further incremental backup files from the existing full backup and re link them to the new virtual full backup.

For now it is by design, but if we recognize this behavior needs to be adjusted - will provide some additional functionality in one of next major release. Thank you.

[trustblomme]

Hi,

Problem was gone for 4 weeks but has returned .

Dima,

If you don't use forever incremental backup chains you can avoid this setting by design?

Thanks,

Steven

[veremin]

Problem was gone for 4 weeks but has returned .

Are you positive that your source backup chain isn't longer that the one you have on tapes? In other words, is the retention of backup jobs shorter than the retention of backup to tape jobs?

If you don't use forever incremental backup chains you can avoid this setting by design?

Yep. Virtual full backup is created only for jobs using forward forever incremental mode. Those are backup jobs (configured that way) and backup copy ones.

Thanks.

[trustblomme]

Thanks for the reply!

-> Problem only occurs for certain backups in the tape job not for all. The job dedect an change on all the vib files and tries to put them back on tape.
-> Retention on disk is 30 days vs 6 weeks for tape.

Steven

[veremin]

Problem only occurs for certain backups in the tape job not for all. The job detect an change on all the vib files and tries to put them back on tape.

So, your issue appears to be a different than the one discussed in this thread. Your backup to tape job copies extra incremental files that are already present on tape mediums, not full ones.

If you're on the latest product version, and positive that the said files already present on tapes (and should not be copied to them again), please, open a ticket with our support team and let them revise your deployment.

Thanks.

[Shunx239]

Continue this thread since it seems to be always about the same problem.
Opened case #02674684:
Two D2T jobs protecting forever forward incremental chains without any planned fulls, at certain point copied/tried to copy entire chain again (duplicating restore points on the tape).

In the job log the restore points already on tape are seen in some way modified.

Looking further in the TapeVmBackup.log one can see for regular run:

Code: Select all

[CTapeBackupJobStorageBuilder] Building storages...
        [CTapeSourceBackupJob][Afrodite B2D] Getting repository storages...
        [CTapeSourceBackupJob][Afrodite B2D] Found 10 source storages: 0 sec
[CTapeSynStorageCandidatesFinder] Building virtual full storages for objectId: 00000000-0000-0000-0000-000000000000
[CSimpleSourceBackupJobSpecific][Afrodite B2D][Backup] Job is forever incremental because, - it is backup forever incremental job
<...>
            [CTapeSourceBackupJob][Afrodite B2D] Last backed up full tape storage creation time: 17/08/2022 00:00:00, backupId: abb106e5-8cc6-46b5-be2f-5e5e9e64d40d, objectId: 00000000-0000-0000-0000-000000000000 mediaSet: Simple
[CTapeMonthlySynthesizedBackupCalendarSpecifics] Virtual full backup dates between 18/08/2022 00:00:00 and 29/08/2022 22:43:23: 
            [CTapeSynStorageCandidatesFinder] [Simple:00000000-0000-0000-0000-000000000000] Virtual Full is not created: no synthesized dates found between 18/08/2022 00:00:00 and 29/08/2022 22:43:23 for source job [Afrodite B2D]
    [CTapeSynStorageCandidatesFinder] Found candidates:<Empty>
[CTapeSynStorageCandidatesFinder] Build virtual full storages for object: 00000000-0000-0000-0000-000000000000 is completed: 0.0526927 sec
[CTapeStorageMap] Tape point (Id:c841bbdc-c5f7-4518-86c6-2e1d06f35bb1) Created On '16/08/2022 21:00:29' was found for tape storage Afrodite B2DD2022-08-16T210029_2C56.vbk. Storage Creation Time 16/08/2022 21:00:29
[CTapeSourceBackupJob][Afrodite B2D] No need to (re)backup current vbk Afrodite B2DD2022-08-15T210044_18F8.vbk because it is forever incremental job and newer vbk already backed up in tape
<...>
[CTapeStorageMap] Tape point (Id:c841bbdc-c5f7-4518-86c6-2e1d06f35bb1) Created On '16/08/2022 21:00:29' was found for tape storage Afrodite B2DD2022-08-16T210029_2C56.vbk. Storage Creation Time 16/08/2022 21:00:29
[CTapeStorageCandidate] Skipping [Afrodite B2DD2022-08-16T210029_2C56.vib:bf3a868b-3363-4750-baad-650c5d2edbd2:5d150977-aea9-496b-84b4-00a9a3b00f49:16/08/2022 21:13:18:16/08/2022 21:00:29:KbBlockSize512:65536] (created on 16/08/2022 21:00:29) because a younger VBK has already been backed up
[CTapeStorageMap] Tape point (Id:c841bbdc-c5f7-4518-86c6-2e1d06f35bb1) Created On '16/08/2022 21:00:29' was found for tape storage Afrodite B2DD2022-08-16T210029_2C56.vbk. Storage Creation Time 16/08/2022 21:00:29
[CTapeStorageCandidate] Skipping [Afrodite B2DD2022-08-15T210044_18F8.vbk:5d150977-aea9-496b-84b4-00a9a3b00f49::15/08/2022 21:00:44:15/08/2022 21:00:44:KbBlockSize512:65536] (created on 15/08/2022 21:00:44) because a younger VBK has already been backed up
[CTapeBackupJobStorageBuilder] New backup Afrodite B2DD2022-08-29T210024_F006.vib will be placed into the media set 

On the next day it is the first time that initial full gets merged with incremental and the new VBK is not recognized anymore:

Code: Select all

Get latest vbk
Got latest vbk old way
Storage with id 5d150977-aea9-496b-84b4-00a9a3b00f49 not backed up to tape detected
[CTapeSourceBackupJob][Afrodite B2D] Backing up vbk Afrodite B2DD2022-08-16T210029_18F8.vbk:5d150977-aea9-496b-84b4-00a9a3b00f49::16/08/2022 21:00:29:16/08/2022 21:00:29:KbBlockSize512:65536 again because last vbk is not backed up yet.
Storage with id 5d150977-aea9-496b-84b4-00a9a3b00f49 not backed up to tape detected

And the same VIBs are put to the tape again, together with the merged VBK, that is from the same chain

Code: Select all

[CTapeBackupJobStorageBuilder] New backup Afrodite B2DD2022-08-30T210045_AE5C.vib will be placed into the media set 
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-29T210024_F006.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-28T210039_6E90.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-26T210052_6F4E.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-24T210137_B2EB.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-22T210050_BAA0.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-19T210047_B0C0.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-18T210038_6359.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] Modified backup Afrodite B2DD2022-08-17T210034_603E.vib will be placed into the media set
[CTapeBackupJobStorageBuilder] New backup Afrodite B2DD2022-08-16T210029_18F8.vbk will be placed into the media set 

If this is by design, the design is poor.

[david.domask]

Hi @Shunx239,

I'm reviewing the case now, as with Forever Forward Incremental, there are only a few exceptions where the VBK might be re-backed up, and I suspect the engineer perhaps has a little confusion.

Look forward to an update on your case.