Reverse Incremental Compact Bug

[Vek17]

I have identified an issue with reverse incremental backups and compacts that can damage a chain and make a backup job un-runnable.

Requirements:
Reverse Incremental backup job with scheduled compacts.
Cause:
A file lock is acquired by another process on a .vbk during the compact causing the job to fail with this warning:

Code: Select all

2/20/2019 8:22:01 AM :: Failed to compact full backup file Details: boost::filesystem::rename: The process cannot access the file because it is being used by another process: "D:\Backups\ReverseIncrementalTest_4-6\VM-04.72bd94a4-d6aa-477a-9c8e-22d9df34787c_3840D2019-02-20T081845.vbk", "D:\Backups\ReverseIncrementalTest_4-6\VM-04.72bd94a4-d6aa-477a-9c8e-22d9df34787c_3840D2019-02-20T081845_to_delete.vbk"
Failed to rename file [D:\Backups\ReverseIncrementalTest_4-6\VM-04.72bd94a4-d6aa-477a-9c8e-22d9df34787c_3840D2019-02-20T081845.vbk]

AfterMath:
Subsequent runs of the job will fail in this manor as the job does not appear to understand that there is a new .VBK from the last job run and attempts to use the previous .VBK (which is now a .VRB)

Code: Select all

Error: Last storage: VM-04.72bd94a4-d6aa-477a-9c8e-22d9df34787c_9B1DD2019-02-19T115027.vrb must be VBK 

Worth noting that recovery still works within the chain, only backups fail

I have been able to reproduce this on demand in my test environment and I have had it impact several jobs in production in the past. Current solution I have gotten from Veeam support is active full the job. I have thus far only test with per VM backup files so this may or may not be possible with a single backup file.

[foggy]

Thanks for reporting, doesn't look like correct behavior. We will try to reproduce internally and, if there's an issue, fix it in one of the future updates.

[Vek17]

Views in order are:
[Backups.Model.Points]
[Backups.Model.Storages]
[Backups.Model.OIBs]

I have a further explanation of what is happening and how it can be resolved.

It appears that the storage_id in the [Backups.Model.OIBs] table is linked to the incorrect storage (it is linked to the .temp file created by the merge that has no links to other storages). If you edit the storage ID to the previous .VBK the backup is able to run as expected without issues. So it appears the issue is when the compact fails in this manor it leaves links to the .temp storage intact in the OIBs which prevents Veeam from understanding the chain.

[foggy]

Thanks! Looks like an easy fix then.

[Vek17]

I encountered another version of this in a Backup Copy job running GFS retention. While this is not a reverse incremental job, I am including it as this appears to be the same broken compact behavior that caused this issue. This resulted in different error messages and job behavior but had a similar work around.

The backup copy successfully uploads points but fails when it tries to merge points after data transfer completes.

Code: Select all

Failed to merge full backup file Error: Storage ('b6ae9f54-2777-42d8-b624-27bce4396bb5', CreationTime '4/3/2019 7:00:00 PM') not found
Failed to generate points Error: Storage ('b6ae9f54-2777-42d8-b624-27bce4396bb5', CreationTime '4/3/2019 7:00:00 PM') not found 

This is the database view of this backup chain from the cloud connect side.


I ended up just renaming the .temp files on the filesystem and altering the names within the database to get the job to run again as a workaround.

EDIT: This is running Update 4a on cloud connect (9.5.4.2753) and Update 4 (9.5.4.2615) on the tenant B&R

EDIT 2: I believe this has been internally classified as a bug that is scheduled for Update 5 for a fix but I wanted to add more information to the behavior in case it is relevant. Internal support case #03427920 (Closed Case)

[foggy]

Yes, this was investigated and scheduled for fixing in one of the upcoming updates. Thanks for providing more details though, much appreciated.