3 Backup Copy Jobs has deleted all monthly GFS after Upgrade to Veeam V11

[Mildur]

Hi there

Case # 04727573

We have upgraded to Veeam V11 on 19th March to GA Build.
Today, I have found out that the first Backup Copy Job run after the upgrade on 19th march has deleted all monthly gfs on three of my jobs. GFS is configured for 13 monthly RP. All _M.vbk were deleted on that day.

Support isn‘t getting back to me. I ask for answer on my.veeam.com. Nothing. Tomorrow I will escalate the case.
I need to be sure, that this is some issue only in this three jobs and that my second backup copy job doesn’t delete my gfs monthlies too.

[foggy]

Hi Fabian, we are not currently aware of any issues with monthly backups retention. Please escalate the case and provide full logs for closer investigation. As a precautionary measure, it is recommended to temporarily increase the number of restore points in the regular chain or even disable the affected jobs (or those that might be affected) until we identify the reasons. Thanks!

[Gostev]

Sorry to hear about this. It's very strange indeed as in V11, the Backup Copy GFS retention is time-based. So very "dumb" and straight-forward, in fact much less bug prone than the one we used before. It's gotta be some corner case I think - we will get to the bottom of this to understand all the dependencies, to give you the definitive answer about the other Backup Copy job.

[Mildur]

Thanks guys.
I have now an answer from support. I think, changing the retention will not help.
Very critical bug. Every GFS Restore Point gets deleted, if the last Full Backup is corrupted.

Based on the log snippet from above it looks like we have met a known issue when GFS retention points are deleted when the last full backup file is corrupted. Our R&D team is already aware of this issue. Currently, there is no fix for this.

I'm not sure, that I want to leave my healthy backup copy Job active for the moment, until I have some months of backup copies from the first job.

[Steve-nIP]

I don't know the exact ins and outs of the case, but that seems like a pretty critical issue.

[Mildur]

Restore Points of all three Jobs were stored on a NetApp E-Series in a second location.
It's connected per iSCSI to a windows server. I think, the corruption was because of that connection.
We will change that this summer to Linux hardened Repo with local disks and S3 Capacity Tier.

The Corruption is ok. Nobody likes it, but it can happen. For that, a second Copy is the way to be safe.
But it should not be responsible to remove all GFS RPs, corrupted or healthy ones.

[foggy]

Increasing the retention will help to avoid new deletions until the new (good) full is created. R&D is already working on a fix and detailed instructions on how to prevent data loss until it is ready.

[Mildur]

Thanks Foggy. I have disabled the not affected backup copy jobs for the moment.
I'm waiting for veeam support until I activate them again.

[Gostev]

So as I suspected, it's a corner case due to the last full backup corrupted in a certain way. From the summary of the issue I've got from developers, the easiest way to ensure you won't run into the issue is prior to upgrade, open backup properties under Backups > Disk (copy) node, and ensure the last full backup is not marked as Incomplete or Corrupted. This ensures you won't run into a condition that triggers the bug.

They will work on fixing this with high priority, and we will issue the new cumulative patch and the new ISO with it, to ensure anyone who's upgrading in future is not impacted even when their last full backup is corrupted.

[Mildur]

Yes, I had canceled "Backup Copy Merging process" to do the upgrade as planned in our maintenance window.
For both Jobs with the deleted GFS RPs.

That was never a problem before, if I had to cancel a backup copy job immediately to shut down a server or to do an upgrade. Normally, the next backup copy Job interval after the upgrade would have continued without any problem.

[Gostev]

To be perfectly honest, we did already have upgrade bugs caused by corrupted backup files some years ago, which is exactly why the upgrade procedure says since then:

2. Ensure there are no active processes, such as any running jobs and restore sessions. We recommend that you do not stop running jobs and let them complete successfully instead.

The main reason here is that it's virtually impossible for our QC to have all possible non-standard scenarios reproduced and tested in the test lab... they WILL miss some corner cases. Ideally, developers of course should just create the reliable code that can handle all unexpected situations in a safe manner, but human errors are always possible.

[Mildur]

Yes, I know. Next time, it won't happen for sure
If it was important that I had to quit the job immediately, I've just quit the job until now.
I will never again cancel a job if it's in the merging phase.

I will clone my volumes to the second data center. This way, I have a second backup Copy again, if I need them Sometime.

In Q3 we will have redesigned the entire veeam infrastructure with Linux hardened repo and S3 Object storage. With the immutable flag on memory, we should be more secure against such a bug.

[Gostev]

Funny you should say this, because this thought has crossed my mind already a few times this year: how this is one other benefit of immutable backups. We always keep thinking about ransomware and hackers, but what about accidental deletions and data loss bugs?

I recently had to systematize the likelihood of various backup data loss events, and based on our support statistics we believe it is the following:
1. (by far) Storage-level or network-level data corruption aka bit rot, entire volume or entire storage loss.
2. Deletion by malicious actors: these guys purposely look for backups to destroy them.
3. Encryption by ransomware: while much more common, it does not get to backups too often due to the lack of intellect (so even minimal separation of backup infrastructure from production environment helps). However, it can let hackers in, then see 2.
4. Accidental deletion by user.
5. Software bug.
6. Accidental deletion by support engineer < although I think support folks added this more as a joke, because it was crossed out

So basically, immutability protects against all issues except the most common one: with storage or network hardware. Which is exactly why maintaining a copy of backups on a different media is essential whether or not you use immutability.

[Mildur]

Thank you Anton, for Showing us the data loss ranking.
Immutability will definitely be a good protection against events 2-6.
we don't hope for Event 6

[mcz]

Anton, (thanks God) it's not exactly the same case as here but I've also experienced a strange situation with the GFS's. Normally, I'd get a full per week and this was the case on the 06th and 13th. The R has the timestamp of the 20th. I've upgraded to v11 somewhen within the last 2 weeks, didn't change retention or such stuff and expected the next weekly full from the 20th and 27th but the following happened:

13th W
20th R
....
....
....
3rd WM

So we "lost" at least the 27th... Is that expected?

[foggy]

Hi Michael, looks differently, indeed, as in your case nothing has actually be deleted but rather not created. I'd contact support to verify whether this is a side effect of the same issue or some other one.

[Mildur]

Looks like the patch is released:

https://www.veeam.com/kb4126

P20210401

Following the upgrade to V11, the retention policy may erroneously delete some of the monthly GFS backups before their retention expires. The issue only affects Backup Copy jobs which have their latest full backup corrupted.