Veeam is deleting it's own backups :-(

[davidhodges86]

Hi,

I've created a support request, 07337909, but I just wanted to put this out here too.

We inherited a Veeam setup recently and they had a job configured with this:


They had a problem which caused the backups to all fail one night.

The next day I ran a "Quick Backup" of a selection of VMs from the job to make sure it would run ok.

After backing up the 3 VMs I'd selected Veeam started deleting all the backups and history from local and S3 storage for all the other VMs in the job.

12/07/2024 08:37:18 :: Backup '*** - *** is outdated and will be deleted
12/07/2024 08:51:12 :: Backup '*** - ***' is outdated and will be deleted
12/07/2024 08:52:47 :: Backup '*** - ***' is outdated and will be deleted
12/07/2024 08:54:42 :: Backup '*** - ***' is outdated and will be deleted
12/07/2024 08:54:42 :: Backup '*** - ***' is outdated and will be deleted
12/07/2024 09:06:32 :: Backup '*** - ***' is outdated and will be deleted
12/07/2024 09:09:37 :: Backup '*** - ***' is outdated and will be deleted

When I spotted this I stopped the job before it could delete the remaining 20 VMs from the backups.

This has got to be bug in Veeam's logic somewhere? Even if the tickbox to remove deleted items is ticked it shouldn't even consider going through the backups and deleting things when running a Quick Backup of a single (or in this case 3) VMs of a job containing 20+ VMs.

I've disabled the deletion of removed items for now but it's not going to get their backups back.

David

[Mildur]

Hi David

Please work with our customer support team to confirm if it's a bug or not. The logs will confirm why those backups were deleted after the quick backup.
I have a suspicion. It could be related to machine exclusion in a backup job. The deleted item retention gets applied to excluded machines. With your setting, after 1 day we will start removing backups for such machines.
And Quick Backup is kind of running a backup job where all other machines in the job are excluded temporarily.

For the setting, consider to configure 7 days or more. Removing backups after 1 day can lead to unwanted data loss.
We have it documented in the help center.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[davidhodges86]

Hi Fabian,

I agree completely. I was shocked when I saw the 1 day delete and turned it off straight away until this is sorted out. We rarely enable that feature as like to manage it manually.

David

[RobMiller86]

Not sure why that's happening, but we generally don't use that feature and leave that unchecked. Veeam will still delete your old backups once they fall out of the retention period that was set when they were written. Immutability is the key to ensuring your backups aren't accidentally deleted.

[Regnor]

I'm with Fabian and also think 1 day could easily cause unpredicted backup deletions. Would be interesting to know what issue caused the backups to fail?

[davidhodges86]

Support have replied basically saying it's by design.

Crazy that data can be completely wiped out from a job while running a "Quick backup" of a VM from the inventory. No-one would expect this behaviour.

I personally never enable this option but as I said, we inherited the entire setup.

But, say it was set to 7 days and you have a major issue causing no backups for 7 days, then when you've got everything back online you run a quick backup of one VM purely to test everything is working... Lo and behold Veeam goes through and deletes the backups and archives for all the other VMs! I'm struggling to see how this is not a bug.

David

[Gostev]

Most likely, you have misunderstood your support engineer as to what specific behavior and in what specific scenario is considered "by design". My guess is that there's something else in play, some detail you're not mentioning - as most certainly it's not a functionality of a Quick Backup to wipe out data from repository. It is designed to do exactly the opposite: create additional, out-of-band restore points. It does not have any retention processing or repository cleanup code to start with, that is done elsewhere in the product. Further, quick backups are not even counted as full-fledged restore points for any of the retention processing tasks.

Ultimately, if this was by design, then like every other important behavior it would have been documented in the Quick Backup section of the User Guide. So when something sounds completely off, like it certainly is in this case, I recommend you simply ask your support engineer for an official reference in the product documentation. This will clear any misunderstanding, or force them to put in a request to add this critical missing piece of information (which is something our Technical Writing team typically does more or less instantly).

[SnakeSK]

This checkbox is not for retention. This checkbox is specifically designed to remove the no longer or excluded machines from backup chain after you no longer need it and want to ensure it wont reside on the backup storage.

We have long retention policies along with gfs chains and we utilize this heavily. We dont need to keep an eye out on no longer processed machines

[davidhodges86]

Hi Gostev,

Thanks for taking the time to respond to this, but now I'm even more confused.

It does not have any retention processing or repository cleanup code

But support did actually send me a link to a documentation page specifically about how the retention policy for deleted items works: https://helpcenter.veeam.com/docs/backu ... tems-works

Which surely contradicts what you're saying about a quick backup not having any retention processing?

UNLESS, you're right about the quick backup job not having any retention processing but instead it was a manual retry of a just failed quick backup. But wouldn't you expect the RETRY to honour the same processing rules as the original job though?

Our data is gone, I just don't want anyone else to suffer the same fate from a badly configured deleted items policy be it by themselves or an inherited configuration like ours was.

David

[Gostev]

I don't see anything following this link that contradicts what I said, nor does it mention quick backup anywhere.

Anyway, perhaps you missed the link I shared in my previous post but quick backups are not full-fledged restore points for any of the retention processing tasks. So it does not even matter where and when the retention is processed, because restore points created by Quick Backup do not exist from the retention processing code perspective.

But you are right, setting Deleted items retention to such a low value is a bad practice anyway, as the User Guide clearly explains... perhaps the UI should enforce minimum setting of 7 days @Egor Yakovlev ?

[davidhodges86]

Hi Gostev,

I'm only talking about quick backups because that is the job that was run when the backups were deleted.

My guess is that there's something else in play, some detail you're not mentioning

I ran a quick backup of 3 VMs, I think it failed and I did a retry (perhaps this is the missing detail), then it would have been during the retry of the quick backup that Veeam started systematically deleting all backups until I terminated the job.

If everyone's happy that the software has operated as it should given that the option to delete removed items after 1 day had been enabled by the previous IT Support then we'll make a note to check that setting on any Veeam installations we take over in future and be a bit more careful when running standalone quick backups of VMs.

David

[Gostev]

From your description it would not appear the software has operated as it should so ideally please don't agree to close the support case to ensure the scenario is better understood and if there's a bug in some corner case involving failures and retries then it gets fixed.

[gcg]

I ran a quick backup of 3 VMs, I think it failed and I did a retry (perhaps this is the missing detail), then it would have been during the retry of the quick backup that Veeam started systematically deleting all backups until I terminated the job.

I'm thinking that's the missing information piece. I don't think you can retry a failed Quick Backup. I'm betting what you actually did was Retry the original failure and if you had removed any of the VMs no longer in that job, with that low setting, it would remove those VMs that were now no longer in the job.

Just a guess.

[RubinCompServ]

Veeam will still delete your old backups once they fall out of the retention period that was set when they were written.

I keep hearing that. Is that new in v12? Because in v11, if a server is removed from a backup job, the backups for that server would sit around forever (or until manually purged, whichever comes first).

[Gostev]

Relatively new, it's called "background retention" and it can be easily disabled as desired. Without this functionality, users had to keep "dummy" jobs (sometimes for years!) with a small VM in it just to ensure that the retention policy keeps being processed for all removed VMs.

But now even if the original job does not exists, the background retention process will clean up backups according to their last know retention policy.

There are however also other/better tools to keep backups indefinitely these days, like Export Backup or Copy Backup features.