Retention is not honored, backup files keep piling up on the repo

[lucius_the]

Dear all,
I am (somewhat) long time Veeam user. I manage a couple of Veeam backup instances for our clients (they are larger and we had them buy Veeam licences) and for my company I'm using the excellent Community Edition, as we have a small number of VMs and this fits for us.

Here in our company, Veeam is installed on another physical machine and is using that machines local storage as a repo (so, DAS).
Pretty simple setup, no proxies, no SMB repos, no complications. The main server that we back up is a single vSphere ESXi host.
The main backup job runs daily and is configured to keep 7 restore points, with Active Full on Saturday. Everything works nicely and quickly, no errors during jobs.

Except I already have 17 backup copies in the repo, including the 3 Fulls. That's looking at the Veeam B&R, under Home -> Backups -> Disk.
Looking at the files themselves, there's 1 large VBK every Saturday, followed by 6 smaller VIBs produced during the week. It's 19 backup files in total, dating all the way back to where the first backup was started. None of them ever got removed.

I don't have this issue anywhere else, just at "my house" Yeah, I know I can manually delete the obsolete backup chains, but I'm wondering if this is maybe worth investigating before I do so, as it seems like something is not working properly. It looks like retention policy is not followed for some reason. And I can't find a reason. There's Active Fulls that do run weekly, so there are clean backup chains here, Veeam just needs to cut off the old ones, but that doesn't happen.

What I do have in this setup is another Backup Copy job. That copies stuff from the main backup repo to an offsite backup repo. That also runs daily. It's scheduled to run half an hour after the main backup job. Runs nicely, finishes quickly enough, no warnings or errors of any kind.

I wonder why Veeam is keeping so many backup copies in the main repo. It shouldn't. But I can't figure out why it's not removing them.

[lucius_the]

I found this: veeam-backup-replication-f2/restore-poi ... tml#p25092

But it doesn't apply in my case.

I do have seveal VM-s in my backup job, but they all exist and are backed up daily.
For every VM that's listed, under Backups -> Disk -> my main daily backup job -> Properties, I have fresh restore points. They all look like this:


So, there are no stale VMs that could be keeping Veeam from removing the old backup files.

[foggy]

Hi David, indeed, provided you have configured the job to keep 7 restore points, the behavior you are seeing is not expected. I recommend asking support to take a closer look at your setup. Thanks!

[lucius_the]

Thanks, I opened a support ticket as suggested (#03996669).

[lucius_the]

Oh, I think I know what this is. I forgot about a couple details I did here.
This backup file was first created for ESXi host that was added to Veeam B&R via IP address. Later I added this same host via the hostname (for flexibility), created a new backup job and mapped it to the existing backup file. Then removed the old server (that was added via IP) and the old backup job that was targeting it (I'm not sure actually if I changed the existing job, or did I create a new job? can't remember).

After a couple of days I removed all of the "old" VM-s from the backup repo, as Veeam was seeing them as belonging to a different ESXi host (although it's actually the same host and same VMs, Veeam showed double entries in the backup, obviously it was unable to recognize them as being backup of the same VMs).

I thought this would work, as I deleted all of the "old backups" that existed in the backup files. But, I guess something didn't go well here, obviously Veeam B&R thinks that the backup in the repo still contains something of the old, even though it doesn't. So it doesn't want to delete the old copies. I guess that's it.

I think I'll just remove everything from the repo and the job should reset itself and start from scratch. This should fix it.
Will try to leave this unchanged for a couple of days, in case someone from Veeam want to take a look. I can't wait too long, though, as I don't have much free space left.

So this was actually a much more complex scenario, but I forgot about those steps I did which are... probably uncommon and not something very clean

[foggy]

This could be the reason, indeed. Veeam B&R tracks VMs by their unique IDs which change in case the VM (or the host it resides on unless vCenter is used) is re-registered in the virtual infrastructure. Those manipulations could result in some database inconsistencies. If you're ok with deleting existing backups, I'd also re-create the job itself so it has clean metadata from the beginning.

[lucius_the]

I can remove the backup files from the repo, that's no issue.

But I'm not sure about creating a new backup job. Because I have offsite backup copy job linked to the existing backup job (it takes this backup job as the source). I'm afraid a new backup job cloud also trigger a new full to the offsite repo, which would take long time to finish. But, I'm not sure if this would happen or not. The backup copy job was indeed created after the host was re-added, so it shouldn't know anything about the "old VM-s".

What do you think ? Would re-creating the backup job cause the backup copy job to loose it's tracking and also start from the start ?

[lucius_the]

Veeam Support responded. In a way that surprised me.
I have some 20-ish years of experience in IT and 15 in running an IT company. The guy (named Victor) crawled through the logs and responded:
- explaining exactly where the problem was
- providing a (surgically precise) solution, with a link explaining the steps
And that was in the very first response I received, no less. No back and forth communication, no stupid questions, nothing of the usual boring stuff we all get with support. Opening a case landed it in the hands of someone who can solve it - right away. No 3 levels of escalations and 2 weeks passing first, until it gets in the hands of someone who can figure it out. No. Right away. First response. "Oh, here. I see it. Please do this, here's how. If you need more help, let me know".

And it wasn't a simple case of me being a completely dumb user either. The issue really is more complex, as I happened to remember later all I did with this backup, as explained in a post above.
Veeam obviously has good people in support. But also the "good guy from support" had to spent time digging through the logs to figure out precisely where the problem was. And that's... well, more than great, that's real effort. I just wanted to post this here, as I think things like this shouldn't go invisible. This was my first experience with Veeam support. No wonder people say good things about you.

Btw, this is Community Edition I'm using
I got an answer in, what, two days - and proper answer. The one that helps, not some robot or "human robot" answer that serves only to "provide a response within x hours".

I'm trying to find a good english word to express how positively stunned I am !
Just to be clear, noone paid me or asked me to write this. People from Veeam simply deserve an big praise from me. Thank you guys.

[lucius_the]

Back to the issue. I removed all files from the backup repo and run the backup job again (I didn't recreate the backup job).
The backup job job finished, it created a new chain starting with a full. The backup copy job then run as usual, it copied just the changed blocks which is what I was hoping for.

All seems fine. I'll know if this is fixed in 1 week. If not, I'll just recreate the jobs, but I think I won't be necessary.

P.S. Unfortunately I couldn't try what Victor (Veeam Support) suggested because I already deleted everything from the repo But if this was anything serious, or not easy/safe to delete and recreate I would have waited on support and tried their solution.

[foggy]

Hi David, thank you for the update, I'm glad that the issue is finally solved. And also thank you for the kind words about our support - always nice to hear that and they really deserve it!

[Kayra]

Hi David,

Good to read your issue is solved, but I have exactly the same problem: main backup jobs run daily (7 restore points), with an Active Full once every week.
But on the repository I see between 7 to 17 restore points (1-3 full backups included). This differs per backup job, but the restore points to keep (7) seems to work for some jobs and not for all.

Can you please share the info you got from Veeam Support:
- explaining exactly where the problem was
- providing a (surgically precise) solution, with a link explaining the steps

Thanks already,
Kayra

[lucius_the]

Hi Kayra,
It is very unlikely that we actually have the same issue. If you read above, I also happen to re-register the ESXi host, the backup job then started producing duplicates (there were "old VMs" in the backup and also "new VMs" in the backup), so then I deleted all of the "old VMs" from backup. That actually emptied the oldest full backup file, but it wasn't removed because "storage cannost exist without vbk".

Veeam support, after looking into the logs I sent them, provided a solution by manually deleting the empty storage file, re scanning the repo and deleting from disk all unavailable backups (https://helpcenter.veeam.com/docs/backu ... l?ver=95u4).

But I'm pretty certain this solution works only for my case and you shouldn't try this.

Looking at your pattern, I believe you should have between 1-2 backup chains (so up to 14 restore points) because if you are doing a weekly active full, and want to keep 7 restore points, VBR needs to keep the last weeks backup chain. As I see it, it can not remove the last backup chain before it creates the new one, so I believe it's possible (on 1 day only) that you actually see 3 fulls and 15 copies in total for a short while. On next run this should be reduced by 7. More info here: https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

Perhaps you could reduce the number of copies to keep to 6, that should never produce more 2 active fulls at any moment, as far as I understand it.
But I could be mistaken, sometimes this puzzles me too. I don't see how you could get 17 backup copies (17 files in the repo, yes, but not 17 backup copies).

[foggy]

Hi Kayra, David is spot on, this is what most likely happens in your case. Please review this KB article with a visual explanation of how retention works for better understanding. Thanks!