Veeam keeps forgetting aged-out restore points.

[Daniel2]

We have a backup copy job in a forever-incremental configuration. Veeam keeps forgetting aged-out restore points. This manifests itself in aged-out .vib files that pre-date the .vbk files, and remain on storage forever and must be manually deleted.

For example:

Code: Select all

G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-04-13T200000_510D.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-04-11T200000_6DC6.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-04-09T200000_C006.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-04-07T200000_0CBB.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-04-04T200000_3E3F.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-04-02T200000_C7A2.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-31T200000_DED4.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-29T200000_60A8.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-27T200000_9EDB.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-25T200000_8458.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-23T200000_89FD.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-21T200000_A04C.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-19T200000_67C5.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-17T200000_5652.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-15T200000_2EEB.vib
G:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-03-13T200000_DE0A.vib
F:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-02-20T200000_97AC.vib
F:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-02-18T200000_CB99.vib
G:\BACKUP_INFRASTRUCTURE_COPY\PRINT-TEST-01.VM-596115D2022-02-16T200000_9BED.VBK <-------FULL
F:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2022-02-16T200000_0F1A.vib
F:\Backup_Infrastructure_Copy\print-test-01.vm-596115D2021-12-17T200000_11A9.vib

Obviously, the last two .vib files should have and probably are merged into the .vbk, and they should have been deleted. They are also not display in the Veeam console under Home > Backups > Disk (Copy). For some reason, they remain on the storage.

This happens regularly and I have been told by the support team in at least two tickets to just remove them by hand. While this is acceptable as a work-around to free storage, I cannot accept this as the solution. If Veeam keeps orphaning old restore points, then this is a misbehaviour that must be fixed.

Case # 05378354

[Daniel2]

Ps, we have 311 VMs with a total of 6316 .vbk and .vib files. Out of 6316 files, 426 are orphaned amounting to a grand total of 1799 GB of wasted storage.

[Andreas Neufert]

Hi Daniel,

I asked the support team to have a look at your case and potentially escalate it to check for the root cause.

I saw something similar happening when the Antivirus system was not configured correctly. Basically sometimes the antivirus accessed the file at similar time (as Veeam openend it to read metadata to check if it is really the needed file) and then we send a delete and the AV blocking this.
Specifically as the reference in Veeam was deleted, but not on disk.

Please check if you have implemented the following correctly: https://www.veeam.com/kb1999

[Daniel2]

I feel like Veeam should not just give up if something like this happens, but retry cleanup and/or do scheduled inconsistency checks on the backup repositories with deletion of orphaned files. Finding these files manually is a ton of work.

[Andreas Neufert]

Let´s first identify why this happens (in the support case). Something is not working well here and the root cause need to be identified.

[Daniel2]

I already discussed this with the support engineer and his last reply was, that it's too long ago this happened and there are no logs to identify the root cause. I'm open to more troubleshooting and I have already a new remote session planned with him, but my main point stands unrelated to finding the root cause.

Let's say you would identify the root cause for this. How are you going to protect my repositories from this happening again due to other causes? That is why I recommend that Veeam does scheduled inconsistency checks. I find it odd that you rely on the customer to identify that.

[Andreas Neufert]

I understand your frustration with the sitaution.

On our side we can only workaround so much. Specifically arround data deletion you can not just automate things. There are many customers that keep the backup data or copy data arround. If we would just delete data that is not known to us, you would have issues in all kind of different situations. Our product is used in more than 1 million installations and we do not face issues in that area frequently. Development time goes to more requested features.

Based on support ticket there are lot´s of errors arround migration. Let the support team clean up the situation with you and then we can monitor the situation to really identify the root cause instead of creating workarounds for something that should not happen in the first place.

[Daniel2]

> On our side we can only workaround so much. Specifically arround data deletion you can not just automate things. There are many customers that keep the backup data or copy data arround. If we would just delete data that is not known to us, you would have issues in all kind of different situations.

I cannot follow your argumentation. There are four ways to "keep backup data or copy data around".

1. Within Veeam with Backup and Copy jobs: Veeam repository can be selected.
2. Within Veeam with Backup Export: Veeam repository cannot be selected, exported to same Veeam repository of restore point.
3. Within Veeam with VeeamZIP: Veeam repository can be selected.
4. From outside Veeam: Manually or with automated solutions.

If you choose options, the absolutely WORST place you can put your backup files is a location that is also a managed Veeam repository.

If you "keep backup data or copy data around" using the first three ways, then Veeam does absolutely know what should be in the repository and what should not. And there absolutely the option that Veeam can automatically take measures to keep the repository in a clean, consistent state.

Now in my case, the engineer found that the connection to the database was interrupted due to unknown reasons. And this was the reason that Veeam orphaned 2 Terabytes of restore points. This happened on multiple days over the span of the past 4 months, leading to an increased backup storage consumption, which eventually led to full storage, one month of not being able to do backup copys AT ALL (!). The fix cost me a lot of time, led to delays in my other projects, cost for additional 14 TB of storage so Veeam can start cleaning up, even more time to identify all orphaned restore points and delete them.

Coming back to the database issue. The Veeam engineer suggested that load was the primary reason the database connection terminated. Looking at the performance history of that server at the particular moment the error was thrown, there was a CPU load of under 20 %, a memory consumption of under 20 % and disk load of under 2 MB/s. At that point, the database was on the backup server itself. It was installed by the Veeam installer. No changes to its configuration were ever made. So I expect Veeam to "just work", as your company keeps telling me.

I don't understand your reasoning at all.

- We kept the SQLExpress database Veeam ships, as we only backup slightly over 300 VMs of supported 500 VMs, as per best practice. We don't use any of the features that suggest to use a standalone SQL server. In addition to that:
- Veeam did not inform me about the database errors in the Veeam console. Neither in the email reports. Only in the debug logs, that I should not have to monitor. So there was no way I could have known about this.
- Veeam did not attempt to continue the merge after the database became available again.
- Veeam practically orphaned these restore points without informing me of anything.
- Veeam offers no option to analyse the repositorys and list orphaned restore points.
- When I asked the engineer yesterday how we can safely identify all orphaned restore points, we had to use the SSMS and run some custom queries he gave me. And even then - at the last moment - I noticed the queries are incomplete and I almost deleted Terabytes worth of production-tier backups from our backup repository.


Frankly, reading your answers makes me boil inside because Veeam shit the bed in my opinion and I don't see any effort or willingness on Veeam's side to improve.

On our side we can only workaround so much

I completely disagree with your assessment, that these are workarounds. Backup repository integrity checks are not a workaround but necessary. Veeam failed a complete chain of operations and offers absolutely NO SUPPORT to me, to fix this situation other than by hand and praying that I don't make a mistake when deleting allegedly orphane restore points. I'm very close to move your infrastructure to a different vendor.

[mweissen13]

I wanna add my 2c here.
We have experienced this problem many times. About 2-3 times a year i am manually cleaning up orpahned .vib files on various customer sites (20+). I never invested the time to find the root cause for the problem. But i can tell 100% that we have NO antivirus whatsoever on any site. We even completely remove the Windows Defender Feature from all Veeam servers plus disable it with group policy. It also happens when using Linux repositories where no application could ever block a file from being deleted (aside from not having installed any antivirus solution on them).
So long story short I don't believe that this is an "outside" problem, but rather some problem in the cleanup process of Veeam.
I also completely agree with Daniel's point that there should be some sort of cleanup process integrated into Veeam (which should be configurable to either run automatically or manually). Since there surely is a myriad of root causes which could result in old backup files not being deleted. And having to manually write my own scripts or other means of cleaning them up is tedious and error-prone.

[foggy]

I guess Andreas's point was that blindly cleaning up something Veeam isn't aware of might cause even bigger issues than just filling up the repository space. A more reasonable approach is to identify the reasons why Veeam B&R wasn't able to correctly process the retention and address those either by improving the retention code to avoid such behavior in the future (in case the reason is on the product side), developing some automated workarounds to typical environmental situations, or tweaking something in the environment itself (if applicable).

[datadrop]

Same with me. During regular checks I see that the repositories are running full because of orphaned backup files, which had not been removed by retention. The solution for me then is to check which oldest backups are still to be found in Veeam and to delete older files than that by script. Not the nicest way to go, because of its error prone nature, but it keeps the repositories usable. I agree with mweissen13.

[ITP-Stan]

I also see this problem from time to time at customers.
Remember to also disable defragmentation in Windows on the repository disks.

[mkaec]

I think Veeam B&R should own the folders that the repositories are assigned to and be free to delete whatever files it wants from them. However, at the least, the existence of foreign files could be reported in the job log, and the job status could be set to warning, so administrators would know about the orphaned files.

[Daniel2]

I guess Andreas's point was that blindly cleaning up something Veeam isn't aware of might cause even bigger issues than just filling up the repository space. A more reasonable approach is to identify the reasons why Veeam B&R wasn't able to correctly process the retention and address those either by improving the retention code to avoid such behavior in the future (in case the reason is on the product side), developing some automated workarounds to typical environmental situations, or tweaking something in the environment itself (if applicable).

Foggy, I opened the ticket on 7 April and closed it on 16 May, after taking the decision to completely wipe and rebuild our 56 terabyte backup storage on the secondary site (we still have our primary and tertiary site online). That is something, we never want to do again!
I sunk so many hours into troubleshooting this issue, which cost a lot of money and required me to postpone several project's milestones. Your support technician was never able to find a root cause. I also have multiple discussions with the escalation manager, because the support engineers had no intention if finding a root cause, but suggested multiple times, to just use a workaround. One of the early engineers even suggested a workaround, that corrupted some backup chains in the progress! The entire support experience was extremely stressful!

If you want to find root-causes, please, do. But not at the cost of creating a huge risk item (backup to secondary site was completely impossible due to full storage for several weeks) and at the cost of my time. How are going to do this? And what are you going to propose until the root cause is fixed? Because I assume everyone who shared his experience in this thread, has a completely different root cause. So, some of us should just wait some more years until you figured it out? Or what's the idea here?

I'm completely baffled about the pushback I see, for a completely reasonable feature request.

[foggy]

Hi Daniel, apologize if it sounded like a pushback, didn't mean that. Your feedback is accepted and the request is noted (this is the main purpose of these forums after all). Was more trying to explain the approach we follow with such issues, which can certainly change based on the customers' feedback. I understand the frustration and am sorry about your support experience - I will let support management know so they could review the case and decide if it could have been treated in a more effective way.

[mweissen13]

I think Veeam B&R should own the folders that the repositories are assigned to and be free to delete whatever files it wants from them. However, at the least, the existence of foreign files could be reported in the job log, and the job status could be set to warning, so administrators would know about the orphaned files.

Excellent suggestion, +1 from me

[ITP-Stan]

Just now I had to check with a customer why the back-up job was failing.
Secondary storage was full.

I checked the settings of the job:
Keep 3 restore points, keep 1 monthly and 1 weekly for GFS.

What do I see on the storage:
1 full VBK that is from late February.
1 full VBK with the (_W) suffix that is a from February.
89 incremental restore points.

I just deleted the whole chain, and have to start over.
I have changed the job, disabled the GFS and just keep 31 restore points, because like this maybe it works more reliable.

I have no intention of opening a support ticket about this, because this customer will not tolerate the time & effort for this.

PS: The secondary storage was on a SMB share on a NAS (not my usual preference).

[woifgaung]

I am afraid this problem is not uncommon. Currently I found about 100TB orphaned files at a customer environment. The last time we saw this, support told us that it could be related to a bug in a Veeam component that the customer was running when the files were orphaned. Because this happened again, I do not belief this is the root cause. But we see more and more errors about temporary not available SOBR extents. Maybe this can cause these orphaned files? Do other see this errors too?

[tjurgens]

I thought I was alone with finding orphaned backups in my repositories. I generally only find them after manually looking through and subsequently cleaning them up.
If Veeam can't auto clean them, or its too risky, at least allow a report to be spit out (preferably in some format that can be easily understood by a script) and allow us to clean them up in an automated way. A registry or config setting could be used so we have that control on what we think is best. Eg: Autoclean, Report Only, Ignore.

[18436572]

I think I had this issue a few Veeam versions back. Why not report on orphaned files (if any) after the health check?