Restore points deleted early by retention policy?

[pshute]

We do a daily week-day incremental backup, and an active full each Tuesday night, and we have a retention policy of 5 restore points. Normally this means the restore points build up until there's two active fulls on disk plus 8 incrementals (full inc inc inc inc full inc inc inc inc) before it can delete the oldest 5 restore points to retain just 5.

I see this reflected in the disk space graph of the drive concerned. It normally shows 2TB free most of the week, then rises to 4TB free for 24 hours on Monday night. Last week it rose to 4TB free on Thursday night. How? That should have left it with just 3 retention points on Friday. One explanation is that there were, for some reason, extra incrementals done. There are no extras present today:

Code: Select all

16/10/2019  08:26 AM 1,821,369,676,288 Backup Job 1 - mainD2019-10-15T200029.vbk
16/10/2019  09:33 PM    65,387,411,456 Backup Job 1 - mainD2019-10-16T200035.vib
17/10/2019  09:32 PM    77,343,162,368 Backup Job 1 - mainD2019-10-17T200029.vib
18/10/2019  10:00 PM    75,412,508,160 Backup Job 1 - mainD2019-10-18T200023.vib
21/10/2019  09:49 PM    94,904,991,744 Backup Job 1 - mainD2019-10-21T200021.vib

Is there any way to see a history of which retention points it has deleted and when?

[PetrM]

Hello Peter!

I'd suggest to look for corresponding message about restore points removing in job session statistics for Thursday night.
In addition, you may check for extra runs between Tuesday (when full is scheduled) and Thursday (when you noticed a peak on free space graph).

One more option is to generate job html-report in which you can see the whole job history, you can look for extra runs there as well.

Probably the job was launched manually by another user or by some PS script?

Thanks!

[pshute]

Thanks. On the 18th (Thursday night) I see:

Code: Select all

18/10/2019 10:01:04 PM :: Removing 'F:\Backups\Backup Job 1 - main\Backup Job 1 - mainD2019-10-14T200045.vib' per retention policy  
18/10/2019 10:01:05 PM :: Removing 'F:\Backups\Backup Job 1 - main\Backup Job 1 - mainD2019-10-11T200027.vib' per retention policy  
18/10/2019 10:01:05 PM :: Removing 'F:\Backups\Backup Job 1 - main\Backup Job 1 - mainD2019-10-10T200041.vib' per retention policy  
18/10/2019 10:01:05 PM :: Removing 'F:\Backups\Backup Job 1 - main\Backup Job 1 - mainD2019-10-09T200130.vib' per retention policy  
18/10/2019 10:01:07 PM :: Removing 'F:\Backups\Backup Job 1 - main\Backup Job 1 - mainD2019-10-08T200022.vbk' per retention policy  

Those are the exact files I'd expect to see removed on Tuesday night. If no more files have been removed since then, it looks like we were left with only 3 retention points that night.

I see no history items suggesting any extra backups were done. What is slightly unusual around that time is a lot of retries because one machine was having VSS troubles and wasn't backing up. Some of these were run automatically by the job, a few were manually run by me.

For one of the manual retries, I accidentally told the job to run instead of retry, then quickly cancelled it. That was on the 15th. If that job had run, there would be a full backup present, and I would expect the older restore points to be deleted. But it was cancelled before any machines even started getting backed up. Surely the retention policy only looks at successful backups, and ignores failures that don't even produce a backup file?

[PetrM]

Thanks for reply!

Retention sees successfully created restore points only and ignores failures.

Please create a support request to investigate the issue and provide us with case number for our reference.
I suspect discrepancy between number of restore point objects at the level of Veeam configuration database and number of restore point files residing on repository file system.

Thanks!

[pshute]

I should probably try to reproduce it with a test job first, then see if it goes away if I upgrade. No point reporting fixed problem. We're on 9.5.0.1038.

[PetrM]

I'd suggest to upgrade at first and don't waste time reproducing the issue.
It would be better to create a support request anyway, if configuration database contained a phantom restore point you would see an error when retention would try to remove
a restore point which had no associated backup files on repository.

By the way, I'm wondering to know the result of the next retention run:
if it happens on scheduled day and removes necessary restore points successfully it will most probably mean that the issue won't reoccur.

Thanks!

[pshute]

I've reproduced it with a small test job. I set it for 5 retention points, then ran it 3 times to produce a full and two incrementals. Then I ran an active full. Theoretically, it's not allowed to delete the first 3 until there are 4 more backups.

Then I ran another active full and immediately stopped it (graceful option). I repeated that several times, and then all the previous restore points vanished, leaving just one.

I wasn't able to reproduce it by starting and stopping incrementals.

[wishr]

In certain situations, a restore point produced by an active full backup that has been stopped may be treated as a valid restore point during the next active full run and thus will be taken into account by retention mechanisms. Most likely (I'll leave the opportunity to confirm this to our support engineers), you have a few such restore points that caused that behavior.

This is just an assumption based on the given information and not a verdict. Please, raise a support case and let our engineers precisely analyze all the details to get a final clarification.

Thanks in advance.

[PetrM]

Just discussed the behavior with QA: by design, retention sees only successfully created restore points.
It's definitely not normal when uncompleted full or increment is treated as a valid restore point and is taken into account by retention mechanisms.

Most probably in your case retention removed previous backup chain due to phantom restore point which existed (or still exists? ) in configuration database.
On the other hand, this is not more than my supposition and detailed investigation must be performed by our support team.

Please inform us with results of investigation.

Thanks!

[pshute]

The problem didn't happen again after a week of normal backups, so there's been no lasting database problems.

I upgraded to 9.5 Update 4 today, and was not able to reproduce the problem afterwards, so it must have been fixed.