All Restore Points on Rotated Drive Are Deleted

[paallergy]

Case #04876521

PROBLEM:
After upgrading to v.11 and decreasing the number of restore points, ALL restore points (including the VBK) were deleted by Veeam B&R as soon as I put in the next drive and re-enabled the BCJ.

Short term, I think the documentation should explicitly state that ALL restore points may be deleted from a rotated drive. I will submit feedback for that page and reference this post.
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Long term, I would suggest that the process be changed so that each rotated drive retains at least the minimum number of restore points for a backup chain. The main reason is that Veeam B&R does not know the state of the other rotated drives. If all the other drives are unavailable (damaged, lost, corrupted, etc...), then Veeam B&R is deleting the only available restore points. The safe course of action is to not delete all the restore points.

I believe most people would expect the following:
- restore points are copied
- retention policy is applied
- VIB data is injected into the VBK file, then the VIB files are deleted

[paallergy]

For me, it would be easier and more intuitive if the retention policy referred to the number of restore points (or days) on each disk.
If I add a disk to the set, I don't need to adjust the retention policy.

[Gostev]

Please note that BCJ in V11 no longer does "VIB data is injected into the VBK file" because it is not compatible with new features like immutable backups.

Why would you need to adjust a retention policy after adding a disk to the set? Retention policy is set on the job according to the business requirements e.g. 30 days. How many disks you are using does not matter, as only the older backups will be removed by retention policy.

[paallergy]

Adjusting the number of restore points is a side issue.
The main issue is that Veam B&R can delete all restore points on a disk.
If that had been my only remaining valid disk, I would have lost all my backups.

[Mildur]

Use days instead of restore point count for the backup job retention and veeam will never delete the last 3 successful restore points.
This value of „mandatory restore points to keep“ can be adjusted in the registry.

Example:
You configure a 30 days Retention.
Backup Job will not run for 35 days because there are some issues.
The restore points from the Day 36, Day 37 and Day 38 will not be deleted, because there isn‘t a newer Restore point since then.

[soncscy]

The main issue is that Veam B&R can delete all restore points on a disk.
If that had been my only remaining valid disk, I would have lost all my backups.

I mean, to be fair, this is part of the definition of retention.

I'm not making light of the situation, I realize you just unexpectedly lost a ton of backups you didn't mean to, but backup applications need to reclaim space.
Even without retention, this can happen if someone in your org isn't being careful about what they do and just "clicking" things. I've had this issue with clients before who didn't take the time to read a bit first before taking an action, and then wondered why they had no restore points.

[paallergy]

Use days instead of restore point count for the backup job retention and veeam will never delete the last 3 successful restore points.
This value of „mandatory restore points to keep“ can be adjusted in the registry.

Example:
You configure a 30 days Retention.
Backup Job will not run for 35 days because there are some issues.
The restore points from the Day 36, Day 37 and Day 38 will not be deleted, because there isn‘t a newer Restore point since then.

1. Will this guarantee that there will always be 3 restore points on each rotated drive?
2. Why not implement the same minimum when number of restore points is used?

[paallergy]

I mean, to be fair, this is part of the definition of retention.

I'm not making light of the situation, I realize you just unexpectedly lost a ton of backups you didn't mean to, but backup applications need to reclaim space.
Even without retention, this can happen if someone in your org isn't being careful about what they do and just "clicking" things. I've had this issue with clients before who didn't take the time to read a bit first before taking an action, and then wondered why they had no restore points.

No, deleting ALL restore points is not the definition of retention. Retention is deleting SOME of the restore points and retaining some restore points so restoring data is possible.

All the restore points being deleted on the drive is the result of how Veeam B&R is programmed, not a resulting of just clicking on things.

[Gostev]

Retention is deleting SOME of the restore points and retaining some restore points so restoring data is possible.

That's exactly how Veeam works. It will only delete restore points which are outside of the retention policy (so the oldest). Recent restore points which are still under the retention policy will be retained, so restoring data is possible.

If you configure your retention to be long enough so that it still covers some of the "old" backups already present on the inserted rotated drive, then those backups will not be deleted. For example, if you have two drives which you rotate every week and you set your backup job retention policy to 10 days, then following the rotation we will keep last 3 most recent restore points of the "old" chain that is already present on the newly inserted drive.

[paallergy]

Let's see if this helps explain it better:
Assume you have 2 rotated drives, drive 1 and drive 2.
You put in drive 1.
Veeam B&R deletes all the restore points on drive 1, according to the retention policy.
(note that this happens before any new restore point is copied to drive 1)
Assume drive 2 is no longer available (lost, damaged, corrupted, etc...).
You no longer have any restore points on drive 1 (deleted) or drive 2 (damaged, ...); therefore, restoring data from this backup set is not possible.
Veeam B&R thinks that there are restore points available on drive 2, but it does not know that drive 2 is no longer valid.

What I am proposing is that the retention policy be applied after 1 or more new restore points are successfully copied to drive 1, so that drive 1 always contains at least 1 restore point.
I understand that the old restore points need to be deleted according to the retention policy. I'm just proposing that in this scenario Veeam should follow its policy of making the safe choice and only apply the retention policy after at least 1 new restore point has been successfully copied.

[NickKulkarni]

I agree Veeam would be unaware of failed HDD and therefore should not delete until a successful copy is made. However I can also see why Veeam would want to delete first before copy. If the backup copy filled the HDD because the remaining space was insufficient the new backup will fail.

The only possible solution I can see is that Veeam moves to delete after fresh right and documents the warning that HDD size selection is factored into the retention policy. i.e. setting the retention policy must include estimate of backup size to prevent filling HDD before deletion of older restore points. Having been in a full HDD situation because backup sizes grew beyond retention policy estimates I know that this causes a failed backup.

[ejenner]

This is funny because on my system with rotated drives I format the oldest drive before reintroducing it to the cycle. I change the drive once a month, store the current drive, put the oldest one back in after formatting it.

[brerrestee]

Hello everyone,

I've been following this thread for a few days with great interest and am amazed that not many more users have already contacted me.
I have observed the same problem and a secure data backup procedure looks different from what B&R has been doing with the rotated drives since version 11. Our approach is as follows: We use RDX media (HDDs) to simply take backup sets out of the house. For the days Monday to Friday we have a medium that we change every day. We also have 5 different media for Fridays, i.e. we use a different medium every Friday until all 5 have been used one after the other and then start again.
To make it even more complicated, we have another 12 media that we put in on the last day of the month.
In B&R 10, we had 4 restore points on each RDX medium, i.e. we had a backup set available outside the home for every day of the month. In addition, 20 weekly backups (5 media à 4 restore points) and on top of that a restore point from the last day of the month 12 months backwards. It was easy, stable and reliable.
I admit that is a bit of an exaggeration, but we work in a security-oriented industry and we had a large number of restore points out of the office with minimal effort, and with a long time range backwards. It seldom happened that a medium overflowed. But that was easy to see, I could intervene manually and we always had a stable backup. The number of possible restore points resulted simply from the size of the media and from time to time I was able to readjust them according to our data growth or procure larger media.
Since B&R 11, this process has become very complicated and completely unsafe. It is absolutely impossible for a backup set to be deleted without an emergency before a new one is written.

Veeam, did you forget what we customers need? Reliability, easy handling and data security. That's the only reason we bought Veeam at some point! I very much ask that this point be discussed again and improved.

If I now have a problem with a medium, I cannot exchange it without losing restore points. Example: There are 4 restore points on a defective medium. I replace the medium with a new, empty one. B&R does not notice that 4 restore points are missing and will delete 4 restore points over the next few days, but they should still be active. With all understanding, this procedure is not thought through to the end ...

I sincerely ask for a repair!
Best regards

[Gostev]

If I now have a problem with a medium, I cannot exchange it without losing restore points. Example: There are 4 restore points on a defective medium. I replace the medium with a new, empty one. B&R does not notice that 4 restore points are missing and will delete 4 restore points over the next few days, but they should still be active.

It sounds like you're using the restore point based retention policy. I recommend that you switch jour jobs to the retention policy based on the number of days.

With daily retention, B&R will not delete restore points based on the count of "known" restore points. Instead, it will only delete restore points that are outside of the specified retention policy required by your business. In other words, it will ONLY delete restore points that your business no longer needs for restore purposes in any case, because the data contained in them is too old to be restored.

From your explanation above, it sounds like you're emulating what is effectively Weekly GFS ("we also have 5 different media for Fridays") and Monthly GFS ("we have another 12 media that we put in on the last day of the month") retention policies. Such retention policy should rather be enabled directly in the job settings, allowing you to protect the corresponding backups from deletion in case a wrong drive is accidentally inserted, and correctly processing their retention by keeping them longer than other restore points.

[paallergy]

It sounds like you're using the restore point based retention policy. I recommend that you switch jour jobs to the retention policy based on the number of days.

It would be preferable if users could select either number of restore points or number of days and still know that the drive will always contain restore points.

Is there a reason that in this case the retention policy is applied before restore points are copied?
Is most/all other cases the retention policy is applied after restore points are copied.

EDIT:
I don't think that switching to days will fix the problem (all restore points deleted).
The issue is not that restore points are deleted. The issue, for me at least, is that all restore points are deleted before new restore points are copied, which results in 0 restore points on the drive.

[paallergy]

From your explanation above, it sounds like you're emulating what is effectively Weekly GFS ("we also have 5 different media for Fridays") and Monthly GFS ("we have another 12 media that we put in on the last day of the month") retention policies. Such retention policy should rather be enabled directly in the job settings, allowing you to protect the corresponding backups from deletion in case a wrong drive is accidentally inserted, and correctly processing their retention by keeping them longer than other restore points.

GFS is not available with rotated drives:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Unless your post is an announcement of an upcoming feature enhancement

[Gostev]

Is there a reason that in this case the retention policy is applied before restore points are copied?
Is most/all other cases the retention policy is applied after restore points are copied.

The issue is not that restore points are deleted. The issue, for me at least, is that all restore points are deleted before new restore points are copied, which results in 0 restore points on the drive.

I believe this can only be caused if you are using the ForceDeleteBackupFiles registry hack, so if it is not a desired behavior then just remove this registry value. You are right in that the backup retention policy for recent backups is always applied at the end of a successful job run (unless some other process has already deleted them).

[Gostev]

GFS is not available with rotated drives:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

@Egor Yakovlev can you please look into this, what is a reason for this limitation.

[paallergy]

I believe this can only be caused if you are using the ForceDeleteBackupFiles registry hack, so if it is not a desired behavior then just remove this registry value. You are right in that the backup retention policy for recent backups is always applied at the end of a successful job run (unless some other process has already deleted them).

I checked, and there is no ForceDeleteBackupFiles registry key in use.

[Gostev]

Then you should open a support case to investigate what process deletes those backups and why.

[Egor Yakovlev]

Jonathan,
can you also check session history - please confirm if there is 1 or 2 Backup Copy session runs on time of a switch?
/Thanks!

[paallergy]

Jonathan,
can you also check session history - please confirm if there is 1 or 2 Backup Copy session runs on time of a switch?
/Thanks!

I'm not sure I understand the question. I uploaded the logs to Case #04935220. Hopefully the logs will answer your question.

[paallergy]

Then you should open a support case to investigate what process deletes those backups and why.

"So after discussing with Tier III this is currently expected behavior, as soon as one drive reaches the specified number of restore points, it then starts a new chain on the next drive that is used."
Case #04935220

Deleting all restore points may be expected behavior for Veeam, but I don't think it's the expected behavior for most end users. Even the Veeam support tech was surprised that the VBK file was also deleted.

@Gostev:
Is it possible to have the retention policy revised for rotated drives so each rotated drive always retains a backup chain (VBK + minimum number of VIB files)?

[Gostev]

We should never delete *all* restore points. Restore points which are still under the retention policy should remain untouched. Only expired restore points should be deleted.
TBH I cannot make any sense from what they told you... @Egor Yakovlev could you please review this case?

As for retaining expired backups which are no longer covered by the retention policy, this is currently not planned.

[Egor Yakovlev]

I have reviewed the case and everything seem to be working as expected.

Let's walk the restore point removal logic together with an example scenario.
(numbers in the example below are adjusted from ones in the real case, for ease of understanding, as it doesn't change the working logic - add more drives, increase retention - it will work the same no matter what)

Case:
- We have 2 rotated drives
- We have Retention set to 7 restore points
- We have daily run schedule
- We are using Backup Copy Job

Working logic:
- Veeam starts it's first job run. Since it's a first run, we go with fresh full backup on Drive-1.
- Following days, an incremental backup is created on Drive-1 for each subsequent job run.
- By the end of a week-1 you will have 7 restore points on Drive-1.
- [Drive switch happens]
- Next day, when job starts, Veeam tries to perform yet another incremental backup to existing chain on Drive-1, but finds out that drives are switched and expected chain is not there anymore. Job fails to perform incremental backup to expected-to-be chain(still job has green "Success", as we are instructed to expect Rotated Drive behavior).
- Veeam finds 0 existing restore points on Drive-2.
- Job instantly restarts with "drive switched, ignore missing chain" logic. Since Drive-2 has 0 restore points, we start a new chain on this drive, with a fresh Full backup.
- Following days, an incremental backup is created on Drive-2 for each subsequent job run.
- By the end of week-2 you will have 7 restore points on Drive-2.
- [Drive switch happens]
- Next day, when job starts, Veeam tries to perform yet another incremental backup to existing chain on Drive-2, but finds out that drives are switched and expected chain is not there anymore. Job fails to perform incremental backup to expected-to-be chain(still job has green "Success", as we are instructed to expect Rotated Drive behavior).
- Veeam finds 7 existing restore points on Drive-1 and applies retention. Since we know Drive-2 has 7 fresh restore points from last week, we are deleting any older restore points on Drive-1, because they do not make any value based on set retention policy. That might look like we are deleting [all] backups, but it is just a cold iron logic of specified retention. We know there are 7 restore points on Drive-2, which means everything older is considered useless to a company based on set retention policy.
- Job instantly restarts with "drive switched, ignore missing chain" logic. Since Drive-1 now has 0 restore points, we start a new chain on this drive, with a fresh Full backup.
- Rinse and repeat, same behavior will continue to work.

To summarize resolution:
1. If you don't want Veeam to delete 2-weeks-old restore points, then retention policy of "7" should be simply adjusted higher, or changed into "days" rather than "restore points".
2. In your case scenario, if offline Drive-2 dies for whatever reason, Veeam cannot predict that in any way, and will still apply retention policy on Drive-1 restore points, otherwise we will violate specified company Retention Policy.
3. We will tune "retention on restart" behavior, so that BCJ will apply retention only after fresh job run(it will leave you with at least 1 valid restore point), however it will not alter Retention logic in any shape or form - if existing restore points are outside of set retention policy, it will be removed.

Hope that example clears it.

[paallergy]

To summarize resolution:
3. We will tune "retention on restart" behavior, so that BCJ will apply retention only after fresh job run(it will leave you with at least 1 valid restore point), however it will not alter Retention logic in any shape or form - if existing restore points are outside of set retention policy, it will be removed.

Thank you for the planned fix in #3 (applying the retention policy after new restore points are copied)

What I am proposing is that the retention policy be applied after 1 or more new restore points are successfully copied to drive 1, so that drive 1 always contains at least 1 restore point.
I understand that the old restore points need to be deleted according to the retention policy. I'm just proposing that in this scenario Veeam should follow its policy of making the safe choice and only apply the retention policy after at least 1 new restore point has been successfully copied.

Is it possible for Veeam to say at this time if the fix/change will be in v12 or a later version?

[Egor Yakovlev]

Too early to say, however task for the change is logged for v12 iteration.
Cannot guarantee it will make it there at this time, though.

/Thanks!

[Anguel]

Is there any way to keep a minimum of 1 or 2 restore points on rotated drives no matter how old these are?
I still have problems with physical computers that are not powered on regularly.
I thought that the "number of restore points" retention would do that but this is not the case, all backups are missing for computers that have not backed up for a longer time.
It is a nightmare to rely on rotated drives for offsite and find out that there is not a single restore point for computers that have not performed recent backups.

[foggy]

Likely due to deleted VMs retention.

[Anguel]

No, this is not set.
How about the „mandatory restore points to keep“ setting Mildur is talking about above in the thread? Is this still applicable and where is it described in details?