[RESOLVED] VAW (paid / licensed) doesn't honor retention policy

[bhagen]

I have VBR9.5 Enterprise and several VAW installs. One of the agent backup jobs has a retention policy of 14 restore points, but there are currently 36 restore points in the backup folder (on the repository).

In this daily job, advanced settings, I create synthetic fulls every Saturday, perform backup file health checks once a month, i *don't* have "remove deleted items data after..." checked.

I notice that starting on 12/29/2018, I have gotten this error every Saturday: "Synthetic full backup creation failed Error: Agent: Failed to process method {Transform.CompileFIB}: There is not enough space on the disk."

So I'm guessing this is the issue, but I'm not sure why. A full plus 13 restore points takes up about 7.5TB. I normally have about 14TB free on this box.

In any case, the critical question is: How can I now get back down to 14 restore points? My repository is now almost full.

Thanks!

[Dima P.]

Hello bhagen,

Looks like with forward incremental backup mode your storage was filled with restore points and now backup job fails due to the lack of free space. As a result - backup job cannot perform retention and delete the retired restore points.

When synthetic full backup (or active full backup) is enabled, previously used full backup file and its subsequent incremental backup files remain on the disk. After the last incremental backup file created prior to the synthetic full backup becomes outdated, Veeam Agent for Microsoft Windows automatically deletes the previous backup chain. Take a look at How Synthetic Full Backup Works article to learn more.

The quick fix would be to delete the unwanted restore points from your storage (to free the disk space) and reconfigure the backup job (you can either disable the synthetic full backup or lower the amount of restore point to keep on your repository). Please make sure that these backup files are no longer needed before deleting the data from the disk. Thanks!

[bhagen]

Thanks Dima. I would think that 3TB free space would allow those functions to happen; how much free space do I need?

As for deleting unwanted restore points, which ones do I delete?

If I sort the folder on the repository by date, newest to oldest, I see:

.vbm (Jan 27, 19)
.vib (22 of these from Jan 7, 19 thru Jan 27, 19)
.vbk (Jan 6, 19)
.vib (13 of these from Dec 23, 18 thru Jan 5, 19)
.vbk (Dec 23, 18)

I moved the .vbk and all vibs from Dec 23 to Jan 5 out of the folder to a sub folder to simulate deleting them. Then I re-scanned the repository and attempted a test restore from one of the remaining restore points. It fails, saying "Backup files are unavailable."

[Dima P.]

bhagen,

I'd strongly recommend to ask our support team for help, as they will make sure you are deleting the data correctly (also they can double check the disk space consumption issue). Please open the case and share the case ID in this thread. Cheers!

[bhagen]

Case#03387347 - RESOLVED

Not a really *good* solution, but it is what it is. Here's the summary, from Veeam support:

Hello Bill,

Thank you for the call today.

As we discussed, it appears you just had bad timing on when the Synthetic Full and Forever Forward merges happened and it prevented this job from running its Synthetic Full. Twice.

This resulted in it keeping too many restore points, killing your available storage.

My current suggestion is to:

1. Delete the old (unneeded) files.

2. Offload the current set to other storage (e.g. purchased external USB).

3. Run a new Active Full.

4 Once it has 14 new restore points then delete the offloaded points.

The problem is that after step 1, I still have over 9TB of backups, and no place to "offload" them to; this is an office across the country from me and we don't have 10TB of free storage just laying around like we do here.

The only saving grace is that the box I'm backing up is replicated via DFS to another box across the country, and both boxes have "previous versions" enabled.

So, I'm just going to delete *all* the backup files now, and start an active full tonight. If somebody needs a file or folder restored from the last 14 days (which is our RPO for this job), then I'll grab them from "previous versions". If the entire physical box dies today, then I at least have an iso bootup file for that server, and all the info can be replicated from the DFS partner back to this one.

What a pain!

The moral of the story? Don't ignore "failed" veeam emails that happen on Saturday, just because none of the jobs during the week work fine!

P.S. Is there a way to mark this thread as "resolved"?

[Dima P.]

bhagen,

Glad to hear that you've found the workaround (but sorry to hear that you end up with deleting all restore points). Thank you for taking the time to share the results with the community. Cheers!