Wasabi immutable retention

[JeroenL]

Hello all,

I have a question regarding retention on Wasabi with Immutability enabled.

We run Veeam B&R V12 and I configured a few Wasabi repositories. Immutability retention is set to 14 days.
Backup Copy jobs are configured with a Wasabi repository as the target. The retention here is set to 28 days.

At this moment I see as much as 49 restore points. When I check the Backup Properties of the job I only see two or three days with 2 restore points. This never adds up to the 28 days configured within the copy job configuration.

What can I do to solve this? I tries to add a weekly full backup using the GFS option. I thought that this might breakup the Forever Forward Incremental chain an cause a cleanup to 28 days, but this didn't help.

Another strange thing is that there was a VM unintentionally added to a Copy Job to Wasabi. This VM was uploaded to Wasabi on May 15th. Immutability is set to 14 days. When I try to delete the file, Veeam reports it is still protected until June 8th. This extends way more than the set 14 days? It more becomes 14 + 10.

I read about Block Generation and that this can add 1 to 10 days of extra retention. For the accidental uploaded VM this 10 days seem correct, but for the other VMs this doesn't. First backup was generated on April 17th. Add another 10 days and retention points should have started to be deleted from May 25 or 26th. Now we are almost two weeks further and diskspace and costs are still adding up.

[Mildur]

Hi Jeroen

That sounds strange.
I recommend to open a case with our customer support. We cannot check your environment over a forum post.
Our support team will be able to check the logs and configuration and compare them against the available restore points.
After their research, they will come up with an answer about the current situation.

Please provide me with the case number.

Best,
Fabian

[J.Leeflang]

Case# 0610052

What I think is causing the problem is that a backup copy job creates it's own backup chain.
This is a forever forward incremental chain.
The immutable option places a lock on all files required for the active backup chain. With a backup copy job there is only one chain and so this entire chain gets locked up.
Am I seeing this right. This can be very easily solved by adding an option to create a synthetic full backup every week. This creates a new chain, so the older chains can be cleaned up as soon as the immutable retention and the regular retion are past.

[J.Leeflang]

I think I solved the issue.
It seems it is required to create a single Weekly full just as with some other tasks to prevent the active chain to become to long.
In this case without Weekly full configured, there is only one chain.
Immutability configures all "required" objects with the same immutable protection as the latest restore points. This represents the entire Active Chain.
Since there is only one full and one chain no cleanup will happen.

By simply adding a single weekly full the chain seems to get split into a new Active Chain and the older chain can be removed as soon as the youngest file meets the retention period.

I hope someone can confirm this and add something in the Veeam manual to clear this out for Backup Copy jobs to Object storage with immutability enabled.

[Mildur]

Hi Jeroen

Active full backups are not required for „direct to object storage“. I‘m still thinking it‘s worth to check whats really happening.

Especially this statement makes no sense for me (without seeing screenshots):

At this moment I see as much as 49 restore points. When I check the Backup Properties of the job I only see two or three days with 2 restore points.

3 days * 2 restore points = 49 restore points?
That doesn‘t sound right.

to prevent the active chain to become to long.

There is no backup chain or backup file on object storage. Just blocks as objects which are used by different restore points if they never change between two backups.

Best,
Fabian

[tjurgens-s2d]

Have you tried configuring a GFS policy? That will create synthetic full backups on the days selected.

[Mildur]

@tjurgens-s2d
There is no synthetic full with direct to Object storage backups. GFS backups are just tagged restore points.

Best,
Fabian

[tjurgens-s2d]

That's fair, but as far as I understand it they essentially work the same way - marking a restore point as a full backup, keeping whatever objects that go into that restore point for the period of time specified by the retention policy.

That said, I agree this situation is a bit odd and creating a GFS point *shouldn't* have any effect on their immutability.

[Mildur]

If you configure GFS settings, those weekly restore points will be immutable for their entire GFS retention period.
With a GFS retention for 4 weeks, weekly backups will be immutable for 4 weeks + 10 days. But I didn't saw a GFS retention in his first post. I saw only 28 days.

@J.Leeflang Did you have enabled a weekly retention policy for 4 weeks? This will lead to 5-6 weeks immutability of those weekly restore points.

Best,
Fabian

[J.Leeflang]

I am starting to understand the system behind object storage a little better, I think

Retention is basically applied to chains and represents a minimum number of days or restore points.
Since there is no merging on the Object Storage end and the entire chain is protected by immutability settings, the regular retenion system cannot cleanup older objects.

I enabled a single Weekly restore point for our jobs. This is a standard single GFS Weekly full. Not an active full. There is no need for this.
This way the Active Chain in the Wasabi storage is shortened to 7 days of objects.
As soon as this entire chain is older then the set retention for immutablity, the normal retention process takes over.

As soon as the youngest objects belonging to the latest incremental file in this chain gets older then the regular retention setting, this entire chain will be removed.
If retention is set to 28 days. I will see recovery points up to 34 days old. As soon as the younst objects are older than the 28 days, I will see 35 days of recovery points2. At the end of the job this chain will be removed leaving me with the required minimum 28 days of recovery points.

Are my assumptions correct?

[AlexHeylin]

@J.Leeflang I can't give an authoritative answer, but all our VBR jobs which end up going to (Wasabi) immutable S3 use a weekly synthetic full for various reasons and we've never seen the issue you're saying with objects being kept "forever". I feel like a full backup shouldn't be required - but I've not tested immutable S3 without weekly synth fulls. There are some configurations where Veeam doesn't prevent you from creating a set of conditions (in case you have a use case for them) but they may not interact as you expect. Perhaps this is one of them. There's a great diagram somewhere of why retention keeps more points than you might expect, but I can't find it right now.

[RubinCompServ]

I enabled a single Weekly restore point for our jobs. This is a standard single GFS Weekly full. Not an active full. There is no need for this.
This way the Active Chain in the Wasabi storage is shortened to 7 days of objects.
As soon as this entire chain is older then the set retention for immutablity, the normal retention process takes over.

As soon as the youngest objects belonging to the latest incremental file in this chain gets older then the regular retention setting, this entire chain will be removed.
If retention is set to 28 days. I will see recovery points up to 34 days old. As soon as the younst objects are older than the 28 days, I will see 35 days of recovery points2. At the end of the job this chain will be removed leaving me with the required minimum 28 days of recovery points.

Are my assumptions correct?

I can't speak to S3, but this has been our experience with regular backup jobs to immutable storage as well. When the customer keeps a Monthly backup and immutable storage is set to 7 days, their backup chain is anywhere from 7-37 restore points and all immutable.