Ever Changing Immutability Dates

[vertices]

I thought I had a handle on immutability but apparently no. As an example, a customer that we no longer have.The backups have not been running. The VMs no longer exist. Immutability was set to 30 days. At first it was telling me I can't delete the backups as they were immutable until sometime in November. Ok I'll wait. The jobs that placed them there don't even exist anymore. They are orphaned. I then try to delete after that date and it said they were immutable until Dec 15th now, no can do. Ok. No problem. I wait again. Now I try to delete them again and it says sorry, immutable until January 4th.

At this point, Veeam appears to be constantly moving the immutability date of these backups forward even though they are not running or being accessed in any way, and therefore I am never allowed to delete them. How does one get out of this debacle?

Thanks.

[Mildur]

Hi Rob

Is this about object storage or linux hardened repo?

For example, With hardened repo, the immutability is set for the entire gfs retention time. If you had monthly full backups to keep for 6 months, then the immutability for each restore point will be 6 month starting at the creation date.

For object storage, the immutability of the objects will be as configured in the object storage properties plus 10 days.

[vertices]

Hmm. This is a SOBR with a hardened linux repo and a capacity tier at wasabi. I did have 3 monthly GFS retention on it. Ok so that makes sense then. Although I’m still not sure why Veeam doesn’t actually know the immutability date and every time it approaches, Veeam then decides nope, it’s actually longer. Shouldn’t Veeam know the real immutability date?

[Mildur]

It could be, that veeam shows the immutability date from the first file it cannot delete when you try it.

You can check for the immutability date in the backup properties.

For SOBR with copy policy, gfs restore point are immutable their entire retention time. Or the configured value in the hardened repo. Gfs retention and hardened repo setting wilm be compared and the longest period will be set for the gfs fullbackup.

For SOBR with move policy, they only are immutable as you have configured in the linux hardened repo property.

[vertices]

So if a person sets a GFS retention time of 2 years, perhaps 6 monthlies, and 2 yearlies, but the immutability period was set to only 7 days, then 3 months later they want to kill the whole thing, they are just stuck with it for 2 years in the cloud? Obviously I could delete them locally with admin access to the local repos, I could just delete them at the file level I would imagine, although I haven't tried. But then because we had selected to keep 2 GFS annual backups, that we no longer need, we would be stuck with that in Wasabi for 2 years with no way out?

[vertices]

Well you def can't simply login to the repo and delete it there at the file level. Operation not permitted. So that's good, as far as security, but I am trying to figure out what I can do. I may have had this set to keep 2 annuals. I can't remember, I have been going through multiple builds of this deployment and making lots of changes, and the original job is gone so I can't review it.

So if a backup was copied to a SOBR, with a short immutability period, but the backups also had some annual GFS retention enabled, then those backups are just locked in on both the local repo and s3 storage for 2 years?

What if you still have the job available? Can you edit the GFS retention on a job, dropping the old ones, so old ones are purged at next run, and they would be provided they were past the immutability period? Or is it that once a job is set to retain a GFS backup on a certain schedule, you simply cannot change it later?

If I have a VM set to retain 2 yearly GFS backups. It's set to immutability of 7 days. I then delete and retire the VM 6 months later. I am forced to store those backups for a full 2 years even though immutability was only set to 7 days?

I don't know, trying to wrap my heard around how immutability plus GFS combine to ultimately determine when it can be deleted, and how this can be managed. It's starting to sound to me that if you ever enable GFS, then the immutability period is completely irrelevant as even if it was set to 1 day only, if you enabled multiple annual GFS backups at any time, you are stuck with those for the entire period.

[Mildur]

Sorry for not answering. Was busy the entire day

You can change the gfs setting in a job, but existing gfs restore points will not be affected. They keep their immutability date.

Well you def can't simply login to the repo and delete it there at the file level.

Actually you can. Removing the immutable flag is possible with sudo/root permissions (sudo chattr -i /foldername). After that, the files should be removable. But it can mess up with the sobr metadata. There was a topic in the forums about that. I‘m not sure if that‘s resolved now.

[vertices]

I guess I can open a ticket, as I still don't understand how an immutability date is set.

Job is created on day 1 to backup a VM every 4 hours and is set to have GFS of 4 weekly, 6 monthly, and 2 yearly. Immutability is set to 7 days.
Job runs according to schedule for 3 months.
On Day 91, I retire the VM.

When can I delete the backups in the above scenario?

[TDog]

Did you ever get an answer @vertices? I'm seeing the same thing with a SOBR offloading to Wasabi. I waited the month, waited again, still nothing. I came across this thread while trying to find an answer.

[Gostev]

Job is created on day 1 to backup a VM every 4 hours and is set to have GFS of 4 weekly, 6 monthly, and 2 yearly. Immutability is set to 7 days.
Job runs according to schedule for 3 months.
On Day 91, I retire the VM.

When can I delete the backups in the above scenario?

Assuming you're asking about hardened repository:
You will be able to delete each weekly 4 week after, each monthly 6 months after, and each yearly 2 years after their creation time.

[Gostev]

Did you ever get an answer vertices? I'm seeing the same thing with a SOBR offloading to Wasabi. I waited the month, waited again, still nothing. I came across this thread while trying to find an answer.

Assuming you're asking about Capacity Tier specifically (Wasabi), you will be able to delete them after the immutability period set in your object storage repository setting plus up to 10 days (depending on what day you're attempting to delete backup).

[AndBig]

Wow, very interesting thing. I had to go try it.
I have a Hardened repo with a 7 day retention set. If I simply back up the incremental forewer with a retention of 31 days, then I can delete the backups after 8 days. If I have a backup job with a GFS configuration of 31 days, 2 weeks and 2 months, then the VM, which I backed up only between 20.11.-29.11. I can't delete anything until 8.1.22! In the properties I see that the protection (Immutable Until) is set only on two weekly backups (vbk). I can't even delete vib alone. This completely changes my view of the Hardened repo in conjunction with GFS.

[Gostev]

Wow, it almost feels like you're not actually using our product because pointing forever incremental backup job to a hardened repository with immutability enabled is impossible in principle (second bullet under Limitations). In any case, you should be able to delete all backups which are not marked as immutable and are not dependencies of backups marked as immutable. If your experience is any different, then have our support take a look at your situation closer.

[mcz]

You can check for the immutability date in the backup properties.

Is that only related to hardened repository? My backups on the object storage (SOBR archive tier) do not show any immutability flags on the properties page at all...

[Mildur]

Yes, only in hardened Repo.
But there was an answer in the forums from a veeam official that they are checking to add this feature to object storage too in a future version.

[vertices]

Sorry have been out on holiday for a bit. Ok this also completely changes my understanding of how Veeam and immutability work with GFS. Gostev can you confirm if immutability is flagged with the same date on both the performance and capacity extents when using a SOBR with GFS? Meaning that if both extents in the SOBR are set to 7 days immutability, but GFS is set to keep 6 monthlies, that the oldest monthly on both with have a 6 month immutability date? It seems that if you have any form of GFS going to a SOBR with perf/capac extents, that the GFS completely overrides any/all immutability dates on both, even if immutability is set to only 1 day. Is that correct?

[Gostev]

In case of SOBR, it depends on the Capacity Tier policy... but in a very straightforward way that is based on common sense so you don't even need documentation to guess how it works. For example, Move policy would not be able to move GFS backups from Performance Tier to Capacity Tier according to offload window settings, if those backups were made immutable for their entire lifetime.