Immutable flag/duration somewhere visible in Veeam?

[pirx]

Hi,

a while ago we added S3 buckets with immutability (15 days) to out SOBR as replacement for our tape backups. We also had to reconfigure/rearrange backup jobs (forever inc with synt. full's as before but different grouping of VM's) and started with new jobs and new full backups. The old backup and backup copy jobs are still in Backups -> Disk (imported). As there is still a lot of backup space occupied by these old jobs, we wanted to delete them. The retention time of these jobs (14 days) is already expired (last backup 16. Sep). Then we started wondering that we could not delete the backups and I found out that this might me related to the additional period of up to 10 days for block generation.

Example:

Code: Select all

09.10.2020 07:58:18 Error    [BJ-xxxxx] Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 10 October 2020 03:47:21.

So I'll wait until that time and check again. My college told me that has seen similar messages and when he checked again the immutability did change to another time (+ 1 week I think). This is what I don't understand, as the jobs and the files are not in use anymore, I'd expect the immutability time to be stable.

Is there a way in Veeam to check the immutability date? It's hard to find the right blocks in S3 bucket that are related to a certain VM or backup job. It's a not that transparent what is going on for me.

In general we see quite some failures/warnings/erros with S3 as offload destination.

Example:

Here I wanted to delete an backup job where immutability was definitely expired and 1:18h later I got this error.

Code: Select all

09.10.2020 09:14:33 Error    [BJ-xxxx] Failed to delete backup Error: Amazon REST error: 'S3 error: Please reduce your request rate.
                             Code: SlowDown', error code: 503
                             Other: HostId: 'xxxxx'

Nearly all our offload jobs show a lot of these failures for the included VM's. That makes it hard to check if everything is ok.

Code: Select all

09.10.2020 05:07:55 :: Processing BJ-xxxxx VMname Error: Transaction (Process ID 406) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.  

https://imgur.com/BSQVKhn


It's probalby best that I open a new thread for our general S3 problems.

[Egor Yakovlev]

Hi Pirx,

we do have some ideas to make immutable backups more transparent in VBR UI in future versions.
Thanks for feedback!

/Cheers!

[vashthestampede]

Hello,
i just upgraded to VBR 11, i see the column "Immutable Until" in a webinair regarding Linux Backup, but why i can't see it in my S3 Backup Repsitory?

[Egor Yakovlev]

Currently this column does not persist in all available immutability cases.
We will expand the list of applicable places in the UI with future updates.

/Thanks!

[vashthestampede]

What do you mean for "all available immutability cases", in my knoledge there are only 2 cases: S3/S3 compatible storage and Linux Hardned, so in wich of these two cases the colum not presist? I hope not in the most used and most diffused S3 storage!

[Gostev]

i just upgraded to VBR 11, i see the column "Immutable Until" in a webinair regarding Linux Backup, but why i can't see it in my S3 Backup Repsitory?

Clearly, you are already aware it's not available for S3 backup repositories at this time?

[vashthestampede]

Yes, but i wanted to be wrong! The next question is, why add this very usefull information only in the less uses case?

[Gostev]

I guess it was not clear how useful it is, because it was never requested by customers before... it was our own idea. But now that we know it is useful, we can expand this to other features too.

[pirx]

@Gostev what do you mean with no requested before? I'm struggling with this since October last year and I think I've mentioned to support multiple times that it is necessary that the immutable date is displayed. Not sure when you started to implement this.

[Gostev]

October was "a bit" too late for including it in V11, which passed the feature-lock stage in summer

Please don't treat this as a guarantee it will be included in V12 though. Keep in mind we have hundreds of other pending enhancements that have been requested by many more users for many more years. So in general, you should not expect instant addition of any feature request based on the very first ask. They will have to get in line, especially those that are not trivial to implement. For example, and I don't know if this is the case here, but if there's no simple and efficient way to query immutability expiration dates for backups in object storage, because the architecture was not designed with this requirement in mind - then we will likely prioritize implementing 10 other, simpler UI feature requests to re-writing the architecture just to implement this one. And in other cases, the feature can be so simple to deliver that we can even add it in a minor update... as always, "it depends".

[roland]

This is an old post but i'm very interrested for this feature, we need to know for regulatory reason which backup is protected and until when.
Is this immutability date available somewhere in v12 ? or even by powershell scripts?
Thanks

[Gostev]

Please check out backup properties, as starting from V12 you should see "Immutable until" value at least for some repository types (I don't remember what are the limitations if any). Thanks

[roland]

This repository is connect directly to an S3 Compatible Storage and the bucket is configured in veeam with a number of days of immutabilty.
Unfortunately I did'nt find any immutability date in backup property, only Name, Data size, BackupSize, deduplication, Compression, Date, Retention (R W etc)
No date information about Immutability of each backup.

[Gostev]

@Egor Yakovlev any comments? I think you were adding this column.

[Egor Yakovlev]

Backups on object storage repositories were excluded from "Immutable Until" column cases intentionally, because there is no clear way to determine immutability date with current database structure and core mechanics behind it. There are dozens of scenarios around restore point visibility in UI and a real immutability date in the object storage - a change of a SOBR tier, download\partial download, re-use of blocks from one object to another, change of retention, etc. At this stage calculating immutability date for all the possible scenarios will require a tremendous amount of work, and at the end will still have some corner cases where state of the restore point in the UI and a real state of immutability date tag on the object storage might differ from one another. However, once we switch from "restore points" logic to "time-based" retention in the future, that problem will go away and we will be able to add true immutability dates to the UI for backups kept on object storage.

[roland]

Thanks for this answer even if it's not satisfying my needs...

What about this ?
You know every blocks that compose a backup and because you have on each block access to metadata and immut date (ie Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 10 October 2020) you could probably query all of them and their metadata and choose the minus date. (the worst date in case a hacker gain access to object storage and delete blocks or files) ?

[Gostev]

Even the "query them all" part alone is a problem due to their sheer number I agree, this will have to wait for the simplified immutability architecture that will be made possible once we no longer have to support legacy restore point based retention.

[roland]

The main goal of this information for me is to know the earliest immutable date of the backup, even if this date is not the same for all blocks that compose the backup.
I understand that because the backup is made of multiple blocks and some are re-used they probably have different date.
Due to immutability, if a block is changed or re-used its date can only be the same or higher, no ?
So i guess this date can be compute during the first full backup without extra work, just keeping in mind for each block processed the earliest date, and updating that information in the backup ?
This date should be the minimum guarantee date for the backup.
Sorry about that, it's just an idea, probably i'm wrong, i don't know the internal process of veeam.
It's also interresting to know when veeam do an incremental backup if the unmodified blocks (but part of the global backup chain) are update on block storage (only the immutability date) to respect the overall integrity.

[sfirmes]

@roland are you looking to see what the immutability dates are for a particular backup job? For example, if I was to do an initial full backup today, are you interested in what the immutabilty/object lock date for the objects in that backup are?

[roland]

@sfirmes Yes exactly , and i'm also interrested to know if during an incremental backup the unmodified blocks (but part of the global backup chain) are update on block storage (only the immutability date) to respect the overall integrity.

[restore-helper]

+1
I too would live to see this immutable dates for S3 compatible storage.

[sfirmes]

As Egor noted earlier, the complexity with the current immutability model makes showing the immutabilty date too difficult.
But he and Anton both mentioned that in the future our "time-based" immutability model will allow for this capability:

However, once we switch from "restore points" logic to "time-based" retention in the future, that problem will go away and we will be able to add true immutability dates to the UI for backups kept on object storage.

Thanks

Steve

[mspencerl87]

In a Linux hardened repository. When we "View Backup Proprieties" we can see when a restore point's immutability will expire.
Can we get the same thing for S3 Repositories? DBOS/SOBRS?

[david.domask]

Hi Spencer,

I've merged your previous topic with an existing topic on the same subject. Please see Steve Firmes (sfirmes) and Egor Yakolev's answers above.

Thanks!