Discussions related to using object storage as a backup target.
Post Reply
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Immutable flag/duration somewhere visible in Veeam?

Post by pirx »

Hi,

a while ago we added S3 buckets with immutability (15 days) to out SOBR as replacement for our tape backups. We also had to reconfigure/rearrange backup jobs (forever inc with synt. full's as before but different grouping of VM's) and started with new jobs and new full backups. The old backup and backup copy jobs are still in Backups -> Disk (imported). As there is still a lot of backup space occupied by these old jobs, we wanted to delete them. The retention time of these jobs (14 days) is already expired (last backup 16. Sep). Then we started wondering that we could not delete the backups and I found out that this might me related to the additional period of up to 10 days for block generation.

Example:

Code: Select all

09.10.2020 07:58:18 Error    [BJ-xxxxx] Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 10 October 2020 03:47:21.
So I'll wait until that time and check again. My college told me that has seen similar messages and when he checked again the immutability did change to another time (+ 1 week I think). This is what I don't understand, as the jobs and the files are not in use anymore, I'd expect the immutability time to be stable.

Is there a way in Veeam to check the immutability date? It's hard to find the right blocks in S3 bucket that are related to a certain VM or backup job. It's a not that transparent what is going on for me.

In general we see quite some failures/warnings/erros with S3 as offload destination.

Example:

Here I wanted to delete an backup job where immutability was definitely expired and 1:18h later I got this error.

Code: Select all

09.10.2020 09:14:33 Error    [BJ-xxxx] Failed to delete backup Error: Amazon REST error: 'S3 error: Please reduce your request rate.
                             Code: SlowDown', error code: 503
                             Other: HostId: 'xxxxx'

Nearly all our offload jobs show a lot of these failures for the included VM's. That makes it hard to check if everything is ok.

Code: Select all

09.10.2020 05:07:55 :: Processing BJ-xxxxx VMname Error: Transaction (Process ID 406) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.  


https://imgur.com/BSQVKhn


It's probalby best that I open a new thread for our general S3 problems.
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Egor Yakovlev »

Hi Pirx,

we do have some ideas to make immutable backups more transparent in VBR UI in future versions.
Thanks for feedback!

/Cheers!
vashthestampede
Novice
Posts: 7
Liked: 1 time
Joined: Dec 10, 2019 10:46 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by vashthestampede »

Hello,
i just upgraded to VBR 11, i see the column "Immutable Until" in a webinair regarding Linux Backup, but why i can't see it in my S3 Backup Repsitory?
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Egor Yakovlev »

Currently this column does not persist in all available immutability cases.
We will expand the list of applicable places in the UI with future updates.

/Thanks!
vashthestampede
Novice
Posts: 7
Liked: 1 time
Joined: Dec 10, 2019 10:46 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by vashthestampede »

What do you mean for "all available immutability cases", in my knoledge there are only 2 cases: S3/S3 compatible storage and Linux Hardned, so in wich of these two cases the colum not presist? I hope not in the most used and most diffused S3 storage!
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

vashthestampede wrote: Mar 29, 2021 9:21 ami just upgraded to VBR 11, i see the column "Immutable Until" in a webinair regarding Linux Backup, but why i can't see it in my S3 Backup Repsitory?
Clearly, you are already aware it's not available for S3 backup repositories at this time?
vashthestampede
Novice
Posts: 7
Liked: 1 time
Joined: Dec 10, 2019 10:46 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by vashthestampede »

Yes, but i wanted to be wrong! The next question is, why add this very usefull information only in the less uses case?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

I guess it was not clear how useful it is, because it was never requested by customers before... it was our own idea. But now that we know it is useful, we can expand this to other features too.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by pirx »

@Gostev what do you mean with no requested before? I'm struggling with this since October last year and I think I've mentioned to support multiple times that it is necessary that the immutable date is displayed. Not sure when you started to implement this.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev » 1 person likes this post

October was "a bit" too late for including it in V11, which passed the feature-lock stage in summer ;)

Please don't treat this as a guarantee it will be included in V12 though. Keep in mind we have hundreds of other pending enhancements that have been requested by many more users for many more years. So in general, you should not expect instant addition of any feature request based on the very first ask. They will have to get in line, especially those that are not trivial to implement. For example, and I don't know if this is the case here, but if there's no simple and efficient way to query immutability expiration dates for backups in object storage, because the architecture was not designed with this requirement in mind - then we will likely prioritize implementing 10 other, simpler UI feature requests to re-writing the architecture just to implement this one. And in other cases, the feature can be so simple to deliver that we can even add it in a minor update... as always, "it depends".
roland
Influencer
Posts: 12
Liked: 1 time
Joined: Oct 25, 2010 4:15 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by roland »

This is an old post but i'm very interrested for this feature, we need to know for regulatory reason which backup is protected and until when.
Is this immutability date available somewhere in v12 ? or even by powershell scripts?
Thanks
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

Please check out backup properties, as starting from V12 you should see "Immutable until" value at least for some repository types (I don't remember what are the limitations if any). Thanks
roland
Influencer
Posts: 12
Liked: 1 time
Joined: Oct 25, 2010 4:15 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by roland »

This repository is connect directly to an S3 Compatible Storage and the bucket is configured in veeam with a number of days of immutabilty.
Unfortunately I did'nt find any immutability date in backup property, only Name, Data size, BackupSize, deduplication, Compression, Date, Retention (R W etc)
No date information about Immutability of each backup.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

@Egor Yakovlev any comments? I think you were adding this column.
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Egor Yakovlev » 1 person likes this post

Backups on object storage repositories were excluded from "Immutable Until" column cases intentionally, because there is no clear way to determine immutability date with current database structure and core mechanics behind it. There are dozens of scenarios around restore point visibility in UI and a real immutability date in the object storage - a change of a SOBR tier, download\partial download, re-use of blocks from one object to another, change of retention, etc. At this stage calculating immutability date for all the possible scenarios will require a tremendous amount of work, and at the end will still have some corner cases where state of the restore point in the UI and a real state of immutability date tag on the object storage might differ from one another. However, once we switch from "restore points" logic to "time-based" retention in the future, that problem will go away and we will be able to add true immutability dates to the UI for backups kept on object storage.
roland
Influencer
Posts: 12
Liked: 1 time
Joined: Oct 25, 2010 4:15 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by roland »

Thanks for this answer even if it's not satisfying my needs... :?

What about this ? :idea:
You know every blocks that compose a backup and because you have on each block access to metadata and immut date (ie Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 10 October 2020) you could probably query all of them and their metadata and choose the minus date. (the worst date in case a hacker gain access to object storage and delete blocks or files) ?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

Even the "query them all" part alone is a problem due to their sheer number ;) I agree, this will have to wait for the simplified immutability architecture that will be made possible once we no longer have to support legacy restore point based retention.
roland
Influencer
Posts: 12
Liked: 1 time
Joined: Oct 25, 2010 4:15 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by roland »

The main goal of this information for me is to know the earliest immutable date of the backup, even if this date is not the same for all blocks that compose the backup.
I understand that because the backup is made of multiple blocks and some are re-used they probably have different date.
Due to immutability, if a block is changed or re-used its date can only be the same or higher, no ?
So i guess this date can be compute during the first full backup without extra work, just keeping in mind for each block processed the earliest date, and updating that information in the backup ?
This date should be the minimum guarantee date for the backup.
Sorry about that, it's just an idea, probably i'm wrong, i don't know the internal process of veeam.
It's also interresting to know when veeam do an incremental backup if the unmodified blocks (but part of the global backup chain) are update on block storage (only the immutability date) to respect the overall integrity.
sfirmes
Veeam Software
Posts: 223
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by sfirmes »

@roland are you looking to see what the immutability dates are for a particular backup job? For example, if I was to do an initial full backup today, are you interested in what the immutabilty/object lock date for the objects in that backup are?
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
roland
Influencer
Posts: 12
Liked: 1 time
Joined: Oct 25, 2010 4:15 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by roland »

@sfirmes Yes exactly :) , and i'm also interrested to know if during an incremental backup the unmodified blocks (but part of the global backup chain) are update on block storage (only the immutability date) to respect the overall integrity.
restore-helper
Influencer
Posts: 24
Liked: 3 times
Joined: Oct 11, 2018 7:38 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by restore-helper »

+1
I too would live to see this immutable dates for S3 compatible storage.
sfirmes
Veeam Software
Posts: 223
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by sfirmes »

As Egor noted earlier, the complexity with the current immutability model makes showing the immutabilty date too difficult.
But he and Anton both mentioned that in the future our "time-based" immutability model will allow for this capability:
Egor Yakovlev wrote: Jan 04, 2024 11:08 am However, once we switch from "restore points" logic to "time-based" retention in the future, that problem will go away and we will be able to add true immutability dates to the UI for backups kept on object storage.
Thanks

Steve
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
Post Reply

Who is online

Users browsing this forum: JaySt and 17 guests