Using object storage as a backup target
Post Reply
pirx
Veeam Legend
Posts: 339
Liked: 32 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Immutable flag/duration somewhere visible in Veeam?

Post by pirx »

Hi,

a while ago we added S3 buckets with immutability (15 days) to out SOBR as replacement for our tape backups. We also had to reconfigure/rearrange backup jobs (forever inc with synt. full's as before but different grouping of VM's) and started with new jobs and new full backups. The old backup and backup copy jobs are still in Backups -> Disk (imported). As there is still a lot of backup space occupied by these old jobs, we wanted to delete them. The retention time of these jobs (14 days) is already expired (last backup 16. Sep). Then we started wondering that we could not delete the backups and I found out that this might me related to the additional period of up to 10 days for block generation.

Example:

Code: Select all

09.10.2020 07:58:18 Error    [BJ-xxxxx] Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 10 October 2020 03:47:21.
So I'll wait until that time and check again. My college told me that has seen similar messages and when he checked again the immutability did change to another time (+ 1 week I think). This is what I don't understand, as the jobs and the files are not in use anymore, I'd expect the immutability time to be stable.

Is there a way in Veeam to check the immutability date? It's hard to find the right blocks in S3 bucket that are related to a certain VM or backup job. It's a not that transparent what is going on for me.

In general we see quite some failures/warnings/erros with S3 as offload destination.

Example:

Here I wanted to delete an backup job where immutability was definitely expired and 1:18h later I got this error.

Code: Select all

09.10.2020 09:14:33 Error    [BJ-xxxx] Failed to delete backup Error: Amazon REST error: 'S3 error: Please reduce your request rate.
                             Code: SlowDown', error code: 503
                             Other: HostId: 'xxxxx'

Nearly all our offload jobs show a lot of these failures for the included VM's. That makes it hard to check if everything is ok.

Code: Select all

09.10.2020 05:07:55 :: Processing BJ-xxxxx VMname Error: Transaction (Process ID 406) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.  


https://imgur.com/BSQVKhn


It's probalby best that I open a new thread for our general S3 problems.

Egor Yakovlev
Veeam Software
Posts: 2233
Liked: 544 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Egor Yakovlev »

Hi Pirx,

we do have some ideas to make immutable backups more transparent in VBR UI in future versions.
Thanks for feedback!

/Cheers!

vashthestampede
Novice
Posts: 7
Liked: 1 time
Joined: Dec 10, 2019 10:46 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by vashthestampede »

Hello,
i just upgraded to VBR 11, i see the column "Immutable Until" in a webinair regarding Linux Backup, but why i can't see it in my S3 Backup Repsitory?

Egor Yakovlev
Veeam Software
Posts: 2233
Liked: 544 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Egor Yakovlev »

Currently this column does not persist in all available immutability cases.
We will expand the list of applicable places in the UI with future updates.

/Thanks!

vashthestampede
Novice
Posts: 7
Liked: 1 time
Joined: Dec 10, 2019 10:46 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by vashthestampede »

What do you mean for "all available immutability cases", in my knoledge there are only 2 cases: S3/S3 compatible storage and Linux Hardned, so in wich of these two cases the colum not presist? I hope not in the most used and most diffused S3 storage!

Gostev
SVP, Product Management
Posts: 28942
Liked: 5291 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

vashthestampede wrote: Mar 29, 2021 9:21 ami just upgraded to VBR 11, i see the column "Immutable Until" in a webinair regarding Linux Backup, but why i can't see it in my S3 Backup Repsitory?
Clearly, you are already aware it's not available for S3 backup repositories at this time?

vashthestampede
Novice
Posts: 7
Liked: 1 time
Joined: Dec 10, 2019 10:46 am
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by vashthestampede »

Yes, but i wanted to be wrong! The next question is, why add this very usefull information only in the less uses case?

Gostev
SVP, Product Management
Posts: 28942
Liked: 5291 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

I guess it was not clear how useful it is, because it was never requested by customers before... it was our own idea. But now that we know it is useful, we can expand this to other features too.

pirx
Veeam Legend
Posts: 339
Liked: 32 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by pirx »

@Gostev what do you mean with no requested before? I'm struggling with this since October last year and I think I've mentioned to support multiple times that it is necessary that the immutable date is displayed. Not sure when you started to implement this.

Gostev
SVP, Product Management
Posts: 28942
Liked: 5291 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutable flag/duration somewhere visible in Veeam?

Post by Gostev »

October was "a bit" too late for including it in V11, which passed the feature-lock stage in summer ;)

Please don't treat this as a guarantee it will be included in V12 though. Keep in mind we have hundreds of other pending enhancements that have been requested by many more users for many more years. So in general, you should not expect instant addition of any feature request based on the very first ask. They will have to get in line, especially those that are not trivial to implement. For example, and I don't know if this is the case here, but if there's no simple and efficient way to query immutability expiration dates for backups in object storage, because the architecture was not designed with this requirement in mind - then we will likely prioritize implementing 10 other, simpler UI feature requests to re-writing the architecture just to implement this one. And in other cases, the feature can be so simple to deliver that we can even add it in a minor update... as always, "it depends".

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests