Using object storage as a backup target
pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Explanation of S3 Immutability + Block Generation needed

Post by pirx »

Hi,

I (or better all colleagues here working with Veeam) have some problems with the Immutability feature. Our goal was to protect the Backup Jobs and Backup Copy Jobs (forever forward inc + syn full) that were transfered to S3 as long as our retention time is (10 weeks). For whatever reason immutabilty was set to only 30 days - this is something we probably have to fix, but we first need to understand how it _really_ works.

What we currently see is that backups/copies have much longer immutability than we have configured. Even for jobs that had their backup on the same day, we see different immutability dates varying from 10.11.2020 - 22.12.2020. Inside this jobs there are VM that have their latest backup from July. But immutability seems not to make a difference when the individual VM's were backed up.

Code: Select all

Jobname generation_expiration_date_utc	 generation_immutable_until_utc 	last_checkpoint_cleanup_date_utc	Last Backup -  Disk (Imported)         Days
#1	     11.10.2020                     	     10.11.2020	                                       21.10.2020	                                       20.09	                                       36
#2	     11.10.2020                                10.11.2020                                         01.10.2020	                                       13.08	                                               63
#3	     11.10.2020	                             10.11.2020	                                       21.10.2020                                       16.09	                                               39
#4          23.10.2020	                              22.11.2020	                                       20.10.2020	                                      10.08	                                               75
#5	     13.10.2020	                             12.11.2020	                                       21.10.2020	                                      14.09                                                43
#6	                                                      11.11.2020	                                                                                              16.09	                                               40
#7 	     22.11.2020	                               22.12.2020	                                       12.11.2020	                                      16.09                                                 70
I found the articles about immutability and block generations.

https://helpcenter.veeam.com/docs/backu ... ml?ver=100
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Maybe I simply do not understand this. But it seems that we never can be sure when data actually can be deleted. We found jobs in Disk (Imported) that were 120 days old and we wanted to delete them, but we got the message that it's still protected for some weeks.

What are we doing wrong? Is our goal to keep the jobs immutable for the time of our retention period and then be able to delete older data no possible because of block generation?

Gostev
SVP, Product Management
Posts: 29896
Liked: 5841 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by Gostev »

pirx wrote: Nov 18, 2020 11:21 amWe found jobs in Disk (Imported) that were 120 days old and we wanted to delete them, but we got the message that it's still protected for some weeks.
This is something you want to investigate with support (if your immutability period was set to 30 days). Restore points should not be protected for much longer than the immutability period you have set in the object storage repository settings.

There's this API costs optimization that extends immutability on each new object for up to 10 days by default, just so we don't have to advance immutability lock on most blocks each day. Because with typical 2% daily change rate, each new restore points consists 98% from existing blocks that were already in object storage. So in the expectation that vast majority of blocks in full backup will be reused in the next 10 restore points, we set their immutability for longer right away - and don't need to come back to them later as the result.

But other than that, immutability period should stick fairly close to your setting. So, don't just assume your understanding is not correct. It is more likely that you're facing a bug in some scenario our QC missed. I checked the User Guide links above and they provide a solid description of how it is supposed to work and I've nothing to add to them really.

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

Hi Gostev,

thanks for your reply. I am (and have been) in contact with support regarding this (00757218). Maybe you or someone else can help me with this example of a job that is still protected.

I get this error message:

Code: Select all

19.11.2020 08:02:02 Error    [BCJ_BJ-xxxx0400-0449-yyyy] Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 25 November 2020 02:39:12.
The last backup job was from mid August, so more than 100 days dago. Immutability is set to 30 days on this S3 SOBR bucket. I understand that not the backup time is relevant, but the time of the offload.

Image


From support I got a sql query that seems to return something about Immutability from DB - as there is currently 0 in the GUI and as far as I can see, nothing helpful in powershell, which is bad, because you have no easy way to check what is going on with Immutability or when data/jobs will really expire. Imagine you set different values or changed the Immutability for whatever reason.

Code: Select all

SELECT generation_expiration_date_utc, generation_immutable_until_utc, last_checkpoint_cleanup_date_utc FROM [VEEAMBR].[dbo].[Backup.Model.BackupArchiveIndices] 
where backup_id in (select id from [VEEAMBR].[dbo].[Backup.Model.Backups] where job_name = 'BCJ_BJ-xxxx-0400-0449-yyyy')

generation_expiration_date_utc generation_immutable_until_utc last_checkpoint_cleanup_date_utc
------------------------------ ------------------------------ --------------------------------
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:16             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:18             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:50:20             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:18             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:19             
26.10.2020 01:37:24            25.11.2020 01:37:24            20.10.2020 06:50:29             
16.10.2020 09:28:27            15.11.2020 09:28:27            20.10.2020 06:50:20             
26.10.2020 01:36:33            25.11.2020 01:36:33            20.10.2020 06:50:44             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:51:09             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:50:25             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:22             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:50:23             
16.10.2020 09:28:27            15.11.2020 09:28:27            20.10.2020 06:50:24             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:25             
26.10.2020 00:43:54            25.11.2020 00:43:54            20.10.2020 06:51:07             
26.10.2020 00:43:52            25.11.2020 00:43:52            20.10.2020 06:51:15             
26.10.2020 01:39:10            25.11.2020 01:39:10            20.10.2020 06:51:28             
26.10.2020 01:39:58            25.11.2020 01:39:58            20.10.2020 06:51:35             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:33             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:34             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:30             
16.10.2020 09:28:27            15.11.2020 09:28:27            20.10.2020 06:50:30             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:51:51             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:35             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:31             
25.10.2020 10:18:17            24.11.2020 10:18:17            20.10.2020 06:50:32             
26.10.2020 01:37:24            25.11.2020 01:37:24            20.10.2020 06:52:04             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:40             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:33             
26.10.2020 01:39:12            25.11.2020 01:39:12            20.10.2020 06:52:12

These dates do not match my expectations. I also do not exactly know what generation_expiration_date_utc, generation_immutable_until_utc, last_checkpoint_cleanup_date_utc are about or what the difference is.

Note: this jobs was disabled for some time and then completely deleted, the data is now under Disk (Imported). Maybe we hit a cornercase here, but still I find it not easy to understand why this data is still protected.

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

Kindly, keep working with our support team, as neither the described behaviour nor the provided dates looks expected. The only reason I can think of why is that immutability period has been changed down the road - from 100 days or something (old value) to 30 days (current value). Thanks!

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

We are working with support, but it seems that this topic is not that easy and a bit bleeding edge.

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

I've passed the information to QA team, we will investigate the reported issue. Thanks!

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

Can you provide a valid support ticket number? The one mentioned above seems incorrect - 00757218. Also, if you haven't attached database dump to support ticket, kindly, do so. The QA team will take a look at the case within next couple of days. Thanks!

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

Right, it's ticket 04443517. I'm also working on some other cases with support where we talked about the status of our S3 repository.

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

Passed to QA team, will share more information, once I have it. Thanks!

scottf99
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jul 29, 2013 2:13 am
Full Name: Scott
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by scottf99 »

Hi I posted a similar question to this forum. I was asked to create a support ticket and haven't yet but can I ask has anything come out of this ticket/thread as it is exactly what I am seeing.
Cheers
Scott

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

No, I did not get any feedback yet.

scottf99
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jul 29, 2013 2:13 am
Full Name: Scott
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by scottf99 »

Is there any more information to share on this? I have old VMs that have not existed for months that are still in Backup Copy and Object Storage as they cannot be deleted. It looks like that if there are still current VMs being backed up in the same job then the immutability is always moved forward for the old VMs as well.
Cheers
Scott

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

Sounds identical to my problem. Until now, nobody could explain this behavior. We also have to check object storage cleanup now, as we have copy jobs that do not have any data in S3 according to Veeam Console but there are still a lot of TB occupied (>40TB) in the corresponding AWS S3 folder, objects in S3 are from December/January. But I don't see how I can correlate these objects to backup jobs or files.

scottf99
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jul 29, 2013 2:13 am
Full Name: Scott
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by scottf99 »

How is your support ticket going?

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

I never got feedback from QA team. I've had a talk with our presales contacts at Veeam, also with support for a couple of times, but they do not yet have an explanation why we have data that is much older than our retention which is still immutable. Nobody could really explain how block generation is working and what blocks are reused. If only blocks in an active chain are reused, it would not explain why we really old vbk files that still are immutable.

Gostev
SVP, Product Management
Posts: 29896
Liked: 5841 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by Gostev »

@veremin please provide an update, it's been over 2 months since your last response.

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

Apologize for any inconvenience caused by my late response.

The provided ticket had been closed about one month before the topic started. So, we could not provide any assistance within the old and long forgotten case.

We're still eager to help, but in order for us to help, may you collect fresh debug logs and attach them to new support ticket? After that kindly post ticket number here and we will make sure that case is given due attention.

Thanks!

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

[MERGED]SOBR and S3 immutability... again

Post by pirx »

I'm having problems with immutability since we started using SOBR + capacity tier. And yet nobody could really explain why backups are immutable much longer than expected. Even with 10 days block generation kept in mind.

Example: last backup of a VM was on 03.01.2022. We have a retention time of 70 days (10 weeks) and 14 days of immutability. How long should the VM be immutable and protected from deletion? If I add 70 days to 03.01 then it would be 14.03. I tried to delete the VM in that week and got an error that it is still immutable (I think it was beginning of April). Now it's mid April and I still can't delete the VM, the message tells me its immutable until 07.05. This is more than 120 days since the last backup and more then 50 days longer than our expected retention. And I do not yet believe that I will be able to delete the VM on that date...

Problem: it's not just one VM, we had to clone jobs due to bugs where offloading was not working any longer and we had to start a new chain for all affected VM's. So this also about storage that is used longer than expected.

But in the end it's about understanding the logic. And what setting we should choose for immutability. First we had set immutability to the same value as retention time. Then support suggested to only use 7 or 14 days for immutability.




Image

Image

Image

Image

Mildur
Veeam Software
Posts: 3123
Liked: 1075 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: St. Gallen, Switzerland
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by Mildur »

Hi pirx

I found your old topic about the same issue and decided to merge them.
Unfortunately, the last case a year ago was closed.

Can you please collect the debug logs and create a new case?
Product Management Analyst @ Veeam Software

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

I'll not open another case. I've had enough discussion about immutability in cases and provided logs in the past 1,5 years. No offense, I just like an explanation on how it should work and what _exactly_ would trigger extension of immutability over and over again (when a VM is not backed up anymore).

This explanation is highly confusing to me https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Gostev
SVP, Product Management
Posts: 29896
Liked: 5841 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by Gostev »

pirx wrote: Apr 20, 2022 3:03 pmwhat _exactly_ would trigger extension of immutability over and over again (when a VM is not backed up anymore).
For VMs which are not backed up anymore, that would be a bug I think. But at this time we're unaware of such bugs to exist. Also, if a VM is not being backed up, then the process that extends immutability should never be running for this VM in the first place, because this process is a part of the offload job which never runs for the given VM (as there are no new backups to process). So this is very strange in general.

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx »

Well, the affected VMs are still backed up. But in a temporary new backup job, as the old one has offload errors that can not be fixed and we needed to start new chains. But I guess this counts as "not backed up anymore".

I only checked backups on disk, now I checked on S3. It's even worse there, there are still backups from September to November. I guess I've to open a new case. Last year we found > 50 TB on S3 that was outdated an never deleted. It took support weeks/month to find a proper way to get rid of this.

Image

Gostev
SVP, Product Management
Posts: 29896
Liked: 5841 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by Gostev »

pirx wrote: Apr 20, 2022 3:51 pmLast year we found > 50 TB on S3 that was outdated an never deleted. It took support weeks/month to find a proper way to get rid of this.
Yes, one year ago there was a bug in rare circumstances that could lead to this. We addressed it in V11 and we even backported this fix to V10 (second to last cumulative patch for 10a).

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

I guess I've to open a new case.
Sounds like a good idea indeed, the reported behaviour (if persists after product update) does not look expected and needs further investigation. Kindly, share the ticket number, as soon as the case is opened, this way we will have QA team working on it from the very start (instead of digging into rather old situation). Thanks!

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx » 1 person likes this post

I created case 05407008 for this. But the reply so far is a bit... vague. Especially when I am talking about backups from last year or January. I hope that this is not due block generations.
To reduce I/O operations and associated costs, Veeam Backup & Replication will automatically add from 1 to 10 days to the immutability expiration date. This period is called Block Generation. You do not have to configure it, the Block Generation setting is applied automatically.
Based on this, the immutability period can be extended to a period longer than expected.

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

I think there might be a certain misunderstanding, cause even with the addition of block generation the immutability period (set for these old backups) looks longer than it should be. We have reached your support engineer and asked him to keep the ticket open for now, until QA team verify the debug logs and find the root cause there. Thanks!

scottf99
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jul 29, 2013 2:13 am
Full Name: Scott
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by scottf99 »

Hi has anything come of this issue? I have the same problem
Thanks
Scott

veremin
Product Manager
Posts: 19542
Liked: 2055 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by veremin »

Kindly reach our support team for further investigation - such type of issue cannot be addressed via forum correspondence, cause we need to check debug logs and see how immutability period was assigned to offloaded restore points. Once the ticket is opened, post its number here and we will follow the investigation internally. Thanks!

pirx
Veeam Legend
Posts: 415
Liked: 50 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by pirx » 1 person likes this post

scottf99 wrote: May 25, 2022 5:59 am Hi has anything come of this issue? I have the same problem
Thanks
Scott
Yesterday I received a potential fix for the problem.
This fix prevents prolongation of the immutability for old VM chains, so that eventually the chains can be deleted manually.

I'll installed this week and hope to finally get rid of those 8 month old backups...

desmith
Influencer
Posts: 13
Liked: never
Joined: Dec 15, 2021 11:41 pm
Full Name: Derek Smith
Contact:

Re: Explanation of S3 Immutability + Block Generation needed

Post by desmith »

I've raised a similar issue with support (05431998) and frankly feel they're making false statements to try and convince me that unexpected increase in immutability period is expected behaviour.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests