Explanation of S3 Immutability + Block Generation needed

[pirx]

Hi,

I (or better all colleagues here working with Veeam) have some problems with the Immutability feature. Our goal was to protect the Backup Jobs and Backup Copy Jobs (forever forward inc + syn full) that were transfered to S3 as long as our retention time is (10 weeks). For whatever reason immutabilty was set to only 30 days - this is something we probably have to fix, but we first need to understand how it _really_ works.

What we currently see is that backups/copies have much longer immutability than we have configured. Even for jobs that had their backup on the same day, we see different immutability dates varying from 10.11.2020 - 22.12.2020. Inside this jobs there are VM that have their latest backup from July. But immutability seems not to make a difference when the individual VM's were backed up.

Code: Select all

Jobname generation_expiration_date_utc	 generation_immutable_until_utc 	last_checkpoint_cleanup_date_utc	Last Backup -  Disk (Imported)         Days
#1	     11.10.2020                     	     10.11.2020	                                       21.10.2020	                                       20.09	                                       36
#2	     11.10.2020                                10.11.2020                                         01.10.2020	                                       13.08	                                               63
#3	     11.10.2020	                             10.11.2020	                                       21.10.2020                                       16.09	                                               39
#4          23.10.2020	                              22.11.2020	                                       20.10.2020	                                      10.08	                                               75
#5	     13.10.2020	                             12.11.2020	                                       21.10.2020	                                      14.09                                                43
#6	                                                      11.11.2020	                                                                                              16.09	                                               40
#7 	     22.11.2020	                               22.12.2020	                                       12.11.2020	                                      16.09                                                 70

I found the articles about immutability and block generations.

https://helpcenter.veeam.com/docs/backu ... ml?ver=100
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Maybe I simply do not understand this. But it seems that we never can be sure when data actually can be deleted. We found jobs in Disk (Imported) that were 120 days old and we wanted to delete them, but we got the message that it's still protected for some weeks.

What are we doing wrong? Is our goal to keep the jobs immutable for the time of our retention period and then be able to delete older data no possible because of block generation?

[Gostev]

We found jobs in Disk (Imported) that were 120 days old and we wanted to delete them, but we got the message that it's still protected for some weeks.

This is something you want to investigate with support (if your immutability period was set to 30 days). Restore points should not be protected for much longer than the immutability period you have set in the object storage repository settings.

There's this API costs optimization that extends immutability on each new object for up to 10 days by default, just so we don't have to advance immutability lock on most blocks each day. Because with typical 2% daily change rate, each new restore points consists 98% from existing blocks that were already in object storage. So in the expectation that vast majority of blocks in full backup will be reused in the next 10 restore points, we set their immutability for longer right away - and don't need to come back to them later as the result.

But other than that, immutability period should stick fairly close to your setting. So, don't just assume your understanding is not correct. It is more likely that you're facing a bug in some scenario our QC missed. I checked the User Guide links above and they provide a solid description of how it is supposed to work and I've nothing to add to them really.

[pirx]

Hi Gostev,

thanks for your reply. I am (and have been) in contact with support regarding this (00757218). Maybe you or someone else can help me with this example of a job that is still protected.

I get this error message:

Code: Select all

19.11.2020 08:02:02 Error    [BCJ_BJ-xxxx0400-0449-yyyy] Failed to delete backup Error: Unable to delete the backup because it is marked as immutable until 25 November 2020 02:39:12.

The last backup job was from mid August, so more than 100 days dago. Immutability is set to 30 days on this S3 SOBR bucket. I understand that not the backup time is relevant, but the time of the offload.




From support I got a sql query that seems to return something about Immutability from DB - as there is currently 0 in the GUI and as far as I can see, nothing helpful in powershell, which is bad, because you have no easy way to check what is going on with Immutability or when data/jobs will really expire. Imagine you set different values or changed the Immutability for whatever reason.

Code: Select all

SELECT generation_expiration_date_utc, generation_immutable_until_utc, last_checkpoint_cleanup_date_utc FROM [VEEAMBR].[dbo].[Backup.Model.BackupArchiveIndices] 
where backup_id in (select id from [VEEAMBR].[dbo].[Backup.Model.Backups] where job_name = 'BCJ_BJ-xxxx-0400-0449-yyyy')

generation_expiration_date_utc generation_immutable_until_utc last_checkpoint_cleanup_date_utc
------------------------------ ------------------------------ --------------------------------
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:16             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:18             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:50:20             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:18             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:19             
26.10.2020 01:37:24            25.11.2020 01:37:24            20.10.2020 06:50:29             
16.10.2020 09:28:27            15.11.2020 09:28:27            20.10.2020 06:50:20             
26.10.2020 01:36:33            25.11.2020 01:36:33            20.10.2020 06:50:44             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:51:09             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:50:25             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:22             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:50:23             
16.10.2020 09:28:27            15.11.2020 09:28:27            20.10.2020 06:50:24             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:25             
26.10.2020 00:43:54            25.11.2020 00:43:54            20.10.2020 06:51:07             
26.10.2020 00:43:52            25.11.2020 00:43:52            20.10.2020 06:51:15             
26.10.2020 01:39:10            25.11.2020 01:39:10            20.10.2020 06:51:28             
26.10.2020 01:39:58            25.11.2020 01:39:58            20.10.2020 06:51:35             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:33             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:34             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:30             
16.10.2020 09:28:27            15.11.2020 09:28:27            20.10.2020 06:50:30             
25.10.2020 10:18:15            24.11.2020 10:18:15            20.10.2020 06:51:51             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:35             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:31             
25.10.2020 10:18:17            24.11.2020 10:18:17            20.10.2020 06:50:32             
26.10.2020 01:37:24            25.11.2020 01:37:24            20.10.2020 06:52:04             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:40             
25.10.2020 10:18:16            24.11.2020 10:18:16            20.10.2020 06:50:33             
26.10.2020 01:39:12            25.11.2020 01:39:12            20.10.2020 06:52:12

These dates do not match my expectations. I also do not exactly know what generation_expiration_date_utc, generation_immutable_until_utc, last_checkpoint_cleanup_date_utc are about or what the difference is.

Note: this jobs was disabled for some time and then completely deleted, the data is now under Disk (Imported). Maybe we hit a cornercase here, but still I find it not easy to understand why this data is still protected.

[veremin]

Kindly, keep working with our support team, as neither the described behaviour nor the provided dates looks expected. The only reason I can think of why is that immutability period has been changed down the road - from 100 days or something (old value) to 30 days (current value). Thanks!

[pirx]

We are working with support, but it seems that this topic is not that easy and a bit bleeding edge.

[veremin]

I've passed the information to QA team, we will investigate the reported issue. Thanks!

[veremin]

Can you provide a valid support ticket number? The one mentioned above seems incorrect - 00757218. Also, if you haven't attached database dump to support ticket, kindly, do so. The QA team will take a look at the case within next couple of days. Thanks!

[pirx]

Right, it's ticket 04443517. I'm also working on some other cases with support where we talked about the status of our S3 repository.

[veremin]

Passed to QA team, will share more information, once I have it. Thanks!

[scottf99]

Hi I posted a similar question to this forum. I was asked to create a support ticket and haven't yet but can I ask has anything come out of this ticket/thread as it is exactly what I am seeing.
Cheers
Scott

[pirx]

No, I did not get any feedback yet.

[scottf99]

Is there any more information to share on this? I have old VMs that have not existed for months that are still in Backup Copy and Object Storage as they cannot be deleted. It looks like that if there are still current VMs being backed up in the same job then the immutability is always moved forward for the old VMs as well.
Cheers
Scott

[pirx]

Sounds identical to my problem. Until now, nobody could explain this behavior. We also have to check object storage cleanup now, as we have copy jobs that do not have any data in S3 according to Veeam Console but there are still a lot of TB occupied (>40TB) in the corresponding AWS S3 folder, objects in S3 are from December/January. But I don't see how I can correlate these objects to backup jobs or files.

[scottf99]

How is your support ticket going?

[pirx]

I never got feedback from QA team. I've had a talk with our presales contacts at Veeam, also with support for a couple of times, but they do not yet have an explanation why we have data that is much older than our retention which is still immutable. Nobody could really explain how block generation is working and what blocks are reused. If only blocks in an active chain are reused, it would not explain why we really old vbk files that still are immutable.

[Gostev]

@veremin please provide an update, it's been over 2 months since your last response.

[veremin]

Apologize for any inconvenience caused by my late response.

The provided ticket had been closed about one month before the topic started. So, we could not provide any assistance within the old and long forgotten case.

We're still eager to help, but in order for us to help, may you collect fresh debug logs and attach them to new support ticket? After that kindly post ticket number here and we will make sure that case is given due attention.

Thanks!

[pirx]

I'm having problems with immutability since we started using SOBR + capacity tier. And yet nobody could really explain why backups are immutable much longer than expected. Even with 10 days block generation kept in mind.

Example: last backup of a VM was on 03.01.2022. We have a retention time of 70 days (10 weeks) and 14 days of immutability. How long should the VM be immutable and protected from deletion? If I add 70 days to 03.01 then it would be 14.03. I tried to delete the VM in that week and got an error that it is still immutable (I think it was beginning of April). Now it's mid April and I still can't delete the VM, the message tells me its immutable until 07.05. This is more than 120 days since the last backup and more then 50 days longer than our expected retention. And I do not yet believe that I will be able to delete the VM on that date...

Problem: it's not just one VM, we had to clone jobs due to bugs where offloading was not working any longer and we had to start a new chain for all affected VM's. So this also about storage that is used longer than expected.

But in the end it's about understanding the logic. And what setting we should choose for immutability. First we had set immutability to the same value as retention time. Then support suggested to only use 7 or 14 days for immutability.

[Mildur]

Hi pirx

I found your old topic about the same issue and decided to merge them.
Unfortunately, the last case a year ago was closed.

Can you please collect the debug logs and create a new case?

[pirx]

I'll not open another case. I've had enough discussion about immutability in cases and provided logs in the past 1,5 years. No offense, I just like an explanation on how it should work and what _exactly_ would trigger extension of immutability over and over again (when a VM is not backed up anymore).

This explanation is highly confusing to me https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[Gostev]

what _exactly_ would trigger extension of immutability over and over again (when a VM is not backed up anymore).

For VMs which are not backed up anymore, that would be a bug I think. But at this time we're unaware of such bugs to exist. Also, if a VM is not being backed up, then the process that extends immutability should never be running for this VM in the first place, because this process is a part of the offload job which never runs for the given VM (as there are no new backups to process). So this is very strange in general.

[pirx]

Well, the affected VMs are still backed up. But in a temporary new backup job, as the old one has offload errors that can not be fixed and we needed to start new chains. But I guess this counts as "not backed up anymore".

I only checked backups on disk, now I checked on S3. It's even worse there, there are still backups from September to November. I guess I've to open a new case. Last year we found > 50 TB on S3 that was outdated an never deleted. It took support weeks/month to find a proper way to get rid of this.

[Gostev]

Last year we found > 50 TB on S3 that was outdated an never deleted. It took support weeks/month to find a proper way to get rid of this.

Yes, one year ago there was a bug in rare circumstances that could lead to this. We addressed it in V11 and we even backported this fix to V10 (second to last cumulative patch for 10a).

[veremin]

I guess I've to open a new case.

Sounds like a good idea indeed, the reported behaviour (if persists after product update) does not look expected and needs further investigation. Kindly, share the ticket number, as soon as the case is opened, this way we will have QA team working on it from the very start (instead of digging into rather old situation). Thanks!

[pirx]

I created case 05407008 for this. But the reply so far is a bit... vague. Especially when I am talking about backups from last year or January. I hope that this is not due block generations.

To reduce I/O operations and associated costs, Veeam Backup & Replication will automatically add from 1 to 10 days to the immutability expiration date. This period is called Block Generation. You do not have to configure it, the Block Generation setting is applied automatically.
Based on this, the immutability period can be extended to a period longer than expected.

[veremin]

I think there might be a certain misunderstanding, cause even with the addition of block generation the immutability period (set for these old backups) looks longer than it should be. We have reached your support engineer and asked him to keep the ticket open for now, until QA team verify the debug logs and find the root cause there. Thanks!

[scottf99]

Hi has anything come of this issue? I have the same problem
Thanks
Scott

[veremin]

Kindly reach our support team for further investigation - such type of issue cannot be addressed via forum correspondence, cause we need to check debug logs and see how immutability period was assigned to offloaded restore points. Once the ticket is opened, post its number here and we will follow the investigation internally. Thanks!

[pirx]

Hi has anything come of this issue? I have the same problem
Thanks
Scott

Yesterday I received a potential fix for the problem.

This fix prevents prolongation of the immutability for old VM chains, so that eventually the chains can be deleted manually.

I'll installed this week and hope to finally get rid of those 8 month old backups...

[desmith]

I've raised a similar issue with support (05431998) and frankly feel they're making false statements to try and convince me that unexpected increase in immutability period is expected behaviour.