Case 04765025 - Hardened Repository restore points immutable

[theviking84]

My case seemingly didn't get someone assigned the first couple of days it was open, so then I used the "ask a manager" feature and got a senior engineer assigned. That engineer is under a big workload it seems so they haven't made a lot of progress yet.

I'd like to see if you folks can take a look into my issue to help with determining what is going on.

The short story is I have 2 specific jobs that each night for a week or more now are giving me the following warning. "VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable."

The setup is 2 SOBRs, and each has local linux immutable server as the performance tier, and then aws s3 buckets as the capacity tier. Immutability is configured in all 4 repos within the sobrs. Locally its 7 days, and in the S3 buckets it is set for 4 days (so 4+10 from block generation=14 days)

All of my backup jobs are identical in terms of retention and immutability, and all the rest of the jobs have not yet ever had this warning. So I don't know why these specific 2 jobs are doing this.

Veeam should be aware of what immutability and retention is there, and so it shouldn't be trying to delete something it can't.

There is no easy answer it seems on which vms in the job are even the ones that it is trying to delete? If we could get which those are and what the immutability dates are, and then whether it is trying to delete them from the capacity tier or the performance tier then it would help make sense.

Thank you

[HannesK]

Hello,

All of my backup jobs are identical in terms of retention and immutability,

backup jobs only have retention. immutability is set on repository level. So the main question is, what are the job settings? Forward incremental with weekly synthetic / active full?

Veeam should be aware of what immutability and retention is there, and so it shouldn't be trying to delete something it can't.

The backup job tries to apply its retention. Immutability and and retention are independent. So the warning itself is okay if someone configures contradicting values. But if really all jobs have the same settings, then the message needs to be investigated.

You have 7 days immutable time on the Hardened Repository. As only forward incremental with synthetic / active full works with hardened repository, retention is never applied earlier than 7 days (assuming you have weekly fulls). So that warning needs to be investigated.

My guess is, that the two jobs are configured different. Maybe a changed GFS setting? I put all kind of information together in a whitepaper https://www.veeam.com/wp-guide-protect- ... ckups.html - maybe that also can help you for a better understanding.

Best regards,
Hannes

PS: I edited your thread title

[theviking84]

Job settings are all weekly synthetic full on Saturdays. (default)

These two jobs go to one out of my two sobrs. The performance tier is a physical linux server with immutability set up for 7 days. The capacity tier is aws s3 bucket with 4 days (plus the 10 for block gen).

Simple retention is 21 days. GFS is 6 weeklies, 12 monthlies, 2 yearlies. Copy and move in sobr are enabled. The GFS settings have not been changed and are the same between all jobs.

I have also just now uploaded a BAK of my veeam sql database as the engineer on the ticket has requested this.

[HannesK]

I asked support to add a second engineer on that topic and it looks like there is an issue with the "remove deleted items data after..."

Thanks for uploading the configuration database.

We are still investigating to see how it can be generally fixed.

[theviking84]

Thank you Hannes.

Yes we have now observed if I disable the "deleted items retention" in the job settings then the warning went away on last night's job run.

He said there was at least one other case where the same might be happening. Sounds like they want to let this run until Monday so that we have some more example job runs, and then they will want to increase the logging verbosity to get better details.

So I will be following up with support after that. Hopefully if this is a bug it is something that can be coded around in a patch or whatever. I had the "delete items after" set at 30 days, and thought that was long enough to outweigh immutability in my setup, but maybe it's deeper than how long you set that.

[gummett]

in the S3 buckets it is set for 4 days (so 4+10 from block generation=14 days)

Just to check/clarify, minimum immutability in S3 will be 4 days, maximum will be 14 days (due to 10-day block generation period).

Also on Linux the immutability date of backup files is extended with every new VIB that references them. Therefore with weekly synthetic fulls on Saturday and 7 days immutability, the immutability expiration date of the previous week’s entire chain will be Friday’s date + 7 for a maximum of 13 days.

I’m not suggesting there are any implications for your support case, just wanted to make sure the effect of your settings was clear in case it helps you or others

[theviking84]

So I've had the "deleted items retention" off now for several days and it has still been fine. I want to turn that feature back on though.

So what are the next steps? I haven't had very good response times in support. Keep having to ask about the ticket and have no more guidance yet on what to do to fix this.

[HannesK]

as you probably ran into a bug and developers currently focus on the next cumulative patch, it will take some time.

First thing is to get a confirmation whether it's a bug or not (meaning asking for confirmation of the bug).

Depending on the complexity of the issue, support might offer a hotfix. A hotfix is not guaranteed (especially if there is a workaround) and in general I recommend waiting for a regular update instead of working with hotfixes.

[theviking84]

Ok, thank you. With this particular issue I can wait for a cumulative update if the bug is fixed in that, or will be done quickly after that update is out etc. This is like a main feature that every customer would be trying to use I would think. So once more people get immutability set up it will affect a lot.

I will try asking them to confirm there is a bug or not.

[theviking84]

I'm still waiting for something to move on this.

It seems they are still trying to replicate the issue outside of my environment.

My environment is pretty simple. I created two sobrs.. each sobr is Linux performance tier with immutability and Amazon s3 capacity tier. Do this and turn on the deleted items retention feature and wait for weeks or months to see it happen.

I've now had a third backup job just tonight that is showing this behavior so I had to turn off the deleted items retention feature on that one too.

Can you confirm if this is a bug or not? I've asked that but that question was not answered and I can't believe I am the only one to report this.

[HannesK]

Hello,
yes, unfortunately something went wrong in support. I escalated the case.

Best regards,
Hannes

[theviking84]

Hello Hannes, I am still waiting to hear anything further on this case. I have provided more logs on Friday, just curious if there is still some investigation going on or where we are at with this.

[HannesK]

Hello,
investigations are going on. But nothing to share right now.

Best regards,
Hannes

[HannesK]

Hello,
bug #317093 was created and should hopefully be fixed in 11a. It looks like there were several misunderstandings in the case and support should also answer to you later

Best regards,
Hannes

[theviking84]

I have yet to hear anything. Would like to hear about what misunderstandings there were because my email chains with support seemed pretty clear on what the issue is.

I am glad a bug is confirmed and I await v11a. I am on the most recent available patch so far.

I will try emailing support tomorrow for an update, but if you are able to prompt them to contact me ok your end that would would be appreciated.

[pirx]

I'm also starting to see the "VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable." warning in my V10 environment. No hardened repository, just standard XFS SOBR and offload to S3. Immutability is set to 14 days, retention 14 dailies and 10 weeklies, remove deleted items after 75 days.

I tried to understand what the root cause of the problem above is, but failed... So does this sound like the same issue or is this a different one worth opening a new case?

[HannesK]

it sounds similar, but I would recommend to double check with support. you can refer to the other case / this forum thread with the bug number #317093. that might speed up confirmation of the issue

[gummett]

@pirx It may also be unrelated and instead due to the 10 day block generation period on S3 immutability. If your retention isn’t at least 10 days longer than your immutability setting then at points in the block generation cycle veeam will be attempting to delete data which is immutable

[soncscy]

Just my $.02

Forgive my simplicity and of course wait for Veeam Support to confirm, but if I get it right, this is triggered by Deleted VM retention? So you'll know if you have it by turning off Deleted VM retention and see if the warning continues or not, at least with some fair confidence as I understand it.

[pirx]

@pirx It may also be unrelated and instead due to the 10 day block generation period on S3 immutability. If your retention isn’t at least 10 days longer than your immutability setting then at points in the block generation cycle veeam will be attempting to delete data which is immutable

Retention is 70 days, immutability is 14 days. So that should be enough time. But immutability is always good for surprises and it would not be the first time that something happens nobody can explain (including support).

[pirx]

it sounds similar, but I would recommend to double check with support. you can refer to the other case / this forum thread with the bug number #317093. that might speed up confirmation of the issue

I wanted to open a case the whole morning but Veeam support portal seems to be broken

(This service is temporarily unavailable
We are busy updating the website for you and should be back in a couple of hours.)

[pirx]

Just my $.02

Forgive my simplicity and of course wait for Veeam Support to confirm, but if I get it right, this is triggered by Deleted VM retention? So you'll know if you have it by turning off Deleted VM retention and see if the warning continues or not, at least with some fair confidence as I understand it.

Yes, I'm quiet sure its related to the delete VM setting. But it's not a hardened repository and not v11, I could not really identify from this thread under what circumstances this happens here (if this is already known).

[soncscy]

Well, as I get it though, maybe it's related to the Capacity Tier immutability? But maybe I'm wrong. If you can spare a test incremental run by just turning off the option, I think it would be pretty conclusive, no?

[gummett]

Retention is 70 days, immutability is 14 days.

But your daily backup retention is 14 days, that’s what I was referring to

[pirx]

I don't think the 14 days are relevant with the value for "Remove deleted items data after" set to 75 days. And if this is not related to this option and only related to the simple retention of 14 days wouldn't we see this permanently?

[pirx]

Opened case 04976893

[theviking84]

I haven't looked at this thread in some time, but is it confirmed at this point that the fix for this deleted items retention deletion issue is in 11a?

[HannesK]

bug #317093 from above should be fixed in 11a, yes.

[pirx]

In 11a there are still the entries about skipped VMs, but they are no warning anymore.

Code: Select all

18.10.2021 20:55:38 :: VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable.  
18.10.2021 20:55:38 :: VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable.