-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Case 04765025 - Hardened Repository restore points immutable
My case seemingly didn't get someone assigned the first couple of days it was open, so then I used the "ask a manager" feature and got a senior engineer assigned. That engineer is under a big workload it seems so they haven't made a lot of progress yet.
I'd like to see if you folks can take a look into my issue to help with determining what is going on.
The short story is I have 2 specific jobs that each night for a week or more now are giving me the following warning. "VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable."
The setup is 2 SOBRs, and each has local linux immutable server as the performance tier, and then aws s3 buckets as the capacity tier. Immutability is configured in all 4 repos within the sobrs. Locally its 7 days, and in the S3 buckets it is set for 4 days (so 4+10 from block generation=14 days)
All of my backup jobs are identical in terms of retention and immutability, and all the rest of the jobs have not yet ever had this warning. So I don't know why these specific 2 jobs are doing this.
Veeam should be aware of what immutability and retention is there, and so it shouldn't be trying to delete something it can't.
There is no easy answer it seems on which vms in the job are even the ones that it is trying to delete? If we could get which those are and what the immutability dates are, and then whether it is trying to delete them from the capacity tier or the performance tier then it would help make sense.
Thank you
I'd like to see if you folks can take a look into my issue to help with determining what is going on.
The short story is I have 2 specific jobs that each night for a week or more now are giving me the following warning. "VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable."
The setup is 2 SOBRs, and each has local linux immutable server as the performance tier, and then aws s3 buckets as the capacity tier. Immutability is configured in all 4 repos within the sobrs. Locally its 7 days, and in the S3 buckets it is set for 4 days (so 4+10 from block generation=14 days)
All of my backup jobs are identical in terms of retention and immutability, and all the rest of the jobs have not yet ever had this warning. So I don't know why these specific 2 jobs are doing this.
Veeam should be aware of what immutability and retention is there, and so it shouldn't be trying to delete something it can't.
There is no easy answer it seems on which vms in the job are even the ones that it is trying to delete? If we could get which those are and what the immutability dates are, and then whether it is trying to delete them from the capacity tier or the performance tier then it would help make sense.
Thank you
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025
Hello,
You have 7 days immutable time on the Hardened Repository. As only forward incremental with synthetic / active full works with hardened repository, retention is never applied earlier than 7 days (assuming you have weekly fulls). So that warning needs to be investigated.
My guess is, that the two jobs are configured different. Maybe a changed GFS setting? I put all kind of information together in a whitepaper https://www.veeam.com/wp-guide-protect- ... ckups.html - maybe that also can help you for a better understanding.
Best regards,
Hannes
PS: I edited your thread title
backup jobs only have retention. immutability is set on repository level. So the main question is, what are the job settings? Forward incremental with weekly synthetic / active full?All of my backup jobs are identical in terms of retention and immutability,
The backup job tries to apply its retention. Immutability and and retention are independent. So the warning itself is okay if someone configures contradicting values. But if really all jobs have the same settings, then the message needs to be investigated.Veeam should be aware of what immutability and retention is there, and so it shouldn't be trying to delete something it can't.
You have 7 days immutable time on the Hardened Repository. As only forward incremental with synthetic / active full works with hardened repository, retention is never applied earlier than 7 days (assuming you have weekly fulls). So that warning needs to be investigated.
My guess is, that the two jobs are configured different. Maybe a changed GFS setting? I put all kind of information together in a whitepaper https://www.veeam.com/wp-guide-protect- ... ckups.html - maybe that also can help you for a better understanding.
Best regards,
Hannes
PS: I edited your thread title
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Job settings are all weekly synthetic full on Saturdays. (default)
These two jobs go to one out of my two sobrs. The performance tier is a physical linux server with immutability set up for 7 days. The capacity tier is aws s3 bucket with 4 days (plus the 10 for block gen).
Simple retention is 21 days. GFS is 6 weeklies, 12 monthlies, 2 yearlies. Copy and move in sobr are enabled. The GFS settings have not been changed and are the same between all jobs.
I have also just now uploaded a BAK of my veeam sql database as the engineer on the ticket has requested this.
These two jobs go to one out of my two sobrs. The performance tier is a physical linux server with immutability set up for 7 days. The capacity tier is aws s3 bucket with 4 days (plus the 10 for block gen).
Simple retention is 21 days. GFS is 6 weeklies, 12 monthlies, 2 yearlies. Copy and move in sobr are enabled. The GFS settings have not been changed and are the same between all jobs.
I have also just now uploaded a BAK of my veeam sql database as the engineer on the ticket has requested this.
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I asked support to add a second engineer on that topic and it looks like there is an issue with the "remove deleted items data after..."
Thanks for uploading the configuration database.
We are still investigating to see how it can be generally fixed.
Thanks for uploading the configuration database.
We are still investigating to see how it can be generally fixed.
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Thank you Hannes.
Yes we have now observed if I disable the "deleted items retention" in the job settings then the warning went away on last night's job run.
He said there was at least one other case where the same might be happening. Sounds like they want to let this run until Monday so that we have some more example job runs, and then they will want to increase the logging verbosity to get better details.
So I will be following up with support after that. Hopefully if this is a bug it is something that can be coded around in a patch or whatever. I had the "delete items after" set at 30 days, and thought that was long enough to outweigh immutability in my setup, but maybe it's deeper than how long you set that.
Yes we have now observed if I disable the "deleted items retention" in the job settings then the warning went away on last night's job run.
He said there was at least one other case where the same might be happening. Sounds like they want to let this run until Monday so that we have some more example job runs, and then they will want to increase the logging verbosity to get better details.
So I will be following up with support after that. Hopefully if this is a bug it is something that can be coded around in a patch or whatever. I had the "delete items after" set at 30 days, and thought that was long enough to outweigh immutability in my setup, but maybe it's deeper than how long you set that.
-
- Veteran
- Posts: 405
- Liked: 106 times
- Joined: Jan 30, 2017 9:23 am
- Full Name: Ed Gummett
- Location: Manchester, United Kingdom
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Just to check/clarify, minimum immutability in S3 will be 4 days, maximum will be 14 days (due to 10-day block generation period).theviking84 wrote: ↑Apr 26, 2021 10:45 pm in the S3 buckets it is set for 4 days (so 4+10 from block generation=14 days)
Also on Linux the immutability date of backup files is extended with every new VIB that references them. Therefore with weekly synthetic fulls on Saturday and 7 days immutability, the immutability expiration date of the previous week’s entire chain will be Friday’s date + 7 for a maximum of 13 days.
I’m not suggesting there are any implications for your support case, just wanted to make sure the effect of your settings was clear in case it helps you or others
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
So I've had the "deleted items retention" off now for several days and it has still been fine. I want to turn that feature back on though.
So what are the next steps? I haven't had very good response times in support. Keep having to ask about the ticket and have no more guidance yet on what to do to fix this.
So what are the next steps? I haven't had very good response times in support. Keep having to ask about the ticket and have no more guidance yet on what to do to fix this.
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
as you probably ran into a bug and developers currently focus on the next cumulative patch, it will take some time.
First thing is to get a confirmation whether it's a bug or not (meaning asking for confirmation of the bug).
Depending on the complexity of the issue, support might offer a hotfix. A hotfix is not guaranteed (especially if there is a workaround) and in general I recommend waiting for a regular update instead of working with hotfixes.
First thing is to get a confirmation whether it's a bug or not (meaning asking for confirmation of the bug).
Depending on the complexity of the issue, support might offer a hotfix. A hotfix is not guaranteed (especially if there is a workaround) and in general I recommend waiting for a regular update instead of working with hotfixes.
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Ok, thank you. With this particular issue I can wait for a cumulative update if the bug is fixed in that, or will be done quickly after that update is out etc. This is like a main feature that every customer would be trying to use I would think. So once more people get immutability set up it will affect a lot.
I will try asking them to confirm there is a bug or not.
I will try asking them to confirm there is a bug or not.
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I'm still waiting for something to move on this.
It seems they are still trying to replicate the issue outside of my environment.
My environment is pretty simple. I created two sobrs.. each sobr is Linux performance tier with immutability and Amazon s3 capacity tier. Do this and turn on the deleted items retention feature and wait for weeks or months to see it happen.
I've now had a third backup job just tonight that is showing this behavior so I had to turn off the deleted items retention feature on that one too.
Can you confirm if this is a bug or not? I've asked that but that question was not answered and I can't believe I am the only one to report this.
It seems they are still trying to replicate the issue outside of my environment.
My environment is pretty simple. I created two sobrs.. each sobr is Linux performance tier with immutability and Amazon s3 capacity tier. Do this and turn on the deleted items retention feature and wait for weeks or months to see it happen.
I've now had a third backup job just tonight that is showing this behavior so I had to turn off the deleted items retention feature on that one too.
Can you confirm if this is a bug or not? I've asked that but that question was not answered and I can't believe I am the only one to report this.
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Hello,
yes, unfortunately something went wrong in support. I escalated the case.
Best regards,
Hannes
yes, unfortunately something went wrong in support. I escalated the case.
Best regards,
Hannes
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Hello Hannes, I am still waiting to hear anything further on this case. I have provided more logs on Friday, just curious if there is still some investigation going on or where we are at with this.
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Hello,
investigations are going on. But nothing to share right now.
Best regards,
Hannes
investigations are going on. But nothing to share right now.
Best regards,
Hannes
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Hello,
bug #317093 was created and should hopefully be fixed in 11a. It looks like there were several misunderstandings in the case and support should also answer to you later
Best regards,
Hannes
bug #317093 was created and should hopefully be fixed in 11a. It looks like there were several misunderstandings in the case and support should also answer to you later
Best regards,
Hannes
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I have yet to hear anything. Would like to hear about what misunderstandings there were because my email chains with support seemed pretty clear on what the issue is.
I am glad a bug is confirmed and I await v11a. I am on the most recent available patch so far.
I will try emailing support tomorrow for an update, but if you are able to prompt them to contact me ok your end that would would be appreciated.
I am glad a bug is confirmed and I await v11a. I am on the most recent available patch so far.
I will try emailing support tomorrow for an update, but if you are able to prompt them to contact me ok your end that would would be appreciated.
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I'm also starting to see the "VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable." warning in my V10 environment. No hardened repository, just standard XFS SOBR and offload to S3. Immutability is set to 14 days, retention 14 dailies and 10 weeklies, remove deleted items after 75 days.
I tried to understand what the root cause of the problem above is, but failed... So does this sound like the same issue or is this a different one worth opening a new case?
I tried to understand what the root cause of the problem above is, but failed... So does this sound like the same issue or is this a different one worth opening a new case?
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
it sounds similar, but I would recommend to double check with support. you can refer to the other case / this forum thread with the bug number #317093. that might speed up confirmation of the issue
-
- Veteran
- Posts: 405
- Liked: 106 times
- Joined: Jan 30, 2017 9:23 am
- Full Name: Ed Gummett
- Location: Manchester, United Kingdom
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
@pirx It may also be unrelated and instead due to the 10 day block generation period on S3 immutability. If your retention isn’t at least 10 days longer than your immutability setting then at points in the block generation cycle veeam will be attempting to delete data which is immutable
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
-
- Veteran
- Posts: 643
- Liked: 312 times
- Joined: Aug 04, 2019 2:57 pm
- Full Name: Harvey
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Just my $.02
Forgive my simplicity and of course wait for Veeam Support to confirm, but if I get it right, this is triggered by Deleted VM retention? So you'll know if you have it by turning off Deleted VM retention and see if the warning continues or not, at least with some fair confidence as I understand it.
Forgive my simplicity and of course wait for Veeam Support to confirm, but if I get it right, this is triggered by Deleted VM retention? So you'll know if you have it by turning off Deleted VM retention and see if the warning continues or not, at least with some fair confidence as I understand it.
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Retention is 70 days, immutability is 14 days. So that should be enough time. But immutability is always good for surprises and it would not be the first time that something happens nobody can explain (including support).gummett wrote: ↑Aug 19, 2021 10:29 am @pirx It may also be unrelated and instead due to the 10 day block generation period on S3 immutability. If your retention isn’t at least 10 days longer than your immutability setting then at points in the block generation cycle veeam will be attempting to delete data which is immutable
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I wanted to open a case the whole morning but Veeam support portal seems to be broken
(This service is temporarily unavailable
We are busy updating the website for you and should be back in a couple of hours.)
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Yes, I'm quiet sure its related to the delete VM setting. But it's not a hardened repository and not v11, I could not really identify from this thread under what circumstances this happens here (if this is already known).soncscy wrote: ↑Aug 19, 2021 11:31 am Just my $.02
Forgive my simplicity and of course wait for Veeam Support to confirm, but if I get it right, this is triggered by Deleted VM retention? So you'll know if you have it by turning off Deleted VM retention and see if the warning continues or not, at least with some fair confidence as I understand it.
-
- Veteran
- Posts: 643
- Liked: 312 times
- Joined: Aug 04, 2019 2:57 pm
- Full Name: Harvey
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Well, as I get it though, maybe it's related to the Capacity Tier immutability? But maybe I'm wrong. If you can spare a test incremental run by just turning off the option, I think it would be pretty conclusive, no?
-
- Veteran
- Posts: 405
- Liked: 106 times
- Joined: Jan 30, 2017 9:23 am
- Full Name: Ed Gummett
- Location: Manchester, United Kingdom
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
But your daily backup retention is 14 days, that’s what I was referring to
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I don't think the 14 days are relevant with the value for "Remove deleted items data after" set to 75 days. And if this is not related to this option and only related to the simple retention of 14 days wouldn't we see this permanently?
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
Opened case 04976893
-
- Expert
- Posts: 119
- Liked: 11 times
- Joined: Nov 16, 2020 2:58 pm
- Full Name: David Dunworthy
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
I haven't looked at this thread in some time, but is it confirmed at this point that the fix for this deleted items retention deletion issue is in 11a?
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
bug #317093 from above should be fixed in 11a, yes.
-
- Veteran
- Posts: 599
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Case 04765025 - Hardened Repository restore points immutable
In 11a there are still the entries about skipped VMs, but they are no warning anymore.
Code: Select all
18.10.2021 20:55:38 :: VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable.
18.10.2021 20:55:38 :: VMs skipped. Unable to delete the selected instance from backup: some of the restored points are immutable.
Who is online
Users browsing this forum: Semrush [Bot] and 62 guests