Persistent "Storage not found" errors offloading to AWS following v11->v12 upgrade

[markw]

I have had a support case open for this for a number of weeks (#06115032) with no resolution and am wondering if anyone else has encountered this issue.

Our backup server, backing up a file server (~25TB, 500k files) and ~30 VMs, sits behind quite a restrictive firewall. We have a SOBR with local performance tier and AWS capacity tier. On upgrading to v12 it appears new outbound allow rules to AWS were required (still not completely determined but temporarily resolved by giving full outbound access to AWS). At first it took a week for a "backup chain index upgrade" to complete. Following that (20th June) we started getting persistent "storage not found" errors on offloads of the file server, though the VM offloads were successful. We were finally informed by Veeam support at the start of last week that this "seems to look like a known issue that should be fixed in one of the upcoming Veeam versions".

Because of this we have no offsite backups of our file server for nearly three weeks. We are also seeing ~$200 average per day in presumably wasted AWS API call costs (nearly $3K so far) before the SOBR offload jobs fail. We’re loathe to stop these as the VM offloads are working, it’s just the file server that errors.

Though we have escalated this case to a severity 1, given the risks to us of no offsite backups - not to mention the AWS costs that this is incurring - I haven't had any significant update for over a week. Though I've just had a "not heard back from you in a while and wanted to make sure that your request was handled to your satisfaction" email.

So thought I'd post here and see if anyone else has encountered a similar issue with v12. And if so, what has been done about it?

[Gostev]

Hi, Mark. I asked the team to ensure we have the right resources on it. The error would indicate some data is missing from object storage that we expect to be there. Which might have been the case for a while now, with V12 merely highlighting the issue, as in general there are no such known issues with V12. Thanks!

[Gostev]

Hi, Mark. I found that a similar bug has already been fixed in 12a branch. There were no support cases (the bug is actually hard to run into), rather it was found internally in our QC lab. Dev/QA/Support have all been connected now to see if you may have stepped into the same issue. Thanks

[markw]

Hi Gostev,

Thanks for your involvement and for aiding with traction on the case, much appreciated. This sounds a promising lead, keeping my fingers crossed.

I've updated the case with questions around alternative solutions in lieu of a fix. Happy to continue the conversation there for now and you or I can update this post for the benefit of others if there''s any developments

Mark

[markw]

Hi again Gostev,

No real updates, but I would like to enquire what level of support attention we should expect from a severity 1 case? I've been posting questions to the case as I think of things, both to help troubleshoot the current issue and also to figure workarounds/alternative solutions to get us back working again. It's taking on average 24 hours to get a reply to anything, even following your request to ensure the right resources are on it. Is that expected?

We're approaching a month without an offsite backup now, I consider that very severe. I would have expected a more active engagement on our case to help us find any solution that would get our backups offloaded with it being such an extended period.

Thanks

Mark

[Gostev]

Hi, Mark. These are R&D forums which is a completely separate organization from Support at Veeam. Your question seems to be related to Support processes and SLAs, and as such it should be answered by the Support Management. You can contact them directly from the Customer Portal using "Talk to a Manager" functionality. Thank you for understanding!

[markw]

Yes I used that functionality on 27th June, hasn't seemed to make any noticeable difference. I appreciate your point on this being R&D forums, just seemed you possibly had some sway with other Veeam teams as I don't know what else to do to get this moving.

[Gostev]

To be fair to Support, ultimately they are dependent on R&D to investigate an issue and provide a hotfix (in case it's a bug). Which can take a long time depending on the complexity of the issue and on the scope of potential impact of a hotfix (determines the amount of acceptance testing required). And until R&D ships a hotfix, there's not much Support can do irrespective of the issue severity level... except ensuring R&D is working on the fix and treating it with the high priority.

[markw]

Thanks Gostev. I understand that, my questions to them have been on other things we need to look at (e.g. how we might minimise the wasted daily AWS costs) and potential impacts of this if it continues (we are approaching our operational restore windows of 30 days) and other things we may try (e.g. whether doing an active full might help). So lots of other stuff needs answering, we can't just wait for the fix for this in silence without exploring what our other options might be and making decisions on when we would need to implement them if a direct fix hasn't surfaced.

Thanks

Mark

[Gostev]

Sadly, the main problem here is that it is not a common issue. It it was, there would be a recipe for remediation already... but when you're the first, it takes time to get to the bottom of the issue, before such a recipe can even be created.