Invalid repositories appearing

[pr_osit]

Hey all,

Another issue has appeared in version 8 for us the last few days, was working well for almost 5 days on v8.0.2.501 without any major issues then this new problem started out of nowhere. First a few jobs failed with invalid repository, then a few more and now I have 20 invalid repositories. A few more seem to become invalid each day.

The issue is 'Repository is in an invalid state. The local cache and object storage are not synchronized.'

I knew how to fix this in v7 but no idea in v8 since the local cache has moved. This is causing the 20 jobs which use those repos to fail constantly now. The sync repo button is greyed out in v8 and there's no local cache folder on the proxy to clear anymore.

A case has been logged with support Case #07440695 and trying to provide them the info they need to help troubleshoot.

Any extra eyes I can get on this issue would be appreciated, thank you.

Kind regards,
Pat

[Polina]

Hi Pat,

In v8, repository caches still exist and synchronization is still required; but instead of a separate database per repository, caches are now stored in a centralized database.

Thank you for the heads up; I'll bring your case to the attention of our RnD team to fix it faster. Please keep working with our support engineers.

Thanks!

[aeph]

"to fix it faster" - what does this mean? We constantly report issues around repo sync stuck and invalid state since two weeks and the message is still the same "RnD is working on it". So it should not be a recent discovery.
We backup a lot users - we urgently need a solution.

Thanks!

[pr_osit]

I received a fix from support, basically had to delete and re-add all of the invalid repos. This is fairly time consuming but has fixed my immediate issue. I do still have some concerns about what would cause 20 repos to become invalid within about a 24 hour period, I'll be looking further into root cause analysis try and identify the issue as so far there's no clear cause of the problem identified.

[aeph]

With fix you mean a private fix with a specific build version number? May i ask which build number this is?
We also got a hotfix but it was not solving the issue.

[pr_osit]

Hi aeph, I didn't get any specific private fix for this issue, the fix was just removing and re-adding all of the affected repos, but I've received fixes for other issues, currently on 8.0.2.501 for about the last 2 weeks, and still having new issues popping up almost daily.

[aeph]

Same for us. But we have 8.0.2.311.

We added the repos again several times, but that didn't solve the problem either. We still have repos that we can't add - for more than two weeks.

[edh]

@aeph 8.0.2.311 is a private fix/version?

[Mildur]

It‘s shared as a private fix. Public releases are listed here:
https://www.veeam.com/kb4106

We plan to release a public cumulative patch soon with all the fixes.

Best,
Fabian

[pr_osit]

To update this case again, I have looked at my jobs today and there's approx 60-80 jobs which all failed with the following error:
6/10/2024 4:13:16 PM :: Nothing to process
6/10/2024 4:13:16 PM :: Repository is in an invalid state. The local cache and object storage are not synchronized.

Veeam, we really need a fix for these constant issues, every day since upgrading to version 8 there has been problems, if I could go back to version 7 I would do it right now.
This update just doesn't seem like it's production ready.

edit: the number of failing jobs with the same error keeps increasing, it's now over 100. I suspect almost all jobs are now experiencing this issue.
I'm not doing the same fix as last time which was move job to temp repo, delete old repo, stop all services, delete lock file from object storage, start services, re-add repo to console, re-add job to repo and let job do a full backup. there's a couple of jobs that haven't even finished a full since the last time this happened. we need a better solution.

My support case has been updated with the new instance of this same issue.

[aeph]

Puh.. one hour ago i had exactly the same issue!

A restart of all my proxies resolved the issue. But it was more of a trial and error. It's unbelievable how unstable the version is at the moment. We currently have to intervene almost every day! Very sad.

And we are having a new issue: A job is failing due to this problem:

“06.10.2024 13:26:34 :: Exception of type ‘System.OutOfMemoryException’ was thrown.”

But it seems that just one proxy is doing working. It seems that the proxy pooling isn't working as expected.

[Mildur]

Hi @aeph
Do you have a case number for me?

Hi @pr_osit
The case was escalated today to the next tier. We will keep an eye on the further investigation.

Best,
Fabian

[pr_osit]

Hi Fabian,

I have checked my jobs again today and now have even more new errors, it's starting to become too many errors to open support cases for them and keep track of it all. Some resolve themselves in time, others need intervention. I'm still waiting 2 days since updating my last case, I saw it has been escalated but nobody got back to me, I was able to fix the majority of the invalid repos on my own but I'm still experiencing major issues, very slow jobs, time out errors, poisoned repo, invalid repo, plus a lot of new interesting errors such as the below.

Code: Select all

6/10/2024 8:29:26 PM :: Processing archive mailbox  failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document (1, 15644466).. The token '"' was expected but found '<'. Line 1, position 15644466. :: 0:15:17
6/10/2024 9:26:45 PM :: Processing archive mailbox failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document (1, 74024638).. The token '"' was expected but found '<'. Line 1, position 74024638. :: 0:13:30
6/10/2024 6:45:50 PM :: Processing mailbox Discovery Search Mailbox failed with error: Unable to cast object of type 'System.Collections.Generic.List`1[System.String]' to type 'System.Management.Automation.PSObject'.
6/10/2024 1:45:14 PM :: Processing mailbox failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document (1, 10016595).. No characters can appear before the XML declaration. Line 1, position 10016595. :: 0:52:27
6/10/2024 4:33:16 PM :: Repository is in an invalid state. The local cache and object storage are not synchronized.
6/10/2024 9:02:54 PM :: Failed to process team:. The response status code does not indicate success: 404 (Not Found). {"error":{"code":"NotFound","message":"NotFound","innerError":{"code":"1","message":"ThreadNotFound-Thread was not found.","date":"2024-10-06T13:03:02","request-id":"3758af4c-2407-4ede-8fba-f33dbc1e28b1","client-request-id":"6a620f05-9072-47c9-845b-f1054e41fb76"}}} :: 0:00:08
7/10/2024 4:35:36 PM :: Processing mailbox failed with error: Mail item data export failed. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host... An existing connection was forcibly closed by the remote host. :: 0:23:42
7/10/2024 4:35:36 PM :: Processing mailbox failed with error: Mail item data export failed. The operation has timed out. :: 0:21:59
6/10/2024 1:06:16 PM :: Failed to process site:  The response status code does not indicate success: 500 (Internal Server Error). <?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Exception of type 'Microsoft.SharePoint.SoapServer.SoapServerException' was thrown.</faultstring><detail><errorstring xmlns="http://schemas.microsoft.com/sharepoint/soap/">Access to this Web site has been blocked. Please contact the administrator to resolve this problem.</errorstring><errorcode xmlns="http://schemas.microsoft.com/sharepoint/soap/">0x81020071</errorcode></detail></soap:Fault></soap:Body></soap:Envelope>
6/10/2024 7:57:45 PM :: Processing site  finished with warning: Failed to backup item: /sites/xxx/redacted.file, Failed to download block in time: 00:29:59.9972011. :: 0:30:38
7/10/2024 5:57:22 PM :: Processing OneDrive  finished with warning: Failed to backup item version: /_vti_history/xxx/redacted.file, version: 1.0, Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. :: 1:51:16
7/10/2024 11:59:09 AM :: Processing mailbox  failed with error: An error occurred while saving the entity changes. See the inner exception for details. :: 0:00:08
3/10/2024 6:33:31 PM :: Failed to process OneDrive:  Request must be retried. :: 34:46:08

I could go on.. this is just recent errors at a quick glance of my console. These all repeat many times throughout my jobs.
Opening support cases and trying to deal with these one by one isn't working for me, I'm wasting entire days at work on this and falling behind in other areas, we have had more issues and problems and spent more time in 3 weeks since upgrading to version 8 than in the last year of version 7 where things worked just fine for the most part.

How do I communicate the severity of these issues to someone senior, to do a remote session or something with me and look at these problems from a big picture perspective, not individual issues as part of a support case, and really see how bad this is for us trying to use the product in production day to day and hopefully help me with some solutions? I know this has been a big update with big changes, and I know support and R&D must be very busy, but it's been going on for weeks now. I'm just one person trying to manage this and do the rest of my job and I just can't keep up with the errors, I don't know what to tell customers, I can't deliver backups as promised, I don't even know if restored have issues too, lucky I haven't had to try yet.

[Mildur]

Hello Pat

I understand that this is a challenging situation for both our service provider and end-customers. Our R&D teams are committed to delivering fixes as quickly as possible for every issue reported to us.
From the case logs I see that your case is now being handled by our Advanced Level support team.

How do I communicate the severity of these issues to someone senior, to do a remote session or something with me and look at these problems from a big picture perspective

The official process would be to use our Talk To Manager option (at least for a single case) and demand a remote session. You may also provide all case numbers you have open in the escalation. But It may be possible that your support engineer will only be able to check his assigned case. Another option would be to contact your local SE team to have a discussion about open issues. Unfortunately I don't expect that they can help much at this stage. Most of the issues you are facing should be known issues by know where we have or are finishing fixes right now. Our first cumulative patch for v8 is also planned to be released really soon (the patch is currently in the final phase of testing).

I suggest we update to the first cumulative patch once it is released. After applying the patch, we can verify with the help of customer support which issues still persist.

Best,
Fabian

[Mildur]

Hi guys

We have released our new patch (P20241008 - 8.0.3.1044), which should resolve many of the issues discussed in this topic. If you have an active support case open, I recommend first consulting with your support engineer before applying the patch.

Veeam Backup for Microsoft 365 v8 - P20241008 (8.0.3.1044)
https://www.veeam.com/kb4656

Best,
Fabian

[pr_osit]

Thanks for the info, I've reached out to my account manager and SE and will be working with them to resolve remaining issues. I've installed the 8.0.3.1044 patch and so far it does seem to be improved, I've had no new issues for a few days, majority of jobs are completing. I still have a few smaller issues to review but this looks to be a good patch so far.