Invalid repositories appearing

[pr_osit]

Hey all,

Another issue has appeared in version 8 for us the last few days, was working well for almost 5 days on v8.0.2.501 without any major issues then this new problem started out of nowhere. First a few jobs failed with invalid repository, then a few more and now I have 20 invalid repositories. A few more seem to become invalid each day.

The issue is 'Repository is in an invalid state. The local cache and object storage are not synchronized.'

I knew how to fix this in v7 but no idea in v8 since the local cache has moved. This is causing the 20 jobs which use those repos to fail constantly now. The sync repo button is greyed out in v8 and there's no local cache folder on the proxy to clear anymore.

A case has been logged with support Case #07440695 and trying to provide them the info they need to help troubleshoot.

Any extra eyes I can get on this issue would be appreciated, thank you.

Kind regards,
Pat

[Polina]

Hi Pat,

In v8, repository caches still exist and synchronization is still required; but instead of a separate database per repository, caches are now stored in a centralized database.

Thank you for the heads up; I'll bring your case to the attention of our RnD team to fix it faster. Please keep working with our support engineers.

Thanks!

[aeph]

"to fix it faster" - what does this mean? We constantly report issues around repo sync stuck and invalid state since two weeks and the message is still the same "RnD is working on it". So it should not be a recent discovery.
We backup a lot users - we urgently need a solution.

Thanks!

[pr_osit]

I received a fix from support, basically had to delete and re-add all of the invalid repos. This is fairly time consuming but has fixed my immediate issue. I do still have some concerns about what would cause 20 repos to become invalid within about a 24 hour period, I'll be looking further into root cause analysis try and identify the issue as so far there's no clear cause of the problem identified.

[aeph]

With fix you mean a private fix with a specific build version number? May i ask which build number this is?
We also got a hotfix but it was not solving the issue.

[pr_osit]

Hi aeph, I didn't get any specific private fix for this issue, the fix was just removing and re-adding all of the affected repos, but I've received fixes for other issues, currently on 8.0.2.501 for about the last 2 weeks, and still having new issues popping up almost daily.

[aeph]

Same for us. But we have 8.0.2.311.

We added the repos again several times, but that didn't solve the problem either. We still have repos that we can't add - for more than two weeks.

[edh]

@aeph 8.0.2.311 is a private fix/version?

[Mildur]

It‘s shared as a private fix. Public releases are listed here:
https://www.veeam.com/kb4106

We plan to release a public cumulative patch soon with all the fixes.

Best,
Fabian

[pr_osit]

To update this case again, I have looked at my jobs today and there's approx 60-80 jobs which all failed with the following error:
6/10/2024 4:13:16 PM :: Nothing to process
6/10/2024 4:13:16 PM :: Repository is in an invalid state. The local cache and object storage are not synchronized.

Veeam, we really need a fix for these constant issues, every day since upgrading to version 8 there has been problems, if I could go back to version 7 I would do it right now.
This update just doesn't seem like it's production ready.

edit: the number of failing jobs with the same error keeps increasing, it's now over 100. I suspect almost all jobs are now experiencing this issue.
I'm not doing the same fix as last time which was move job to temp repo, delete old repo, stop all services, delete lock file from object storage, start services, re-add repo to console, re-add job to repo and let job do a full backup. there's a couple of jobs that haven't even finished a full since the last time this happened. we need a better solution.

My support case has been updated with the new instance of this same issue.

[aeph]

Puh.. one hour ago i had exactly the same issue!

A restart of all my proxies resolved the issue. But it was more of a trial and error. It's unbelievable how unstable the version is at the moment. We currently have to intervene almost every day! Very sad.

And we are having a new issue: A job is failing due to this problem:

“06.10.2024 13:26:34 :: Exception of type ‘System.OutOfMemoryException’ was thrown.”

But it seems that just one proxy is doing working. It seems that the proxy pooling isn't working as expected.

[Mildur]

Hi @aeph
Do you have a case number for me?

Hi @pr_osit
The case was escalated today to the next tier. We will keep an eye on the further investigation.

Best,
Fabian

[pr_osit]

Hi Fabian,

I have checked my jobs again today and now have even more new errors, it's starting to become too many errors to open support cases for them and keep track of it all. Some resolve themselves in time, others need intervention. I'm still waiting 2 days since updating my last case, I saw it has been escalated but nobody got back to me, I was able to fix the majority of the invalid repos on my own but I'm still experiencing major issues, very slow jobs, time out errors, poisoned repo, invalid repo, plus a lot of new interesting errors such as the below.

Code: Select all

6/10/2024 8:29:26 PM :: Processing archive mailbox  failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document (1, 15644466).. The token '"' was expected but found '<'. Line 1, position 15644466. :: 0:15:17
6/10/2024 9:26:45 PM :: Processing archive mailbox failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document (1, 74024638).. The token '"' was expected but found '<'. Line 1, position 74024638. :: 0:13:30
6/10/2024 6:45:50 PM :: Processing mailbox Discovery Search Mailbox failed with error: Unable to cast object of type 'System.Collections.Generic.List`1[System.String]' to type 'System.Management.Automation.PSObject'.
6/10/2024 1:45:14 PM :: Processing mailbox failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document (1, 10016595).. No characters can appear before the XML declaration. Line 1, position 10016595. :: 0:52:27
6/10/2024 4:33:16 PM :: Repository is in an invalid state. The local cache and object storage are not synchronized.
6/10/2024 9:02:54 PM :: Failed to process team:. The response status code does not indicate success: 404 (Not Found). {"error":{"code":"NotFound","message":"NotFound","innerError":{"code":"1","message":"ThreadNotFound-Thread was not found.","date":"2024-10-06T13:03:02","request-id":"3758af4c-2407-4ede-8fba-f33dbc1e28b1","client-request-id":"6a620f05-9072-47c9-845b-f1054e41fb76"}}} :: 0:00:08
7/10/2024 4:35:36 PM :: Processing mailbox failed with error: Mail item data export failed. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host... An existing connection was forcibly closed by the remote host. :: 0:23:42
7/10/2024 4:35:36 PM :: Processing mailbox failed with error: Mail item data export failed. The operation has timed out. :: 0:21:59
6/10/2024 1:06:16 PM :: Failed to process site:  The response status code does not indicate success: 500 (Internal Server Error). <?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Exception of type 'Microsoft.SharePoint.SoapServer.SoapServerException' was thrown.</faultstring><detail><errorstring xmlns="http://schemas.microsoft.com/sharepoint/soap/">Access to this Web site has been blocked. Please contact the administrator to resolve this problem.</errorstring><errorcode xmlns="http://schemas.microsoft.com/sharepoint/soap/">0x81020071</errorcode></detail></soap:Fault></soap:Body></soap:Envelope>
6/10/2024 7:57:45 PM :: Processing site  finished with warning: Failed to backup item: /sites/xxx/redacted.file, Failed to download block in time: 00:29:59.9972011. :: 0:30:38
7/10/2024 5:57:22 PM :: Processing OneDrive  finished with warning: Failed to backup item version: /_vti_history/xxx/redacted.file, version: 1.0, Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. :: 1:51:16
7/10/2024 11:59:09 AM :: Processing mailbox  failed with error: An error occurred while saving the entity changes. See the inner exception for details. :: 0:00:08
3/10/2024 6:33:31 PM :: Failed to process OneDrive:  Request must be retried. :: 34:46:08

I could go on.. this is just recent errors at a quick glance of my console. These all repeat many times throughout my jobs.
Opening support cases and trying to deal with these one by one isn't working for me, I'm wasting entire days at work on this and falling behind in other areas, we have had more issues and problems and spent more time in 3 weeks since upgrading to version 8 than in the last year of version 7 where things worked just fine for the most part.

How do I communicate the severity of these issues to someone senior, to do a remote session or something with me and look at these problems from a big picture perspective, not individual issues as part of a support case, and really see how bad this is for us trying to use the product in production day to day and hopefully help me with some solutions? I know this has been a big update with big changes, and I know support and R&D must be very busy, but it's been going on for weeks now. I'm just one person trying to manage this and do the rest of my job and I just can't keep up with the errors, I don't know what to tell customers, I can't deliver backups as promised, I don't even know if restored have issues too, lucky I haven't had to try yet.

[Mildur]

Hello Pat

I understand that this is a challenging situation for both our service provider and end-customers. Our R&D teams are committed to delivering fixes as quickly as possible for every issue reported to us.
From the case logs I see that your case is now being handled by our Advanced Level support team.

How do I communicate the severity of these issues to someone senior, to do a remote session or something with me and look at these problems from a big picture perspective

The official process would be to use our Talk To Manager option (at least for a single case) and demand a remote session. You may also provide all case numbers you have open in the escalation. But It may be possible that your support engineer will only be able to check his assigned case. Another option would be to contact your local SE team to have a discussion about open issues. Unfortunately I don't expect that they can help much at this stage. Most of the issues you are facing should be known issues by know where we have or are finishing fixes right now. Our first cumulative patch for v8 is also planned to be released really soon (the patch is currently in the final phase of testing).

I suggest we update to the first cumulative patch once it is released. After applying the patch, we can verify with the help of customer support which issues still persist.

Best,
Fabian

[Mildur]

Hi guys

We have released our new patch (P20241008 - 8.0.3.1044), which should resolve many of the issues discussed in this topic. If you have an active support case open, I recommend first consulting with your support engineer before applying the patch.

Veeam Backup for Microsoft 365 v8 - P20241008 (8.0.3.1044)
https://www.veeam.com/kb4656

Best,
Fabian

[pr_osit]

Thanks for the info, I've reached out to my account manager and SE and will be working with them to resolve remaining issues. I've installed the 8.0.3.1044 patch and so far it does seem to be improved, I've had no new issues for a few days, majority of jobs are completing. I still have a few smaller issues to review but this looks to be a good patch so far.

[aeph]

Since a few days now we have been having massive problems again with repos that fall into the “Invalid state”.

We use Wasabi. Does anyone else have these problems or how do you deal with them?
We have 8.0.4.29 installed.

[pr_osit]

@aeph

I am still getting about 3-5 invalid repos a week, going to log a new support case for it. I have a fix i've been doing to resolve the issue when it occurs and I can't be bothered dealing with support but getting sick of it and need it to be fixed by R&D so going to work with support on it.

The fix is stop all veeam services
Stop nats service
Browse to C:\ProgramData\Veeam\Backup365\nats
Delete/move the jetstream folder out of this location
Restart all of your proxy servers if you have multiple
Restart your management server
Veeam services will restart
Jetstream folder will be recreated
Invalid repos should resync and begin working again

I'm on 8.0.5.20 and it's not fixed in the newest release either, there's been a handful of minor bug fixes but there are still a lot of bigger problems that need to be resolved.

[aeph]

We do exactly the same.

Stoping the faulty job is not working in our case as well. The Job changes to "Queued" state but the job is not stopped.

Is the repo resyncing after your procedure (removing jetstream folder and restarting the services) or do you need to remove and readd the repo?
Yesterday we removed and readded a repo but this results in a Full-Sync and that takes an extremely long time.

@Mildur Do you have more info regarding this? That is definitely a big problem for us.

[pr_osit]

@aeph
Advise from support is usually to remove and re-add the repo, I don't like this as it takes significant time to do for multiple repos, and it does result in a "full" which is very time consuming and high load.
If you do the above procedure (I've done it about 12 times so far) when the services start, the repos should sync automatically within a few mins and fix themselves.
I've discussed this procedure with support before and they had no issues with it as it works, but do at your own risk obviously.

[pr_osit]

Have opened a new support case today for this
Case #07525970 — Invalid repositories appearing every few days

[Polina]

Hi All,

@pr_osit Thanks for sharing the new case ID. It will be escalated to RnD for additional research.

@aeph Do you also have an open support case on this issue? (I looked through the above thread, but didn't see it)

[aeph]

@Mildur, @pr_osit yes support case id is #07525662.

More and more repos are falling into invalid state...

[Polina]

Thanks noted. I'll pass this information on to RnD.

[aeph]

@Polina, @pr_osit now he Repo synchronization failed with "Synchronization failed" "The response ended prematurely. (ResponseEnded)".
...

[pr_osit]

sorry haven't seen that one before, best to work with support on it and push for an escalation/remote session if you need to.

[aeph]

Update from my end:

Removed the two invalid repos yesterday and readded them again. Sync was started and took a while and failed with the same message above. So even removing and readding resulted in a "Synchronization failed" state. After a while the reindexing/ syncing started again (automatically) than it worked out. But due to removing and readding the jobs are running a full sync now. The two Repos are having round about 60TB so its taking very long..

I am happy that its "stable" now but if this happens over and over again we do not meet our SLAs because we are losing restore points.

What i have tested during indexing/ synchronization:
- Tested the Repo endpoint from the server by running a speed test against the repository endpoint (ping was around 1-2 ms and speed was nearly the same as our bandwith). So its definitly not a bandwith/ firewall/ proxy problem.

I still think that its a NATS/ pool problem or a problem with the postgres cache sync.

What I have noticed:
- I did a "rescan" (rescan button) on all of our proxies (5 proxies) and one of them did not answer although he was shown as active/connected in the list. Even a restart of the proxy doesnt solve this behavior. I removed this proxy from the pool and did a rescan again and than the proxy responded.

@pr_osit maybe it works if all repos belong to a dedicated proxy and do not run in the pool? But it would be a shame to lose the advantages of proxy pooling.

[pr_osit]

@aeph
my config sounds very similar to yours, we have 8 proxy servers in a single proxy pool. I also believe the issue is related to NATS or PGSQL on the management server
will see what support come back with and i'll update this with any progress

[dennis.yu]

@pr_osit
I have the same issue (Repo show invalid) on v7a. I’d like to ask how you resolved this issue previously.
Is it use the following steps?

The fix is stop all veeam services
Stop nats service
Browse to C:\ProgramData\Veeam\Backup365\nats
Delete/move the jetstream folder out of this location
Restart all of your proxy servers if you have multiple
Restart your management server
Veeam services will restart
Jetstream folder will be recreated
Invalid repos should resync and begin working again.

[pr_osit]

Hi @dennis.yu
That advice is specific to issues on v8 only, for v7 the fix is completely different as the way the proxy and repo works is totally different in v7, I suggest logging support case they will guide you on resolving it for v7.
It basically involves stopping the service on the proxy and clearing the proxy local cache (each proxy/repo has it's own cache on v7).