Invalid repositories appearing

[aeph]

@dennis.yu there is a KB article especially for V7: https://www.veeam.com/kb4468

May i ask which Repo-Type/ Provider you are using? We are using wasabi and they had a few issues the last weeks (unplanned maintenance) etc (Ref.: https://status.wasabi.com/history).

This is just an idea that I cannot substantiate. But nevertheless the state should not switch to invalid that easy.

[pat_ren]

Just an update on my situation, I had another 8 repos become invalid today, was able to fix and resync once again with the steps I mentioned before, however still waiting for a reply from support on the case I logged about this.

[aeph]

Thanks for the update. I also uploaded fresh logfiles regarding my case yesterday. I have not yet received an answer from support.

May i ask if you are using Wasabi as well?

[aeph]

@pr_osit any updates on your case?

Whats also new is the following message (on object level):
Failed to process site: XXX. Failed to apply immutability to object Veeam/Backup365/Veeam/Organizations/PATH.../Files/Index/Root.00000000000001BB because the object is already immutable.. Access denied. Could not perform the operation because of missing permissions or wrong data center region.

Having 2-3 jobs with about 5-6 objects having this issue.

[pat_ren]

@pr_osit any updates on your case?

Whats also new is the following message (on object level):
Failed to process site: XXX. Failed to apply immutability to object Veeam/Backup365/Veeam/Organizations/PATH.../Files/Index/Root.00000000000001BB because the object is already immutable.. Access denied. Could not perform the operation because of missing permissions or wrong data center region.

Having 2-3 jobs with about 5-6 objects having this issue.

Hi @aeph, my case is still ongoing, still getting about 2-3 invalid repos every couple of days. Working with level 3 support on this now and we are narrowing down the issue.
We do use Wasabi but we do not use immutability so I have not seen your error before.

Our errors look like this:
19/12/2024 9:38:46 AM :: Unable to check object storage lock (reason: Another process has rewritten the consistency information (expected value for LastWrittenBy: c54f5419-c109-4b97-9392-4e5f6d3b48f6, actual value: 1091aeef-9aae-455b-8826-80e264424b41).)
19/12/2024 11:10:56 AM :: Repository is in an invalid state. The local cache and object storage are not synchronized.

Currently the issue appears to be related to object storage time markers but still investigating.

[aeph]

Hi @pr_osit, we encounter the same error (Repository is in an invalid state. The local cache and object storage are not synchronized.) when it comes to a invalid repo.

The above error regarding immutability is just on object level so the job itself is running through.

Wasabi had another major maintenance, this time I disabled all jobs during this maintenance and so far there have been no further problems from my side.
What we still have, however: Jobs hang up irregularly and these jobs cannot be terminated via the UI.

[pat_ren]

Just updating this, support have gotten back to me today, after a couple of weeks working with R&D (or everyone being on leave) they have worked out the root cause of this issue, and should be fixed in v8.1 next week. I received a detailed analysis of the logs and the issue playing out step by step between multiple proxy servers, however i won't share all that as it contains client details, so here is the short version. This is good news for anyone else experiencing this issue such as @aeph

Proxies in the Pool can have different roles that define their responsibilities - Scheduler, Processor, Resolver, etc. One of these roles, the repository agent, is a role that allows the proxy to work with repository metadata, which is stored in blobs like RepositoryConfig, BackupKeys, and RepositoryLock. While all proxies in the proxy pool can write backup data to every repository in parallel, only one proxy with a Repository Agent role can modify the metadata of the given repository. The Repository Agent role can be delegated from one proxy to another.

Root cause:
A new proxy in the pool acquired the Repository Agent role before the previous Repository Agent proxy released resources. The previous agent couldn't release resources quickly due to errors with DB.

[pat_ren]

Just updating this, the issue of invalid repositories has not been fixed with the v8.1 update, we still have a lot of issues after upgrading.
Checked things today and found 30+ invalid repos and 18 out of sync repos (that's new) so going to continue working with support on this issue.
Starting to get very frustrated with this product/update.

[Polina]

Hi Pat,

I see that your support case #07440695 has been closed. Since you confirm that the issue persists after the upgrade to 8.1, can you reopen it so that we could investigate it further?

[MaartenA]

Same issue Case #07601224

[edh]

Same issue here... "Cannot find expected repository lock..." 2 Repos today

We need some UI or Powershell cli for fix repos and unlock fast this issues. Opening a ticket...

[Polina]

@edh

Please post your case ID once you get it.

[edh]

Same issue here... "Cannot find expected repository lock..." 2 Repos today

We need some UI or Powershell cli for fix repos and unlock fast this issues. Opening a ticket...

Case #07603113

[pat_ren]

Hi Pat,

I see that your support case #07440695 has been closed. Since you confirm that the issue persists after the upgrade to 8.1, can you reopen it so that we could investigate it further?

Hi Polina,
I have had this case open for a while for same issue, I have been working with Peter and the R&D team and seems like they are making some progress and may have identified the issue.
Case #07525970

[edh]

I'm really worried that we only handle a few hundred storages and I'm sure Veeam handles tens of thousands. If it affects us, how is it possible that it does not affect Veeam and if it affects you, there is still no permanent solution to these blockages.

[Polina]

RnD is actively researching. So far, there's no clear pattern and the root cause is still unclear.
I'd like to ask you to upgrade to the latest build 8.1.0.331. If the issue persists, please immediately inform our support engineers (and provide your logs, of course).

Thanks!

[LandonB]

Hello,

Also having this same issue as of two days ago. Out of nowhere our backup repository is showing Invalid. Spent the better half of the day today trying to get this working going back and forth with Support (Case #07604771).

I upgraded to 8.1.0.331 today as well and the issue is still persisting. Following the steps from the support rep, I removed my storage repository and attempted to re-add it, however I am not able to re-add the storage repository.

[edh]

Same here today, 6 repos appeared . Our version is 8.1.0.331.

Some of the repos go online again removing the job, removing repo, re-add repo, re add job. But this cause again a sync, and some of them are about 70Tb... and will take too much

[Polina]

Noted, thank you for the information.

[LandonB]

just received an update to my support ticket that provided a hotfix for the issue. Support provided version 8.1.0.2026, which after installing the update and allowing the repository to resync, reindex, and upgrade to the latest version - fixed my issue and my S3 repo is showing normal again.

[lmendel]

We have the same issue for 2 repo's at the moment. Case with support opened this morning. I've managed to fix the issue myself in mean time, by taking the linux proxy servers out of pool and rebooting then adding back to pool. You can remove all but 1 server from the pool at the same time.

My guess is that there was a process locking or messing something up on a proxy somewhere. We have 4 proxies in the pool and 2 were stuck waiting for MongoDB process to stop for a minute or 2 while rebooting, the other 2 rebooted almost instantly.

[aeph]

We are having the issue from time to time again.

Since updating to latest 8.1 its getting worse. Some bugs addressed in the release notes are not fixed in our case as well.

What I was able to observe: When we back up our controller server, this server is unavailable for a few seconds. This apparently ensures that NATS no longer knows its way around and the repos go into the invalid state. In this case, rebooting all proxies and the controller at the same time ensures that the valid status is restored.

Apparently NATS is very susceptible here if the controller has short interruptions, the reconnect does not seem to work properly here.

Is there any new release/ fix in the pipeline? The problems seems to increase since 8.1 release..

[pat_ren]

I have received a private fix for support from this, in addition to changing the repo maintenance from daily to monthly, and tweaking some settings in the proxy, we haven't had the same issues with this that we were having before. From all the logs provided they were able to work out what seemed to be causing the issue (communication between multiple proxies failing to handle tasks correctly) and so the fix does appear to be working for now.

I can't share it but if you reference my case support may be able to assist you with the same fixes
Case #07525970

[aeph]

Nice to know!

Has this hotfix also fixed the stalling of jobs / jobs that no longer perform as well?

[pat_ren]

Nice to know!

Has this hotfix also fixed the stalling of jobs / jobs that no longer perform as well?

No unfortunately not yet, I'm still waiting for a fix for that but one possible cause has been identified. I believe that fix should be available by end of this week.

[Polina]

@aeph

The latest patch 8.1.0.3503 has several fixes for jobs that appear stuck. I'd highly recommend you upgrade first to see if it resolves your current issues.

[lmendel]

We updated to 8.1.0.3503 4 days ago, and we just had issues with 2 repo's

[dasfliege]

issue with invalid repos hit us as well today. Case 07680027

[dasfliege]

Tried clearing NATS jetstream and also re-adding the repos, but without success. I am not even able to re-add the repo. Getting the same error: "Repository is invalid.: Exception of type 'System.Exception' was thrown"

[mij4sure]

We also see problems in the latest Veeam version of Microsoft 365 with several customers that the WASABI repos go offline (invalid). Case is open but no feedback yet. Also "Repository is invalid.: Exception of type 'System.Exception' was thrown". Respository is invalid.: Cannot find expected repository lock in object storage: re-add also not working.