Multiple problems (indexing stuck, copy job stalling, poisoned repo) since updating to v8

[aeph]

Hi all,

since the update to v8, we are constantly experiencing new problems.

Indexing Stucks at X% (24 hours and more); (Case-Id: 07418245):
One of the first problems was and is indexing repositories. Sometimes the repos were stuck in the “Indexing” status for more than 20 hours at a fixed percentage. We were provided with a private hotfix that partially solved the problem, but now only one repo is still in “Indexing” status at 6% and that for more than 24 hours.
Nevertheless, when creating new repos it always seems to be a matter of luck whether it works or not. We have also tried removing repos from Veeam and adding new ones. In this case, this was also unsuccessful.

In general, it seems that the problems are somehow related.

Poisoned Bucket
At one organization, the regular jobs were running stably with v8 until, out of the blue, the job started with the error “Failed to create a restore point: A poisoned bucket index is detected. Trying to add entities to an already occupied bucket: ... Job failed.” and no restore point could be created. Several reruns were also unsuccessful. (Case-id: 07438125).

Copy job stalling (Case-id: 07438124)
Copy jobs are started and run for 20 hours, but only process approx. 100 MB, for example. The jobs do not report any real errors, they simply do nothing more.

We are in contact with Veeam support, except for emptying folders (temp appdata folder for nats) no solution has been presented to us yet. However, it is probably a problem that has been reported by several customers.

Unfortunately, this is very inconvenient for us because we currently have to disable the copy jobs.

So all in all we have several problems and have been in contact with Veeam for over 2 weeks and have really not been able to get any improvement.

We are urgently looking for a solution to address all the above problems.

[Polina]

Hi aeph,

I'm sorry that the upgrade to v8 wasn't smooth for you. This release has so many changes under the hood that it is probably one of the most complex of what we have delivered so far.

RnD is working on fixes for the issues you listed as we speak. If things go well, I expect the next patch (which should include even more fixes) next week.

[aeph]

Thank you, as mentioned above, a solution is absolutely essential for us. It is very important that we get the environment back into stable and fully functional operation.

It would be nice if we could be informed about the fix in advance so that we can prepare the resources for it.

[lungtendo]

Hi aeph,

I experienced some of the issues - frozen indexing and failing to run jobs - which were resolved by disabling the firewall between the VBO server and the remote proxy, but I'm waiting for some answers from the support (case #07436087 ). The jobs that were assigned to the local proxy didn't have any issue. Other issue I have is not being able to explore backups prior to v8 upgrade (case #07436260).

[aeph]

Having another new problem: Job Runs sometimes start as Full instead of Increment (without a change being made to the organization). Of course, this takes a very long time.

All in all, v8 is a real pain, we would like to see updates as soon as possible! Struggling with the problems for more than two weeks now.
When will we get a fix?

[thasenack]

Hi aeph,

I am also experiencing some similar issues. Indexing on the repos completed successfully.

However, I've seen “Failed to create a restore point: A poisoned bucket index is detected. Trying to add entities to an already occupied bucket: ... Job failed.” and no restore point gets created.

Interestingly, this only seems to be affecting our jobs that include SharePoint, OneDrive or Teams. I have a large number of email only jobs that are completing without error.

I've also observed that many jobs are not running as efficiently as before v8. Not sure if that is because of the other issue or because of v8 in general.

Just need to see what Support comes back with (Case-id: 07443249).

[Djlocke]

Hi @ all,

same problem Indexing Stucks and upgrade der repo's.

Case id: 07446294

[Mildur]

Hi all

Our RnD teams are working on the fix for the stuck "indexing issue".
I have no ETA, but the progress looks good so far. I expect more information soon.

Best,
Fabian

[Mildur]

Our RnD team has finished a private fix regarding indexing issues.
It can be requested through a support case.

Best,
Fabian

[sjdgmt]

@Mildur we have similar issues after upgrading to v8, multiple repositories had to be removed and re-added to upgrade them. After indexing and a new backup cycle we're missing all past restore points. I've been in contact with veeam support but no luck so far. Is there a definitive fix for the issue, veeam support installed a patch to update vbo to 8.0.2.501 but that didn't help. Current case number 07417104, it's getting a bit frustrating.

Hi all

Our RnD teams are working on the fix for the stuck "indexing issue".
I have no ETA, but the progress looks good so far. I expect more information soon.

Best,
Fabian

[pat_ren]

is there a fix for poisoned repo? I'm getting this one today now too. have not opened a support case yet, it's supposed to be my day off so hoping it's a quick fix.

[aeph]

Our case was not solved. In the short time available and because we wanted to comply with our customers' SLA, we decided to create a new repo and pull a new full backup. With the poisned repo, we were at least able to load the restore points, which helped us here.

We were lucky that it was a rather small customer and the full was completed in one day.

All in all, however, it was an absolute emergency solution and ultimately, of course, it cost us more money. If this happens to one of our big customers, we're doomed. Transparency from Veeam would be very important here.

[Mildur]

Hello @sjdgmt

I apologize for the late answer. The first cumulative path is expected soon (it's in the final phase of testing).
Let me shortly review your case.

@pr_osit
I see that a fix for the "poisoned repo" issue is included in the upcoming patch. However, I can't determine from a forum comment whether it addresses your specific situation. Either way, we recommend deploying the patch when it is released, as it will resolve many known issues with version 8.

@aeph
Which one of the three cases are you referring to? The "poisened Repository"?
Discussing three different situations in a single topic makes it difficult to follow. Normally, we should use dedicated forum topics for each issue.

Best,
Fabian

[aeph]

@Mildur, yes i was referring to the "poisned repo" issue with the case number (07438125).
But good to see that a fix to this is part of the new patch.

[Mildur]

Hello Aeph

Thank you.
I reviewed your case, and your support engineer noted a workaround for the "Poisoned Repositories" issue. It involves updating a property in the configuration XML file, but it only works under specific circumstances and requires our support team to verify the logs first. I understand that this workaround came up after you decided to create new buckets with full backups.

I also verified the internal bug number in your support case and can confirm that the root cause is scheduled to be fixed with our first patch.

Best,
Fabian

[iju]

Hello Fabian,

I have 2 repos (size 12+tb) which doesn't certain any restorepoints after sync/indexing. Is this also fixed? (Case #07424488)

Br Ib

[igibason]

We have been down since September 11th after upgrading to v8. Battling with the error "Failed to create a restore point: A poisoned bucket index is detected." on our SPO, OneDrive, and Teams backups. EXO backups seem to be working so far. Case: 07416613

Is there an official bug report or announcement posted I can refer to? We have never been down this long with backups before and its pretty concerning.

Any updates on a fix would be appreciated.

[Mildur]

Hello igibason

I checked your case notes. Please confirm, are you currently only facing the "poisoned repository" issue? Are the other issues solved?

Any updates on a fix would be appreciated.

We expect the final fix for the "poisoned repository issue" to be included in the upcoming patch (final phase of testing).

Is there an official bug report or announcement posted I can refer to?

We will provide information about solved issues in the release information when we release the patch: https://www.veeam.com/kb4656

Best,
Fabian

[Mildur]

Hello @iju

Yes, we plan to include fixes for the "missing restore points" issue in our upcoming patch.

There is currently a workaround available in the form of a special fix along with some manual steps, which can be provided by our customer support. However, I recommend waiting for our official patch if you can.

Best,
Fabian

[igibason]

Hi Fabian,

Thanks for the reply and looking into our case. Correct, we are facing the "poisoned repository" issue along with our copy jobs not finishing case number 07416613. They have been running since Oct 1st and seem to be stuck.

I did read that release information but was thinking it may be a good idea to post current known issues so others can make a decision before updating.

Looking forward to the upcoming patch

Thanks again!

[Mildur]

Hi guys

We have released our new patch (P20241008 - 8.0.3.1044), which should resolve many of the issues discussed in this topic. If you have an active support case open, I recommend first consulting with your support engineer before applying the patch.

Veeam Backup for Microsoft 365 v8 - P20241008 (8.0.3.1044)
https://www.veeam.com/kb4656

Best,
Fabian