Veeam VBO v6 issues

[matsnijman]

Hi all,

As a long time user of VBO and heavy user i have seen my fare share of issues with VBO.
However hoping time would heal all wounds and Veeam would fix them with new updates i happily got to work with Veeam V6.

I have seperate instances of VBO running and decided to upgrade my most default instance.
No REST api used, only 2 orgs, 4 jobs per org and 1 repo per org.
No seperate proxy and all is being offloaded to Wasabi.

I am running Windows Server 2019 with 16GB and 8 cores.
So very basic.

Started the installation which went fine, but after the upgrade i had to upgrade the repository.
This is where the first issue started.
I was unable to upgrade the repo.

[11.03.2022 10:15:19] 50 (6712) Failed to upgrade repository database
[11.03.2022 10:15:19] 50 (6712) Error: JetError -1504, JET_errNullInvalid, Null not valid
[11.03.2022 10:15:19] 50 (6712) Type: Veeam.Jet.JetError

There were no jobs running at the time.
I rebooted the server after updates, so i know all existing connections were closed.

After this is contacted support and i had to stop the proxy, emtpy the persistentcache folder and start the proxy.
This fixed the problem. Upgrade went succesful and i was able to start the jobs again.
However taking a lot of time because it needed to fill the cache again.

Now after the weekend another issue popped up with OneDrive backup.
This suddenly stops working and i get issues with the repo again.
14-3-2022 07:45:02 :: Failed to process OneDrive: (https://xxx-my.sharepoint.com/personal/xxx). Unable to access repository (xxxx\PersistentCache\repository.adb) JetError -1605, JET_errKeyDuplicate, Illegal duplicate key

I am a little baffled about how this product can be released which such issues coming up from even the most basic of environments and doing just the bare minimum of upgrading a repo and starting a job.
I opened a new case to fix this latest issue, but i image there are a lot more problems out there. (Case # 05329513).
Also i am very reluctant to even try to upgrade our biggest environment with this new release.

Just wanted to share my thoughts and maybe see if there are more issues i need to be aware of.

[Mike Resseler]

Mats,

As you can imagine I am not happy with your issues and I hope they resolve them quickly. Unfortunately I cannot give you a solution, support will need to dig into it. I am a bit surprised though, as we already have over 1000 upgrades without any issues, but I do notice also a couple of reports about jet DBs having issues in getting upgraded. I hope our engineers find (besides the solution) also the root cause, but we have not seen this issue in our testing and trying to "mimic" it seems very difficult at this point in time.

I apologize for the inconvenience

[matsnijman]

Hi Mike,

It's just strange that with even the most basic of environments these issues happen.
Got in contact with support and had to clean out the persistentcache once again.

As you are familiar to our other environment you can see my worry about upgrading.
Will keep you updated about the situation and about any issues popping op.

[Mike Resseler]

Thanks Mats. From my side I will push QA a bit more for the investigation on the root cause

[matsnijman]

Clearing the cache again solved the issue.
Root cause investigation would be great.

I will wait a while for updating our other environments till these issues are ironed out.

[matsnijman]

I am currently om my third issue since the upgrade.
This is for my small environment with 2 orgs and all pointing to wasabi as repo's.

Now i am unable to backup 1 of my sharepoint sites.
23-3-2022 13:47:55 :: Failed to process site: https://xxxx.sharepoint.com/sites/xxxx. Unspecified error :: 0:00:21

Have a case again (05345720), and again they are asking me to clear the persistent cache.
Which did not solved the problem, but this is the 3rd time since the upgrade i had to clear the cache of 1 org.

Seems like vbo v6 is far from stable.

[Mike Resseler]

@matsnijman

Let me dive into the case because I fear something else must be wrong. We have no other knowledge of that many issues with an upgrade. On the contrary, purely from the statistics and number of requests it seems rather stable.

[helpdesk@gmb.eu]

I have the same problem, needed to clear te persistant cache for another problem, I couldn't update my backup repository, only when I renamed de cache file.
But it takes 3 days before a full backup run is done, so if there is another solution or patch I would like te hear it too.

I get the error on mailboxes, onedrive and sharepoint sites :
Failed to create a restore point in the repository: Unable to access repository (E:\BackupStorage\AzureBlobStorage\PersistentCache\repository.adb) JetError -1605, JET_errKeyDuplicate, Illegal duplicate key

[Mike Resseler]

@helpdesk@gmb.eu Not sure if I understand it. That error you are receiving is still here today or is it solved after the persistant cache clearing? If not, what is the case ID for that issue?

[helpdesk@gmb.eu]

after installing I had problem that my Repository was out of date and couldn't be updated (case 05327000). That was fixed with deleting the cache file.
But after the first initial run which takes a long time) it started giving errors mentiond above. I have not yet made a case as I was hoping this would be solved in this threath.

[matsnijman]

@helpdesk@gmb.eu This was indeed the 2nd problem what popped up after upgrading the repo.
Clearing once again the cache solved this issue for me, so you can try this method again.

[helpdesk@gmb.eu]

Deleted the cache and rerun the backup, after 80 hours of running the full backup again I still have the same issue.

[Polina]

Hi @helpdesk@gmb.eu

Do you see the issue on the same repository?
Could you please attach the new logs to your support case?
I'll follow up on it with our engineers.

[Polina]

Quick update: Our support team has opened for you a new case (05362090) and escalated it to T2.

[Johan M]

I also have this problem on 2 servers, one for a very large customer and one for a small customer.
We had a problem where we had to resync our cache into a new folder and after that we got the same error as the others.
My case id is 05359192.

[Polina]

Hi Johan,

Thanks for the heads-up, I'll forward this info to the support team.

[Polina]

Johan,

Your case has been escalated to RnD. As of now, I expect this issue to be resolved and fixed in the next cumulative patch.

[helpdesk@gmb.eu]

After contact with support I received a hotfix, applied this for the Veeam.Archiver.Repository.ObjectStorage.dll file...
Then had to remove the Persistent Cache file again and start a new sync.

Started new backup and have no more issue wth this errormessage.

[TStawney]

I have experienced the same issues listed here. It really feels like this version was not ready for production. My problems have been going on for almost a month now and response from support has been abysmal. Below are my two case numbers for reference.

05347452

02419352

[Johan M]

Just like gmb I also got the Veeam.Archiver.Repository.ObjectStorage.dll file.
After a new sync with a new persistent cache it worked with no errormessage.

[Mike Resseler]

@TStawney Please let me know if you haven't received updates today from support.

[TStawney]

Support did reach out to me and I received the hotfix, but I am still receiving the error message below even after clearing the persistent cache (again). I just uploaded more logs to the case.

Unable to access repository (xxxx\PersistentCache\repository.adb) JetError -1605, JET_errKeyDuplicate, Illegal duplicate key

[Mike Resseler]

@TStawney

Hmmm, as far as my information goes, that hotfix should solve it. I will follow the case further as we are about to bring out the cumulative patch for the hotfixes post v6

[Bjoern]

Hi,
unfortunately, i ran into the same error: Unable to access repository (xxxx\PersistentCache\repository.adb) JetError -1605, JET_errKeyDuplicate, Illegal duplicate key

So i installed the cumulatice patch P20220413 (6.0.0.379).
Afterwards i stopped the services, deleted the persistent cache, restarted the services and started the Jobs again.
Now i have a lot more "duplicate key" errors (also some different errors too) and the performance is very poor. So, more issues than before.

Any help or hints appreciated.
Case # 05443730

Best regards, Bjoern