O365 Backups fail since 1.1.2022 01:00

[ibschreiber]

Hi,
happy new Year to all of you!
Since exactly 1 hour after midnight, almost all O365 Backupjobs of all my clients fail with the following error:
01.01.2022 01:00:46 :: Failed to process team: COMPANYNAME. Organization (ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) which the team (ID: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy) belongs to not found.

Does anyone experience the same failure? Any ideas what the cause could be for this?#

Have a good time!
Michael

[micoolpaul]

Hi,

Couple of questions? Does this impact the entire tenant? Aka Exchange + SharePoint + Teams or just a portion?

Modern or basic auth? Which patch version? There’s been some hot fixes…

Potentially related, Exchange 2016/2019 have had issues with anti-malware since midnight due to the numbering of definitions exceeding the maximum value of a signed long variable type, could be that we’ve got a similar code issue on Microsoft’s side (hopefully not Veeam’s side)

[ibschreiber]

Version: 5.0.1035, no hotfixes installed.

I'm not sure what exactly is failing, but I think it's only Teams that are broken.

Modern Auth is used in all accounts.

[nielsengelen]

Hi Michael,

Please open a support case for assistance and let us know the case ID. We are not aware of any Microsoft changes at the moment.

[ibschreiber]

Hi,
just opened a case with id 05208240

Thank you all!

[kavaa]

We have the same issue, also created a case for this.
We don't have it with all backups just one at the moment...

[nielsengelen]

Please also open a case for insight. It may all be related to the Exchange bug with this specific number but insight is required.

[murchaid]

We are seeing the same issue across multiple tenants that i manage. All just within teams and with that exact "belongs to not found" error.

[FrancWest]

Same issue here. Case #05208909.

[MPG]

We have also the same issue in our enviroment.

[DaStivi]

got the same error on different tenants... but interestingly not all tenants are affected... at least not yet

[FrancWest]

Reaction from support:

Currently this is a known issue that we where able to identify, the RnD and QA are currently investigating this and i will get back to you as soon as possible with an update.

[Mike Resseler]

To those who haven't created a support call yet, please do.

I kind of fear this has to do with the specific date

[Regnor]

Coming back from my holiday and renewing the NFR license for my lab, I can reproduce this issue.
Currently I don't have any affected customer environment.
@Mike: Does it somehow help you if I open a ticket for the NFR environment or do you already have all necessary details?

[mkobylanski]

Same here: Support Case # 05211128

[ibschreiber]

quick update: As requested I created a new Repository and did a teams only backup. Worked flawlessly. So, for now, I'll create teams jobs on separate repositories until this issue is resolved.

[dasfliege]

Same here. Case #05212265

[kavaa]

The client that we have problems with has over 600GB of files in Teams Channels... So adding an extra repo for that will take some time to process

[Mike Resseler]

Update to all

We are preparing a hotfix for 5d CP2 and soon after that for 5a and b and so on. Stay tuned, support will contact you once the fix is ready and tested

[murchaid]

Interesting enough I have one server backing up 5 different tenants at one site and 4 of the 5 tenants are experiencing the issue when backing up teams... I have other sites at the same version without the issue.

[Renaud]

Case #05213356
Same issue for 2/7 tenants.

[PascalVorsmann]

Case #02480340
Same issue for 1/1 tenants.

[dasfliege]

Any ETA for the fix?

[Polina]

Hi All,

The hotfix for the latest 5d CP2 (build 5.0.3.1051) has been provided to support. RnD is now working on the patch; I'll update this thread with a link to KB once it's ready.

[FrancWest]

I just received the hotfix for 5.0.3.1051, backing up Teams is working again.

[Mike Resseler]

Good to know that it is working. Support will reach out to all of you as soon as they can (and depending on what version you are running)

[Liberta]

Unfortunately 5d CP2 (build 5.0.3.1051) does NOT fix the issue for me!

[Dima P.]

Hello Liberta,

To start the investigation of your issue please open a support case and share the case ID in this thread. Thank you!

[Liberta]

I am running a free test setup, so it is not possible to open a case.

[Dima P.]

Liberta,

It should be possible to raise a support ticket even for Community Edition via veeam.com (yes it goes to the free queue but RnD team can pick it up from there). Thanks!