Failed to process team: Team XXXX- . Cannot parse page operation response.

[dotdk]

Hello all

Anyone else experiencing this error since friday 11/6, and maybe have a solution ?

• Failed to process team: Team - "NAME" . Cannot parse page operation response. [ ref:_00D30RWR._5007V1rvZki:ref ]

We are seeing this across a large number of our customers, but not on all their teams sites.

We have logged a case: Veeam Support - Case # 04858043 - but no solution yet.

Initially we were told it might be related to this:
https://www.neowin.net/news/microsoft-t ... rtificate/

but since that has been resolved and the errors we were facing are still present, it must be something else.

We are on version 5.0.1.252 using Modern Authentication

[Johnny L]

We got exactly the same here, started right before the weekend. I got a list of 4 steps to try from support (which all seemed un-likely to fix the issue) that didn't do anything but waste an hour of my day.

It looks like it's a template being sent out, as it's ending with "If the issue is not fixed let the customer know that we are aware of this issue and we are awaiting for a fix to be released soon."

[daety]

We have the same error. Please fix.

[Mike Resseler]

We are aware of this and our teams are investigating since last Friday. Please keep working with our support teams so our engineers have enough logs to go through.

[Gostev]

Let's hope this is yet another temporary issue on the Microsoft side. Seems to be a tradition to have them weekly these days! I mean, literally

[Mildur]

It would be time for an official API from Microsoft to do Backups of all this objects and metadata.
Without an official API, this issues can always happening out of nowhere.

[Switchie_Urs]

Same error for 1 out of 3 tenants on same VBO server. Version 5.0.1.252 using Modern Authentication. Other 2 tenants Teams backup jobs keep working normally.
Should I log a ticket also?

[johan.h]

Please do. It would be good to have the support logs for this.

[dotdk]

If needed I can upload additional logs from some of our other customers who also have this problem. I only uploaded one log bundle to keep it as simple as possible for support.

[Switchie_Urs]

Please do. It would be good to have the support logs for this.

just logged #04862618

[Pagrossman]

Hi there,

We have the same problem.
Last success job: 6/10/2021 6:06pm
First failed job: 6/10/2021 7:00pm

Case #04860462 — Failed to process team: <TEAM> . Cannot parse page operation response.

I sent the logs to support and got the information that this is know bug.

[Aaron Wells]

Hi everyone,
We are seeing the same issue for one of our tenants (our own) but not others

First Failure 12/06 03:46

Failed to process team: <TEAM> . Cannot parse page operation response.

Running version 5.0.1.225

Case #04864861

[Mike Resseler]

Please keep working with our support teams. It looks like we have found the root cause at this point in time, and we can make a hotfix to make sure that we can protect it. However, the problem is restoring (the most important part). We are now working with Microsoft to understand why this is. The more logs we have, the more it helps us in letting MSFT understanding what issue we are facing (and that there are many suffering from it).

[dotdk]

I have uploaded additional logs from more of our customer to our Case # 04858043

[agrob]

have the same issue on some of our teams...

[tmollgoodwill]

I am also having the same issue since Friday with only 2 of my Teams.

Case #02356261 with logs.

[ks.any.cloud]

+1 we have a customer which just created a teams job, and now are facing this error on some of the teams that are being backed up.
If logs is wanted, i can also uploade them.

[Gostev]

To provide a little more details on the issue we're seeing:

The root cause is that for Tabs of some Teams (in our lab it is the team created more than 2 years ago), O365 API suddenly started to return an integer value in the String format as their ID, whereas previously it was a GUID. Interestingly enough, when we changed our code to expect and accept this new tabID value type, we were able to perform backup - but then faced an issue with restore. Because when restoring such Tabs from backup, Microsoft then complains with error 400 that "tabId needs to be a valid GUID" and we do agree with this statement!

So by all means, it looks like MIcrosoft broke something on their end for some Teams unintentionally, so we opened a support case with them.

[lharmer]

We also are seeing this on 2 tenants being backed up, logs uploaded on 04861093

@Gostev support are still advising steps to run through re-configuration on client, excluding Teams and recreating jobs etc etc. Is it worth ensuring the message has reached support that this is a known issue being worked on? Save everyone a load of wasted time!

[Gostev]

Totally agree, I passed this feedback.

[UPDATE] Response from the VBO support leader:

"Based on the information I have - the pattern of the issue is still not very clear - Microsoft and our QA asks to gather as much info from customers as possible, therefore we still ask the customers to run isolation tests"

[Keith.Gregory]

I had three Teams give this error starting last week. One had a "One Note" tab that wont open in Teams - I removed it. I tried restoring it from a while ago and still it wont open in Teams - so definately an MS issue. Hasn't been used in ages so no loss. The other two open fine in Teams so can't see whats upsetting it.

[serversiggi]

same issue
#04860105

this is what I found in the logs:
6/17/2021 5:36:18 AM 89 (11768) Warning: Failed to read response content
6/17/2021 5:36:18 AM 89 (11768) Warning: Error converting value "1158469987" to type 'System.Guid'. Path 'value[1].id', line 1, position 815.
6/17/2021 5:36:18 AM 89 (11768) Warning: Could not cast or convert from System.String to System.Guid.
6/17/2021 5:36:18 AM 89 (11768) Error: Cannot parse page operation response.
6/17/2021 5:36:18 AM 89 (11768) Type: System.Exception

[lharmer]

Support/R&D have provided us with a hotfix this morning which seems to have resolved the error. Great work

[agrob]

Have received the fix as well. Teams backup is running fine again.
Thanks to Veeam Support and R&D! Great Work!

[Mike Resseler]

Awesome, thanks for letting everyone know guys! I hope this solves all issues. I know we are still talking to MSFT at this point in time

[daety]

Support/R&D have provided us with a hotfix this morning which seems to have resolved the error. Great work

where can the hotfix be downloaded?

[Gostev]

You need to contact support. It will not be published until it is proven more.

but then faced an issue with restore. Because when restoring such Tabs from backup, Microsoft then complains with error 400 that "tabId needs to be a valid GUID" and we do agree with this statement!

To work around this particular issue, the hotfix will restore tabs as new. Basically, we will create new Tabs with the new GUID-based tabId, instead of restoring the backed up tabId. We're not sure if not preserving the original tabID will cause any issues in the Teams client, so just something to keep in mind and monitor for those with the hotfix.

[dotdk]

I have gotten the hotfix from support and have applied to it to one of the failing customer, and that has fixed the issue, so I expect it to fix it for the remaining failing teams.

[ks.any.cloud]

When can we expect an official and stable release of this fix, that does not break something else ?

[Gostev]

To the best of our knowledge, the current hotfix meets all your requirements already "official, stable and does not break something else".

Of course, in the ideal world Microsoft will simply fix down the road whatever they broke for those old team objects, making the workaround that the hotfix brings unneeded. The regression seems to only affect teams created long ago, the more current teams behave just as before.