Team chat sync problem

[dbeerts]

Hi all,

Every once in a while, a job keeps running on an mailbox object forever. In the logs you can find this warning:
Warning: Possible team chat sync problem was detected
Solution is to create a new job with same repository with the specific object included, so a full read of the object is done. When this job completes successfully, you can retry the original job and it should process the object successfully again.
My questions:
-Am I the only one facing this issue?
-Why keeps Veeam processing this object for hours, days, weeks, ... instead of cancelling the job after X retries? Now I have to cancel the job with the result no restore point is created for all other objects.

[Mildur]

Hello DBeerts

There is configuration parameter which could fix this kind of error. I can see multiple cases.
Please open a support case and let our customer support team confirm from the logs that you are affected by a known issue.

Best,
Fabian

[pr_osit]

Just saw your post after mentioning something similar related to 365 group backups, teams posts or mailbox backups getting stuck forever. Don't think you are alone, something weird going on. Will see what support say, but I'm interested in the outcome of this.

[dbeerts]

Hi Fabian,

Support provided me a fix and it solved the sync issue for this one, thanks!
As I mentioned, it's not the first time we have this error message. The first time was some years ago, so I'm not sure we are hitting a new or old 'bug'.
Will this fix be implemented in future releases or?

[Mildur]

Hi DBeerts

Thanks for letting me know.
I can try to find that out if you share the case number with me.

Best,
Fabian

[dbeerts]

Hi Fabian,
Case # 07155787
Thank you!

Dries

[Lewpy]

I've just encountered this issue with a particular Microsoft Team, that is holding up both the Mailbox backup and the Teams backup: both of these separate backups (on separate proxies) are currently sat processing the same Team.
Mailbox Job Log is looping through saying:

Code: Select all

[09.04.2024 07:37:05.156]   82 (4120) Processing mailbox: team@client.com...
[09.04.2024 07:37:05.160]   82 (4120) Syncing folder items: Team Chat...
[09.04.2024 07:37:05.202]   82 (4120)  Sync time: 0.0423093
[09.04.2024 07:37:05.203]   82 (4120)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:37:05.203]   82 (4120) Warning: Possible team chat sync problem was detected
[09.04.2024 07:37:05.205]   78 (7904) Processing mailbox: team@client.com...
[09.04.2024 07:37:05.212]   78 (7904) Syncing folder items: Team Chat...
[09.04.2024 07:37:05.289]   78 (7904)  Sync time: 0.0757653
[09.04.2024 07:37:05.289]   78 (7904)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:37:05.289]   78 (7904) Warning: Possible team chat sync problem was detected
[09.04.2024 07:37:05.292]  102 (1260) Processing mailbox: team@client.com...
[09.04.2024 07:37:05.298]  102 (1260) Syncing folder items: Team Chat...
[09.04.2024 07:37:05.342]  102 (1260)  Sync time: 0.0420771
[09.04.2024 07:37:05.342]  102 (1260)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:37:05.342]  102 (1260) Warning: Possible team chat sync problem was detected
[09.04.2024 07:37:05.344]   70 (8084) Processing mailbox: team@client.com...
[09.04.2024 07:37:05.351]   70 (8084) Syncing folder items: Team Chat...
[09.04.2024 07:37:05.401]   70 (8084)  Sync time: 0.0497281
[09.04.2024 07:37:05.401]   70 (8084)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:37:05.401]   70 (8084) Warning: Possible team chat sync problem was detected

Teams Job Log is identical:

Code: Select all

[09.04.2024 07:44:37.438]   99 (1204) Processing mailbox: team@client.com...
[09.04.2024 07:44:37.438]   99 (1204) Syncing folder items: Team Chat...
[09.04.2024 07:44:37.485]   99 (1204)  Sync time: 0.0423228
[09.04.2024 07:44:37.485]   99 (1204)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:44:37.485]   99 (1204) Warning: Possible team chat sync problem was detected
[09.04.2024 07:44:37.485]   99 (1204) Processing mailbox: team@client.com...
[09.04.2024 07:44:37.485]   99 (1204) Syncing folder items: Team Chat...
[09.04.2024 07:44:37.547]   99 (1204)  Sync time: 0.0490471
[09.04.2024 07:44:37.547]   99 (1204)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:44:37.547]   99 (1204) Warning: Possible team chat sync problem was detected
[09.04.2024 07:44:37.547]   60 (2252) Processing mailbox: team@client.com...
[09.04.2024 07:44:37.547]   60 (2252) Syncing folder items: Team Chat...
[09.04.2024 07:44:37.594]   60 (2252)  Sync time: 0.0402483
[09.04.2024 07:44:37.594]   60 (2252)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:44:37.594]   60 (2252) Warning: Possible team chat sync problem was detected
[09.04.2024 07:44:37.594]   61 (2556) Processing mailbox: team@client.com...
[09.04.2024 07:44:37.594]   61 (2556) Syncing folder items: Team Chat...
[09.04.2024 07:44:37.657]   61 (2556)  Sync time: 0.0543962
[09.04.2024 07:44:37.657]   61 (2556)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:44:37.657]   61 (2556) Warning: Possible team chat sync problem was detected
[09.04.2024 07:44:37.657]   61 (2556) Processing mailbox: team@client.com...
[09.04.2024 07:44:37.657]   61 (2556) Syncing folder items: Team Chat...
[09.04.2024 07:44:37.719]   61 (2556)  Sync time: 0.0512755
[09.04.2024 07:44:37.719]   61 (2556)  Changed items: 0, deleted items: 0, read state changes: 100
[09.04.2024 07:44:37.719]   61 (2556) Warning: Possible team chat sync problem was detected

To be honest, I didn't think the two backup job "types" would overlap and both be backing up the Teams mailbox, but not a huge issue there.

I recently patched the servers from the latest ISO (downloaded yesterday) and applied the fix for KB4569 https://www.veeam.com/kb4569

Code: Select all

<Proxy SkipTeamsMessagesDataFolders="True" />

but this is happening after this fix (and was happening before too).

Is the fix for the above issue now Public, or is it still Private (contact Support)?

Thanks.

[Mildur]

Hi Lewis

Not yet.
Please contact support to receive the parameter to fix the issue (after they have confirmed that you are affected by the same issue.

Best,
Fabian

[Lewpy]

Thanks Fabian.
Case #07215788 raised.

[Lewpy]

Support provided modified proxy.xml files with a parameter "fix" in them, and this has resolved the issue

[Mildur]

I'm glad it solved the issue in your case as well.

Best,
Fabian

[mjr.epicfail]

Hi Fabian,

Will this fix be implemented in a Cum. update soon?

[Mildur]

Hi Danny

Do you see the same issue in your environment?
I will ask the responsible RnD team about our plans.

Best,
Fabian

[mjr.epicfail]

Hi Fabian,

Yes I do see them once in an while, for me the solution currently was to isolate the object being backedup in a separate job and run a full on that specific job.
After that the original job runs just fine again (with the affected object)

[Lewpy]

Yes I do see them once in an while, for me the solution currently was to isolate the object being backedup in a separate job and run a full on that specific job.
After that the original job runs just fine again (with the affected object)

We tried this first, but it didn't help: the separate job for the single isolated user (runs as a -Full backup) got stuck in the same loop as the main backup did.
Only after that did we get the modified XML files to try, and they resolved the problem.
Although we haven't now tried to remove the XML fixes to see if the fix made a permanent change, or whether the fix needs to stay in place.

[Mildur]

Hi Danny

I discussed it yesterday with the team.
The key was introduced last year to solve a problem with on some tenants with "legacy team journaling" folders.
Recently some other issues started with processing of inbox folder in public folder mailboxes. The same key works as a workaround.
Our support team reports to us that the issue is not widespread and we therefore we don't plan to change the default configuration. But we are expecting more statistics from support to see if any changes are required.

Best,
Fabian

[mjr.epicfail]

Hi Fabian,

Thanks for discussing it and the feedback back on this forum!
Is there an option to set this per tenant instead of per proxy? Or doesn't that make any sense?

[Polina]

Hi Danny,

The config key can only be applied per proxy. If you would like to use it per tenant, you may consider isolating a specific tenant on a dedicated proxy.

Thanks!

[mjr.epicfail]

Hi all,

Got another issue (within the same tenant).
Is it possible that this issue is adressed in V8 ?