Maintain control of your Microsoft 365 data
edh
Service Provider
Posts: 298
Liked: 92 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Error in deserializing body of reply message for operation 'ExportItems'

Post by edh » 1 person likes this post

Hi,

Before writing this post, I’ve been working with support for several weeks regarding an error that didn’t occur in V7. Case #07440112.
Processing archive mailbox XXX@XXX failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'. There is an error in XML document (1, 54789806). The token '"' was expected but found '<'. Line 1, position 54789806. :: 0:06:55
Supposedly, this issue occurs when trying to process an item larger than 512 MB. Support has suggested reducing the number of items per batch and adjusting the BatchMaxItemSize. However, when reviewing the proxy startup logs, I noticed another key, BatchMaxSaveItemSize, which sets the maximum size to 256 MB.

Reducing the number of items per request increases the total number of global requests for backup, which causes us to more quickly hit the organizational limitations of Microsoft APIs.

This case has been open since September 27 without a resolution, and it is affecting multiple clients who had no issues previously. The problem is no longer minor—customers are beginning to lose trust in Veeam and in us as their Office 365 backup providers.

There’s no SLA or document to reference for this issue. Previously, customers would occasionally check if their Veeam backup was working, trusting that the solution functioned consistently. Now, they check every morning at 7 AM, only to complain about repeated failures, as if we, the providers, aren't trying to fix the issue.

To avoid more serious issues with clients, we’ve had to separate the users experiencing these errors into work tasks, which we label as “Backup Errors.” This at least allows the client to see more green than red on their console.

As a service provider, I would like to request that the R&D team provide a list of known bugs in version V8 and indicate which ones are actively being worked on. I also suggest publishing this information, at the very least, in the service provider forum.
Mildur
Product Manager
Posts: 9848
Liked: 2610 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Processing archive mailbox XXXX completed with warning: Mailbox XXXX is already processed by another job

Post by Mildur » 2 people like this post

Hi Manuel

I'm sorry to hear about the challenges. Let me check with our QA Team if we have some similarity between yours and other cases.
As a service provider, I would like to request that the R&D team provide a list of known bugs in version V8 and indicate which ones are actively being worked on. I also suggest publishing this information, at the very least, in the service provider forum.
I forwarded your request to our core team for Veeam Backup for Microsoft 365.

Update: I found a few similar cases and have asked for an escalation of your case.

Best,
Fabian
Product Management Analyst @ Veeam Software
Mildur
Product Manager
Posts: 9848
Liked: 2610 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Mildur » 1 person likes this post

Hi Manuel

I discussed your case and the similar ones with support management. The other two cases have a slightly different error stack and the cause is not related to your case.

Your case is currently in our Advanced Support level. We can consider escalating the case, but first, we want to ensure that we have tried everything possible. Your support engineer will review the shared action plan from October 3rd again. They will reach out to you after the review. Please apply the reviewed action plan as requested.

Best,
Fabian
Product Management Analyst @ Veeam Software
edh
Service Provider
Posts: 298
Liked: 92 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by edh » 1 person likes this post

Support suggest to downgrade MaxItemSize to 20MB

" BatchMaxItemSize — I propose keeping it like that, as having a single item 20 MB in size is still rather unexpected and may be helpful in case the 'obstacle' batch of 546 MB is processed"

Main issue here is why this happen in V8 and not in V7. And not just a single tenant, in multiples companies we saw the same issue. As far as i understood in the logic, if the request is more than 512MB Veeam should "split" current request into new one to get smaller ItemSize, based into BatchMaxItemSize definition. With this solution we will get an "amplification" request , just for get 512 MB we will need 25 more EWS query. Just do the calcs with a single TB. 2048 vs 51200 Querys..

If those explanation is still valid

Keys explanation (default values):
• BatchSize (100) - Maximum number of items requested from EWS (Exchange Web Services) per one query
• BatchPart (100) – Should be set the same as the BatchSize at this moment
• BatchTimeout (600) - Timeout for getting a reply with results from EWS in seconds
• BatchMaxItemSize (512) - Maximum size of an item (in MB) requested from EWS per one query. If the item size exceeds BatchMaxItemSize, item will be requested separately

As proxy service is not able to "hot" reload the proxy.xml params, everytime encounter this issue we need to enter in maintenance , change the <source> and reload all configs.
Mildur
Product Manager
Posts: 9848
Liked: 2610 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Mildur » 1 person likes this post

Hi Manuel

I currently cannot comment on the specific configuration parameters. I understand your concerns, but we need to start somewhere with the troubleshooting process. Our QA team may request the same configuration changes during an escalation if they find them relevant.

As a intermediate step before doing the configuration change, I recommend updating to our latest patch release of V8 and then checking the job status again: https://www.veeam.com/kb4656.

You might also want to discuss the patch first with the assigned Support Engineer and determine whether the configuration change is still necessary after applying the patch.

Best,
Fabian
Product Management Analyst @ Veeam Software
edh
Service Provider
Posts: 298
Liked: 92 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by edh » 3 people like this post

Thanks Fabian, scheduled patch for tomorrow morning.

KR,
Manuel
Mohamad Raya
Lurker
Posts: 2
Liked: 2 times
Joined: Oct 23, 2024 9:06 pm
Full Name: Mohammad Raya
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Mohamad Raya » 1 person likes this post

Hello,

Did the patch fixed the problem ?
Please advise finding a solution for this issue if available

Thank you;
Mohammad
Mildur
Product Manager
Posts: 9848
Liked: 2610 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Mildur » 1 person likes this post

Hi Mohammad,

The patch didn't solve the problem. Manuel's case is now with R&D to verify the suggested configuration parameters.
If you encounter the same issue, please open your own case and provide me with the case number. Each case could help us better understand the root cause of this issue.

Best regards,
Fabian
Product Management Analyst @ Veeam Software
Mohamad Raya
Lurker
Posts: 2
Liked: 2 times
Joined: Oct 23, 2024 9:06 pm
Full Name: Mohammad Raya
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Mohamad Raya » 1 person likes this post

Hi Fabian,

Yes we are facing the same issue in veeam backup for Microsoft 365 VBO while running incremental backup (we didn’t face it on full backups). Several mailbox returned the same error message.
Please keep us updated when you get a clear solution from veeam.
Meanwhile, we will also open a support case with veeam and provide the case ID.

Thank you;
Mohammad;
aeph
Enthusiast
Posts: 60
Liked: 4 times
Joined: Sep 26, 2024 11:02 am
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by aeph » 1 person likes this post

@Mildur, @Mohamad Raya i am facing the same error with a mailbox. (CaseID 07478215)
edh
Service Provider
Posts: 298
Liked: 92 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by edh » 1 person likes this post

We're still with the problem, we got a reply from R&D but need to test tomorrow morning when proxys finish the night backups... But in case the param needed solve the issue i think this logic should be implemented into VBO proxy.
aeph
Enthusiast
Posts: 60
Liked: 4 times
Joined: Sep 26, 2024 11:02 am
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by aeph » 1 person likes this post

@edh We got the reply to play around with:

<Source BatchMaxItemSize="100" />

Haven't tried it yet because jobs are running.

Did you get a new/ different answer? It seems that you have already played with the BatchSize and had no success. Is there a new fix/ workaround?
gabrieldd
Service Provider
Posts: 21
Liked: 5 times
Joined: May 15, 2024 7:46 pm
Full Name: Gabriel Doring
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by gabrieldd » 1 person likes this post

Hello, I'm getting the same error on one mailbox. Do we have any feedback with tests regarding this?

Case #07481580
gabrieldd
Service Provider
Posts: 21
Liked: 5 times
Joined: May 15, 2024 7:46 pm
Full Name: Gabriel Doring
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by gabrieldd » 1 person likes this post

edh wrote: Oct 28, 2024 2:24 am We're still with the problem, we got a reply from R&D but need to test tomorrow morning when proxys finish the night backups... But in case the param needed solve the issue i think this logic should be implemented into VBO proxy.
Have you had any return from Veeam Support or any success into fixing this?
edh
Service Provider
Posts: 298
Liked: 92 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by edh » 1 person likes this post

Hi,

We still not able to apply to our proxies the proposed proxy.xml changes. We will try this monday. Will keep informed.

Regards
domonkos.zsolt_GBL
Lurker
Posts: 1
Liked: 1 time
Joined: Nov 06, 2024 9:08 am
Full Name: Zsolt Domonkos
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by domonkos.zsolt_GBL » 1 person likes this post

Hi,

Can anyone provide an example of exactly where, in which file, and how these modifications can be made? After reading the post, it’s not entirely clear to me. Thank you.
edh
Service Provider
Posts: 298
Liked: 92 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by edh » 1 person likes this post

Hi ,

Changes proposed dont work... I prefer not share the proposed solution, to prevent more issues in your installations.
Polina
Veeam Software
Posts: 3212
Liked: 779 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Polina » 1 person likes this post

Hi All,

Please do not apply any configuration keys if our support engineers have not directly recommended you do that. It may cause additional complexity in case troubleshooting. I already asked the team to involve RnD and set up another round of issue research. Please stay in contact with our support team.
gabrieldd
Service Provider
Posts: 21
Liked: 5 times
Joined: May 15, 2024 7:46 pm
Full Name: Gabriel Doring
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by gabrieldd » 1 person likes this post

edh wrote: Nov 07, 2024 8:01 am Hi ,

Changes proposed dont work... I prefer not share the proposed solution, to prevent more issues in your installations.
Can confirm this. So far nothing has changed.
I hope Veeam finds a solution soon...
neilp
Influencer
Posts: 19
Liked: 3 times
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by neilp »

Im having the same issue.
This seems to happen with version 8 but not my version 7 server.
Polina
Veeam Software
Posts: 3212
Liked: 779 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Polina » 1 person likes this post

Dear All, when you report a similar issue with v8, do not forget to add your support case ID to the post. It will help us to track and follow up on the issue properly. Simple '+1' posts are not that helpful.
neilp
Influencer
Posts: 19
Liked: 3 times
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by neilp »

Veeam Support - Case # 07500815

Error: Processing archive mailbox XXX failed with error: Mail item data export failed. Error in deserializing body of reply message for operation 'ExportItems'.. There is an error in XML document"

We tried isolating the failing mailbox with single job targeting that mailbox (then running a fullbackup) but that archive mailbox fails.
neilp
Influencer
Posts: 19
Liked: 3 times
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by neilp »

Hello is there an update to this? We are having the same issue.
LeeMackie
Service Provider
Posts: 25
Liked: 7 times
Joined: May 25, 2022 12:37 am
Full Name: Lee Mackie
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by LeeMackie » 1 person likes this post

Receiving the same issue with backing up an archive mailbox.

Case ID 07504728
neilp
Influencer
Posts: 19
Liked: 3 times
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by neilp »

There was a new firmware released yesterday. 8.0.5.20 (2024-11-14). I patched my veeam for office 365 server but it didn't fix our issues. Hopefully Veeam R/D is working on a fix for this.
Polina
Veeam Software
Posts: 3212
Liked: 779 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by Polina » 2 people like this post

RnD is researching the problem; the solution is not available yet.
neilp
Influencer
Posts: 19
Liked: 3 times
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by neilp »

Thank you. Were really anxious on getting a fix for this.
neilp
Influencer
Posts: 19
Liked: 3 times
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by neilp » 1 person likes this post

Quick update.

We are able to backup the Archive mailbox now. Nothing was changes on our end, except I installed this month's windows updates on the backup server.

Perhaps this months windows updates fixed this issue for us or maybe something in the backup changed ( MS Office 365).

Ill report back if we continue to have the XML error.
LeeMackie
Service Provider
Posts: 25
Liked: 7 times
Joined: May 25, 2022 12:37 am
Full Name: Lee Mackie
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by LeeMackie » 1 person likes this post

Mine has also self-corrected with no changes.
mij4sure
Lurker
Posts: 1
Liked: 1 time
Joined: Jul 27, 2017 7:42 am
Full Name: Johannes Mittelböck
Contact:

Re: Error in deserializing body of reply message for operation 'ExportItems'

Post by mij4sure » 1 person likes this post

After Veeam Call. on VBO 8 we added in Proxy.xml we added on the top.
We have already added these lines in advance but in the wrong place in the Proxy.xml

--- edited by moderator ---


Attention corrupt files are also skipped. :D
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 6 guests