Fails with an error in XML document

[cdlane]

Please find the error as discussed with Mike Resseler at https://www.veeam.com/blog/how-backup-o ... email.html

26/09/2016 13:07:13 :: There is an error in XML document (1, 31559).. '', hexadecimal value 0x08, is an invalid character. Line 1, position 31559.

It ran for 8 minutes and 50 seconds before failing.

Best regards,

Chris

[bfoley]

I set it up to backup all the mailboxes in my tenant. It fails within 12 seconds with the error "There is an error in XML document (1, 15712).. '', hexadecimal value 0x08, is an invalid character. Line 1, position 15712."

Here is the error from the log:

Code: Select all

9/26/2016 1:31:50 PM   22 (43192) Preparing folders for items processing...
9/26/2016 1:31:51 PM   20 (42840) Error: There is an error in XML document (1, 15712).
9/26/2016 1:31:51 PM   20 (42840) Stack:
9/26/2016 1:31:51 PM   20 (42840)    at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at EWServices.ExchangeServiceBinding.FindFolder(FindFolderType FindFolder1)
   at Veeam.Ews.ExFolder.<GetFolders>d__21.MoveNext()
   at Veeam.Ews.ExFolderWalker.Walk(IFolderVisitor visitor)
   at Veeam.Archiver.Source.MailboxActivator.PrepareFolders(IMailboxContext bind, ExcludedFolders excluded, CancellationToken cancel)
   at Veeam.Archiver.Source.MailboxActivator.DoWork(CancellationToken cancel)
   at Veeam.Archiver.Source.WorkAction.WorkWrap(CancellationToken cancel)
9/26/2016 1:31:51 PM   20 (42840) Error: '', hexadecimal value 0x08, is an invalid character. Line 1, position 15712.
9/26/2016 1:31:51 PM   20 (42840) Stack:
9/26/2016 1:31:51 PM   20 (42840)    at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
   at System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32 startPos, Boolean expand, StringBuilder internalSubsetBuilder, Int32& charCount, EntityType& entityType)
   at System.Xml.XmlTextReaderImpl.ParseCharRefInline(Int32 startPos, Int32& charCount, EntityType& entityType)
   at System.Xml.XmlTextReaderImpl.ParseText(Int32& startPos, Int32& endPos, Int32& outOrChars)
   at System.Xml.XmlTextReaderImpl.ParseText()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlReader.ReadElementString()
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read85_ExtendedPropertyType(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read384_FolderType(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read448_FindFolderParentType(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read449_FindFolderResponseMessageType(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read454_ArrayOfResponseMessagesType(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read501_FindFolderResponseType(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderExchangeServiceBinding.Read925_FindFolderResponse()
   at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer347.Deserialize(XmlSerializationReader reader)
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
9/26/2016 1:31:52 PM   21 (44992) Job processing failed

[PTide]

Chris, Brian

This has to be investigated by support team, please contact them and post your case IDs

Thank you

[bfoley]

I tried. Since it is not a supported product, they would not create a case ID.

[PTide]

Sorry, my bad. Reported directly to QA team.

Thanks

[Kostya]

I tried. Since it is not a supported product, they would not create a case ID.

Brian, could you please upload logs to FTP? I've sent you the FTP link via PM.

[Kostya]

26/09/2016 13:07:13 :: There is an error in XML document (1, 31559).. '', hexadecimal value 0x08, is an invalid character. Line 1, position 31559.
It ran for 8 minutes and 50 seconds before failing.

Chris, could you please upload logs to FTP? I've sent you the FTP link via PM.

[cdlane]

Thanks for the PM but I have exactly the same error with the fix.

[Kostya]

Could you please upload new logs with the fix installed?

[Kostya]

Experimental fix provided via PM.

[bfoley]

I can confirm with the last private fix, this problem seems to be corrected.

I was able to successfully backup 122 mailboxes in 1 day, 13 hours. A total of 477.53 GB (2191298 items). Processing rate was 3.60 MB/s, read rate 95.50 KB/s, write rate 8.65 MB/s.

I should mention that it might have been faster, however we currently have another service running that does Office 365 backups (from CodeTwo), which I did not shutdown (it runs on another server though, actually it runs on my Veeam backup server).

[jchillman]

Have you run an incremental backup yet? I would be curious as to how this compares.

[bfoley]

Incrementals are MUCH slower. Mainly because it has to read through every mailbox and look for changed items. On 122 mailboxes it's running for about an hour and a half for each incremental run (I think it's set to rerun every 5 minutes) at a processing rate of roughly 60 KB/s.

[Kostya]

Brian thanks for letting us know. So, it takes 37 hours for initial backup and 1,5 hours for incremental in your case, which is pretty good.

[cdlane]

HI All,

The latest private fix resolved my XML and my network share repository errors in B&R.

Initial Backup - 79 mailboxes in 1 day, 1 hours. A total of 320.47 GB (1,152,493 items). Processing rate was 3.55 MB/s, read rate 2.86KB/s, write rate 11.08 MB/s.
Latest Incremental - 81 mailboxes in 24 minutes. A total of 187.09 MB (3,721 items). Processing rate was 94.11 KB/s, read rate 719.77KB/s, write rate 9.74 MB/s.

All to a Buffalo NAS for testing purposes.

Best regards,

Chris

[Mike Resseler]

Thanks Chris for the details!

Very much appreciated

Mike

[foggy]

The fix for this and several other issues reported for Veeam Backup for Office 365 beta 1 is available here.

[mlsl]

We are having the same issue. Not able to backup and mailboxes with the stated XML error. Where can we get the patch?

[Vitaliy S.]

Hi Mark, did you try to click on the link provided in the post above?

[mlsl]

Yes, Noticed that after the fact.

Thanks,

[Kostya]

It is solved in the release candidate. You can find all the information in this thread.

[craig.etienne]

I am having the exact same issues and was wondering if the fix could be shared with me via PM so i can test please.

[Vitaliy S.]

What version of Veeam Backup for O365 are you currently using?

[craig.etienne]

Version 1.0.0.912

Menu->About

[Mike Resseler]

That would be with update 1 so please log a support call.

Thanks
Mike

[craig.etienne]

Thanks. Already did. Awaiting response.

[Mike Resseler]

Could you update this thread with the case ID please? Thanks!