New Mailbox backup Issue?

[timmyvergauwen]

Same issue here since yesterday evening. Logged a case at Veeam support.

[sga]

Hi,
since the 18th of january, we have the same issue on multiple accounts of our M365 tenant. Some mailboxes are backuped fine, some fail with the mentioned description (Processing <XXX> failed with error: Unable to process the Exchange Web Services item.. Invalid serialized stream, unknown opcode found: 4.
Case #07096663

Regards,
SGA

[itteamaszende]

Same problem from yesterday night, in one our tenants.

Support ticket opened: #03025681

Thanks

[timmyvergauwen]

Got this message from support.

According to the details provided, it seems we are facing a known issue which our RnD team is currently investigating.
Unfortunately, there is no ETA on further feedback but my colleagues and I will work together and we will strive for an update as soon as we’re able to, at which point I will reach out with next steps.

Thanks a lot for your understanding and cooperation.
Should you have anything to clarify or add in the meantime, just let me know.

[Artero]

Same issue.

18 mailboxes, 13 ok 5 ko

[JKring]

Noticed this on one of our tenants this morning.
Opened case #07096807 to get updates on this.

[Cetrom_George]

Case #07096945
Seeing the same issue on random organizations and Mailboxes

[sga]

Referring to my opened case I got an email with the advice to activate extended logs in VBO365 and download a third Party tool called fiddler. After installation and configuration for https logging, the support wanted me to upload the recorded session and the logs to the case.

We'll see.

[ks.any.cloud]

I have also noticed this for at least 1 customer. failed with error: Unable to process the Exchange Web Services item.. Invalid serialized stream, unknown opcode found.

[Polina]

Hi All,

RnD is investigating the issue atm. Typically, Microsoft rolls out changes gradually across different regions and data centers, and this seems to be the case this time as well. As of now, our QA team cannot isolate and confirm the issue in our own labs, and your active cooperation in providing logs and fiddler traces would be much appreciated. This will help our RnD to faster isolate and resolve the issue.

Thanks!

[brainerdmobil]

Same issue in our environment. Started last night with a few mailboxes. Tried running the job again this morning manually and many more are failing now. At this point I'm not opening a ticket as is seems several others already have. I'll keep monitoring this discussion for a potential fix.

[DW_AT]

Case #07096436
same error here - 9 out of 22 permanently failing ... have to do a migration this weekend and need an functional MS365-backup ... frustrating

[ACrispiels]

Same error for me since this morning. Just for one organization until right now. Just some mailboxes within that organization.
Server is up-to-dated and has been already rebooted.
Veeam 365 also up-to-dated (latest release 7.1.0 1401).

[utleyab]

I am seeing the same issue across multiple organizations and multiple tenants

[Perceptum]

Case #07097431
This is happening to me as well. We take hourly backups and the number of failed mailboxes has increased from 3 to 13 in less then 24 hours.
Running version 7.1.0.1301
I backup mail and sites, but only mail is affected.

[DW_AT]

just checked for updates and 7.1.0.1301 was upgraded to 7.1.0.1401 -
still receiving the same error on the same 9 mailboxes as before ...

[chtn]

I haven't opened a ticket as this now seems a well known issue. My backups started failing at 7PM EST yesterday evening on three mailboxes, and has slowly increased to now 50 out of 112 mailboxes failing. I am still on version 5.0.3.1033, so I'm way out of date (new Veeam server VM is on my to do list). Just adding this info as it seems to more clearly point to something Microsoft has changed. I'll just watch this discussion for now.

[kavaa]

Issue is still ongoing. This Morning only about 4-5 jobs that had issues now over 30. Veeam support is telling us that its a Microsoft issue…

[stegostoreus]

We just got this tonight. Retry fails as well.
Case # 07097682

[DW_AT]

Case #07096436
right now we see increasing the issue on more mailboxes of the same tenant and also more tenants starting to fail

[cpfleger]

Same problem here since January 19th (VBM365 7.0.0.4388 P20231015)
Mailboxes are in the same region, Windows update (including .NET) did NOT solve the problem

Update:
updated VBM365 server to 7.1.0.1301, no difference, same error ("Processing mailbox xxx@xxx.xxx failed with error: Unable to process the Exchange Web Services item.. Invalid serialized stream, unknown opcode found: 4")

Regards!

[FrancWest]

Same here in my lab, which only has 1 mailbox. In production with 150+ mailboxes we don’t have the issue.

[FrancWest]

Also, the job end result is ‘warning’, but the mailbox backup task has status ‘error’. Shouldn't the job result then also be ‘error’?

[cpfleger]

Update 2: updating to 7.1.0.1401 did also not solve the issue.

Regards!

[DW_AT]

Case #07096436
received private-hotfix as by now, applied it and it looks good so far

[johntbbp]

Upgrade from 1301 to 1401 and Hotfix resolved it for me.

[Pricepd67]

Updated to 1401 and I'm still having the issue on a 13 of 50 maiboxes.

[Polina]

Hi All,

The hotfix has been delivered to the support team (as you can also see from the above comments).
Please note, that upgrading to 1401 will not resolve the issue - you also need to install the hotfix on top of it.

[ProfileX]

So we can't resolve this ourselves, we have to open a support ticket?

[Gostev]

That is correct. The hotfix has literally been just produced and will be in controlled distribution by support for the time being.