M365 Mailbox Backup increase since friday / latest August MS Patch installation

[agrob]

Good Day

No news from support other than "we are working on it with Microsoft". It is now arround 14 Days since our last backup because of the huge data increase we stopped the backup. Is there any ETA until we get a final feedback how we have to proceed? i don't can run the backup and have all the data backuped again! It would result in several TB of additional and not needed backup data because of the fault MS did. Not acceptable.
Thanks for any update

Best Regards

[tosis]

My team raised case 02860734 on 2023-08-22 after a single session running more than 7 days was cancelled at around the 30% mark, and further sessions faired no better. We're still trying to get past this for one tenancy where after 96 we've only processed 19% of discovered objects. It's now been 3 weeks since this fiasco started.

My understanding is that the issue is with EXO items having been bulk-modified. If my understanding is accurate, should we not be considering excluding EXO from job configurations so that we can at least attempt capturing SPO and Teams data (yes, I recognise Teams utilises Exchange)?

[agrob]

When i remember correctly, it is suggested to keep same data in the same repo. in that case, i would configure it like this:

Mailbox Job (Exchange Online) > to Exchange Repo
Sharepoint Job > to Sharepoint/Teams Repo
Teams Job > to Sharepoint/Teams Repo

I have this config and Sharepoint and Teams Backups are running fine.

If you like to have one repo for all Jobs, i would at least create seperate jobs for each Data type. that means, create an Exchange Job in which you only backup Mailbox Data. Create another job for Sharepoint in which you only backup Sharepoint Data....

[tosis]

I note that KB4483 was updated on Friday last week - issue resolved; the changes are valid; get on with your backups.

In addition to what we've been discussing here, one of our team found the following Veeam Community blog post describing how to temporarily relax throttling on Exchange Online Web Services (EWS)
https://community.veeam.com/blogs-and-p ... ttling-720

We'll implement this for the remaining problem tenancy today and hopefully it will ease the burden following this event.

[agrob]

That means, we as customer has to deal with the fault from microsoft and have to spend more money on backup repos as this change from MS does result in serveral TB of data which we have to write to our Backup repo again (even if the data was already protectet).

[Mike Resseler]

Hey @@agrob

Unfortunately... yes. The worst part is that we can not identify what has changed because of this fault, unless we take a risk of potentially removing data that is actually removed. And even if we would do that, we would get the next backup the same data again (as we see it differently compared to the previous backup).

[agrob]

Thanks Mike, I understand. For me it seems like an MS fault which we have to life with (once again). Thanks for your support and clarification

[AlexHeylin]

Thanks Polina.
Any news from MS / R&D in the last two weeks please?

[Mike Resseler]

@AlexHeylin We know now that the situation won't appear again. What caused this has been changed so it should not appear again. For the official statement, we are finalizing that one (sorry about that)

[AlexHeylin]

Thanks Mike. So it's safe for those who've suspended their backups to resume?
We understand that the official blame<<<<<statement will take some time.

[Gostev]

Those who were affected, could you check if you see the official statement from Microsoft about the issue in the M365 Health Center? I'm hearing it was posted there back on Tuesday.

[agrob]

Haven't seen any information about that in our M365 health center. Does someone know what the official Title or ID of the message is? I'm curious to read MS official statement about the issue. thanks

[agrob]

Thanks Mike. So it's safe for those who've suspended their backups to resume?
We understand that the official blame<<<<<statement will take some time.

I started our backup job again on Monday. Job is still running (should be soon over) and transfered again arround 1.7TB of data. just make sure you have enough space on your repo
I'm curious if the next run will go back to normal regarding the transfered data

[AlexHeylin]

I'm curious to read MS official statement about the issue.

"We fu<<broke it by accident. Soz"

[Mike Resseler]

Hi guys,

First, our apologies that it took so long to figure everything out. It hasn't been my easiest ride to get confirmation of the root cause (at least written).

For those that were affected, they should get a message in their M365 Health Center which is:
Title: Some users are seeing an influx of older email messages unexpectedly resurface within their Exchange Online Mailbox
The message number: EX670041
Root cause: An update made to the Exchange Life Cycle (ELC) service caused a regression that resulted in some older email messages being misclassified and resulted in their resurfacing within user's Exchange Online mailboxes

For those who have halted backups: unfortunately, it cannot be undone. Those messages have a changed status which means we see them as different and our solution will back them up again. After that, operations should go forward as normal.

If you are suffering from this, and haven't seen that message in message center, then please contact MSFT support with the above reference to get confirmation (if you want to of course...)

Hope it helps
Thanks for everyone's patience
Mike

[AlexHeylin]

Thanks Mike & team

[RossAtCC]

Folks - neither the Veeam nor MS notices on this explain what this actually means in practice. Can anyone from Veeam advise?

For example, tenant who we've been backing up for years, with storage usage consistently around 50GB, over the course of 4 days it has increased to 300GB.

So are we backing up duplicated data as a result of this "re-index", or is there data that was being omitted from backups before?

In case (1) we can probably reset their backups and bring their costs back down, in case (2), how the heck do you explain that to hundreds of clients? "You're monthly bill just went up by 400%, also we weren't properly backing up all your data".

Which is it?

[Gostev]

Effectively, it's duplicated data. Many objects that did not actually change, were erroneously marked as changed by M365, and thus had to be copied into a backup repository by Veeam. But because in reality their content did not change, they are effectively duplicates.

[agrob]

Thanks to the veeam team!

[RobM_Dutch]

Final statement from Veeam

"Working closely with the Microsoft Engineering Group we managed to find the root cause of the issue and to present the final statement.
Microsoft has confirmed that there was a limited incident on August 7th – August 21st where some Microsoft Exchange Online environments unexpectedly had older emails being resurfaced, resulting in excess email data in some Exchange Online mailboxes that should not have been, resulting in the situation we’ve observed. This issue has been confirmed by Microsoft and some organizations with affected mailboxes will see the message EX6700041 in the Microsoft 365 Admin Center.
Unfortunately, as the items are considered 'changed' due to the modifications, there is no way to reclaim the space used by the backup of the resurfaced emails, and the only way forward is to let the jobs complete.
This situation is discussed in further detail on our forums at veeam-backup-for-microsoft-365-f47/m365 ... 12-30.html"
"

My reaction to this:

This is just an explanation then, but no solution. Thanks for that. We and all other customers will have to deal with a los in storage space for years to come. Because of the retention period.
This is very, very bad.
What i would like to know is how Veeam is going to prevent issues like this in the future? We can’t "solve" this issue but we need to set everything in action to prevent it, going forward. Veeam and Microsoft need to learn from there mistakes.
And, once again, i’m not at all happy with the outcome. And i believe i’m not the only one.

[AlexHeylin]

RobM_Dutch - this is something you really need to blame MS for and robustly discuss with your MS rep. Veeam's software did exactly as it was supposed to when presented with modified data. Veeam didn't modify the data - MS did. This will have affected other O365 backup products too - not just Veeam.

[Mike Resseler]

Hi @RobM_Dutch

I am not disagreeing with you that it is not a fun place to be. You are right that there are quite some items that will be stored now double for as long as the retention is there. But unfortunately, there is not really something we can do about it. Some of these items have had a changed modified date, and (at the same time) a changed metadata item. So Veeam needs to see this as a changed item, which means we need to protect it again. That's what the solution supposed to do. Protect the changed data.

Even if we would want to prevent something like this in the future, we still won't be able to do so I'm afraid. Because it will always be seen by us as a new item. Internally we did an investigation to see if we could remove the obsolete data. But unfortunately, that was not possible without the risk of breaking things and having those specific items not restorable anymore when you need them.

I'm aware that this is very inconvenient for you, just like it is for us internally (and with our larger service providers). I do apologize for this annoyance, but I'm afraid our hands our tied here and our software did exactly what is was supposed to do. Save the changed data.

Feel free to ping me personally if you want to have more details or a chat on this
Cheers,
Mike

[RossAtCC]

We've just done our billing run for August, and are now dealing with the customer fallout.

The official explanation doesn't cut it, although I appreciate this is not a Veeam fault, as said above, there really needs to be a fix deployed by Veeam, although I have no idea how this might work. Can Veeam deploy a fix to be run on demand which can dedupe and delete duplicated objects by checking other fields and comparing the backup date to determine likely duplicate objects and giving us the ability to remove them?

The magnitude of the increase in data usage makes no sense. I have a one-man band client who's backup went from 3GB to 23GB. Other example, 70GB to 620TB, 1TB to 6TB.

If data was duplicated, I could understand a 100% increase in storage use, but this is ridiculous.

We're now looking at deleting / reseeding loads of backups, and we're crediting angry customers for additional storage costs.

[Gostev]

If some custom script on their side touched and updated all items in all mailboxes, would they still be upset with you? Probably not? They would be upset about the person who created and ran the script, right?

But then in this case, when same script on Microsoft side did touch and update all items, why are they upset with you (and not Microsoft)?

[dasfliege]

We also just had to do the billing for our customers and decided to don't charge them for MS faults. We just charged last months storage consumption. For the upcoming billing periods, we consider charging the actual volume minus the amount that has been added between 11. and 15. august. We "only" have 40 customers for which we do M365 backup, so a manual calculation is possible. But i don't want to know how bigger service providers are dealing with that.

Just one thing i don't really understand: We have a few customers running Avepoint for M365 backup. They did not run into this problem. Neither did their backups took any longer, nor did they had an increase in backup volume. How is that possible? If MS changed attributes that are essential to uniqly identifiy an object, this should affect all backup solutions then?

[Gostev]

Because relatively few M365 orgs got impacted overall. Even among Veeam customers, and we protects more M365 orgs than any other backup vendor worldwide.

Luckily, whenever Microsoft rolls any change out, they always do it in stages, starting from a very small scope.

[m.novelli]

If some custom script on their side touched and updated all items in all mailboxes, would they still be upset with you? Probably not? They would be upset about the person who created and ran the script, right?

But then in this case, when same script on Microsoft side did touch and update all items, why are they upset with you (and not Microsoft)?

This stupid answer show why modern IT is broken

For customers it's impossible to reach any physical person in big IT company with power or willing to really help

Customers can just reach and blame the distributor / Partner that is the black hole for all complains and need to work 10x to manage issues out of his control

Marco

[BaptisteT]

Because relatively few M365 orgs got impacted overall. Even among Veeam customers, and we protects more M365 orgs than any other backup vendor worldwide.

Luckily, whenever Microsoft rolls any change out, they always do it in stages, starting from a very small scope.

I can confirm that in France, none of my providers reported me any issues

[talltim]

This stupid answer show why modern IT is broken

For customers it's impossible to reach any physical person in big IT company with power or willing to really help

Customers can just reach and blame the distributor / Partner that is the black hole for all complains and need to work 10x to manage issues out of his control

Marco

I think you are being unfair there. You are literally replying to a physical person, who is high up in Veeam. However it is not Veeam who changed anything, it is MS.
You might have a point about finding someone in MS who will answer you tho!

[AlexHeylin]

This stupid answer show why modern IT is broken
For customers it's impossible to reach any physical person in big IT company with power or willing to really help

You just replied to THE physical person in Veeam who has the power and is willing to help - and would get this changed if it was practical to do so.
Look at Gostev's job title near his name in the right column .

Also - having spoken with Gostev a lot, both on here and in person, I'm completely sure that if it was practical to safely improve this situation from the Veeam side Gostev would have had them do that. He has the power, he's had them look at it and he's said it's not safely practical, and I don't think he'd misrepresent that.

You're angry with the wrong person in the wrong vendor. Take your frustrations out on Microsoft who changed the data. Good luck getting hold of anyone remotely as senior as Gostev in MS. This is why we love the Veeam R&D forums - the people who know the product and have real power to make real change hang out here and listen to us.