M365 Mailbox Backup increase since friday / latest August MS Patch installation

[bielekovve]

next day, next pain... We're saving TB of exchange "rubbish" data twice to our onprem S3... >6TB the last two days... 50 exchange jobs still running... When does it stop??? *HELP*

Case #07210033

[f.neumann]

During the last few weeks we're seeing noticably higher growth on incremetal backup-size across multiple different customer environments.

Is there a way to figure out where this is coming from?
Currently with one specific customer (CaseID #07211942) we're seeing longer processing time (around 1 hour) for 3 mailboxes, resulting in increment-size of 20 GB within 1-hour timeframe.
But there were only around 70 mails in total within 24h. And those 3 mailboxes are only 25 GB size in total.

We're using snapshot-based retention on repository.

We nearly can't keep up extending disk-space on our repository-disk due to this unpredictable growth. And as there's no real change within mailbox-data we do not want to waste disk-space.
Anyone facing the same issue? Any known changes on microsoft side that might result in veeam backing up same mailbox-items multiple times?

[Mildur]

Hi Florian

This is a known issue.
Most likely some attributes on the items changed in Exchange Online. When this happens, we see those items as changed and have to download it again. It is currently investigated by us and Microsoft engineers.
Please see the KB: https://www.veeam.com/kb4483

Best,
Fabian

[agrob]

We have the same issue starting around end of March. normally the Incremental Backup is arround 10-15GB per Day. Since then i see 100GB or 200GB or more per Day. It is unbelivable that it seems that the same issue happens again. MS should pay us addition backup storage...

[EVeloso]

We started with the same issue from March 26th. Randomly, a user's mailbox downloads 3 or 4GB every time the job run.
Already updated to latest version: 7.1.0.1501 P20240123

[replicatius]

Unfortunately we're also seeing the same behavior, back again it seems. Extremely frustrating as we're scrambling to extend storage to accommodate.

[dotdk]

Unfortunately, we're also seeing the same behavior, currently two of our customers are affected. One of them has their weekly backup data amount increased from ~25 TB to ~60 TB. Been going on for two weeks now.

[dbeerts]

We see the same behavior for some tenants..
Any advice what to do now? Last time the only option was to let the backup run and protect the 'new items', I assume it's the same for now?

[Butha]

Hi all,

Followed the manual KB (edit config file) a few days ago and it resolved the warning and backups started running fine again, but it's caused a new issue I have not seen mentioned here.

It seems the exchange backup job is reprocessing our complete mailbox + archive stores from scratch - jobs running many days and TB's of exchange data already backed up. (not managed to get through a complete exchange backup yet.) Anybody else seen this?

EMEA region - also affected by recent event from Microsoft (https://azure.status.microsoft/status/h ... d=VT60-RPZ) - so not sure if it's related to our MS region only.

# 07217383

[Mildur]

Hi @Butha

I moved your comment from the other topic.

The configuration change from the IPM error is not responsible for this new situation.
Please check this topic and our KB: https://www.veeam.com/kb4483

Our teams are working on it with Microsoft Engineers.
But there is not much we can do for now to stop "protecting already protected data again".
The items in the mailboxes changed somehow in Exchange Online.
Our backup job will see those items as changed and start downloading them again. Unrelated to the IPM error from the hidden teams message folder.

Best,
Fabian

[Butha]

Thank Mildur - apologies - I missed this post, saw it was merged into the correct post now.

Had a quick look through previous jobs - started on the 30th March for us. Maybe the Incident listed above (azure VT60-RPZ) has caused some metadata changes in the background that could be the cause? The outage affected a number of Azure regions. (Might help the MS engineers who you are dealing with)

quick sizes for reference -

March
30th - 1TB
April
1st 960GB
3rd - IPM issue start so failed - but processed 460GB
4th - IPM
5th - IPM fix applied - 615GB
6th - 300GB interrupted - window exceeded
7th -350GB interrupted - window exceeded
8th - 400GB - etc
9th - 400GB - etc
10th - in process

So issue is real - and good luck getting MS to refund for the TB's of storage this adds to the bill...

[Steve-nIP]

This is wild. Having jobs that suddenly take forever and consume endless space is unsustainable. I get it's not Veeam's fault as such, but there needs to be some workaround or fix.

[tsanfilipp]

I'm beginning to wonder if this is a covert way for MS to begin pushing their own backup process, over 3rd party solutions.

[WolfgangKr]

As our backup is now disabled for several days, is anyone else thinking about changing retention settings on M365 side to not loose any data?

We are discussing to enable the "retention hold" setting for our mailboxes.

[mikame]

Got again an email again about disk space full, can't extend the HD forever. I think I will just delete the exchange backups repository/job and start from scratch, luckily I have only about a TB of exchange data and I have offline backup on USB device.

[dbeerts]

Any update on this? Are objects still being changed by MS?

[NN25]

Same issue also, is there any update regarding this, or some due date when this will finished?

[doublem]

Also seeing this increase in time and data since 26 of March. Case #07223474 .

[Polina]

Hi All,

From the updates that I receive, the spike of changes seems to decrease already. However, I cannot guarantee that the issue has passed already - like any other updates, these changes are rolled out across M365 region by region.

[sdv]

@Veeam, Thanks for the update.

Reference Veeam case 07222939 we got pointed to the KB article, which related to this topic.
Starting around 27 march we see the same behavior, last night (15/04) was the worst because we were transfering atleast 1 TB changed data of mailbox archive data, regular change rate is around 10 Gb, since march 27th around 100-250 Gb. Hence decided to disable the backup job mailbox archives per 16 april (today) until further notice by either MS or Veeam.

It feels like Microsoft is changing a lot lately around Exchange in the cloud, last night we also received an increase of Exchange related incidents (365 alerts). Last week specific incident(s) were not communicated in time, until a few days later.

For non archives (mailboxes) it look like it calm down a bit compared to other days, however it is not yet back to normal change-rate.
Fingers crossed.

[m.novelli]

Actually I have two customers severely impacted, with backup server with disk full

I'm thinking to delete the repository and start again backups

Marco

[MaartenA]

last night was also our worst night yet. We have already had a 50TB growth since this problem started. I hope it will stop now. EMEA region

[VNR-LF]

we also had a huge spike last night. for some mailboxes, up to 60% of total mailbox size has been backed up.

[m.novelli]

50 TB!!!

Marco

[nathano]

Also seeing huge spikes in usage today.

[piotr.olejarz]

unbelievable that such a serious company treats customers this way. The problem has also been with us since March 27. Data increment of several hundred percent. Created ticket in the ticket system. Let's report as much trouble as possible on this issue - maybe they will finally do something.

[reanimat0r]

Same here. We notice huge spikes in storage consumption since March 27, 2024.
Veeam support blames Microsoft and says there is no way to reclaim the space.
Additional effect is that backup jobs utilize a lot of RAM now and sometimes end with "out of memory" error.

[m.novelli]

So far I have deleted all backups for two customers and I'm starting from scratch to reclaim space

Marco

[Polina]

Hi All,

Please be sure that despite the lack of responses in this thread, our RnD, support, and management teams are working closely with Microsoft to find ways to prevent similar issues in the future. Mitigating consequences for those who are already hit by the issue is another big challenge - we're now estimating possible ways and potential risks. This will take some more time, and I hope for your understanding.

[dotdk]

Case #07216030

We made an overview of one of our customers weekly amount of data being backed up pr week, and we saw the increase starting from week 13, where we normally added approximately 25 TB pr week to their repositories. Even since amount has increased rapidly topping last week with 83 TB additional data. Almost 60 TB more than on a normal week.

| Week Number | Weekly diff |
| ----------- | ----------- |
| 15 | + 83 TB
| 14 | + 63 TB
| 13 | + 56 TB
| 12 | + 27 TB
| 11 | + 25 TB
| 10 | + 26 TB
| ----------- | ----------- |

We saw the same issue on another customer but, they are back to normal and have been for roughly 10 days, so the increased amount of data problem will stop at some point.

We have had some really good help from Veeam and raised tickets with Microsoft with all the needed information from Veeam.