M365 Mailbox Backup increase since friday / latest August MS Patch installation

[agrob]

Good Day

Friday i installed the latest MS patches on the on Prem Server on which VBM365 is installed. Since then, Mailbox backup job is dramatically increasing. I don't know if it is related to the patches...
Before, each Job run (once a day) was arround 13-15GB of transfered Data (runtime arround 3-4 hours). Friday Job did run 28h and transfered 570GB of Data. The job which started next, was still running today and transfered arround 460GB of data...
I had a look at the logfile from the friday job and saw some mailboxes with 60GB of transfered Data. Or my Mailbox with arround 3.3GB of Data and i'm sure i did not genereate or import mails to my mailbox which would explain those 3.3GB... It looks like something is happening... Did anyone else have this Problem? Dont' want to fill up my Repo because of an issue

Thanks

p.s already opened case with id 06237342

[Mildur]

Hi Andre

I'm not aware of any known bugs. I also checked cases from the last 14 days and couldn't find something similar. Let's give our support team chance to check the logs. Logs should tell us what's happening with the backup jobs.

Best,
Fabian

[agrob]

Hi Fabian

Thanks for you quick reply. I was on Version 7 (3007) and did update now to tha latest build, rebootet the server and startet the mailbox job again. I'll check this changes anything and will wait until i get feedback from support.
Regards

[agrob]

Short Update: Support is investigating the issue. It looks like in the meantime they have some similar cases form other customers....

[Mildur]

Thanks Andre

I saw the case comments.
We (RnD) are checking the case too from our side.

Best,
Fabian

[agrob]

Thanks Fabian. fingers crossed that the issue can be found soon and additional needed storage space can be released again....
Best Regards

[rkn]

Hi all

"Good" to hear that we are not alone. Same issue on our VBM365 Exchange jobs. Data growth started around 11.08. for us. Daily +100GB with ~2'000GB total mailbox size.

We're running the latest VBM365 version on Windows 2022 with July patches. Reboot didn't help. Support case created a few minutes ago.

[Mildur]

Hi RKN

Can you please share your case number?

Best,
Fabian

[agrob]

Short Update: Last Jobrun duration was 48h (normaly 3-4h) and transfered 1.3TB (normaly arround 12-15GB) of data... i'll start a new job run now and hope that it does not transfare this much of data otherwiese our repo is getting full....

Update: Stopped the newly startet jo again because after 15 of 1600 object, already over 3GB are transfered... for me this seems that the job again will backup to much of data...

[RobM_Dutch]

Same here. Was investigating this issue which started 11.08.2023 for us. I'm happy to find this post now, i was really thinking we got some malware or something.
I would also like the case number (or a link) to keep track of this issue (and the possible) solution. R&D has to give this high priority.

I wondering if changes in Defender on the server could cause issue's like this. Going to change some settings to test it. We run Sophos AV on our server. Which AV systems do you all use.... (just guessing a cause).

[rkn]

Case #06241407. Already got a response from Veeam Support:

Kindly be informed that we are seeing this trend and our team is working with Microsoft to identify and resolve the issue.
We will update you as soon as we have any update on this.

[Mildur]

I would also like the case number (or a link) to keep track of this issue (and the possible) solution. R&D has to give this high priority.

What's your case number? If you open a case with our customer support, you will get updated as soon their is a solution to this issue.

Case from Andre is now with RnD.

Best,
Fabian

[RobM_Dutch]

@Mildur: I just opened Case #06241547

[Mildur]

Thank you.
We will check with Microsoft if there was a change recently which may affect our products.
Keep you updated.

Best,
Fabian

[AlexHeylin]

Also our Case #06242901
I hope you find and fix soon, we can't keep adding storage for long.
Thanks.
Alex

[agrob]

Just got an update from Suppprt. Hope it is ok to post this here:

*********
Our development team is still investigating this issue and they have found the root cause.

They compared the same items at different restore points (before and after the issue occurred) and found that the fields “PR_INTERNET_ARTICLE_NUMBER” and “PR_LAST_MODIFICATION_TIME” were changed.

According to VB 365 logic, if the item was changed, we back it up, so it is an expected behavior.

We are already working with Microsoft on this issue and once we will have an update, I will get back to you.
*********

Dear Microsoft........

[StanO]

We're experiencing what appears to be the same issue.

Case #06244647

Thanks,
Stan

[Mildur]

Thank you for the case numbers. We are still checking with Microsoft.
It seems, there was a change on those two attributes for items in specific folders. Veeam Backup for Microsoft 365 checks those attributes to decide if an object in the mailbox was changed or not. If changed, we will backup the new version.

We will update this topic with a conclusion as soon we have all the information.

Best,
Fabian

[Mildur]

A short update. We have published a kb article and it will be updated when more information becomes available.
https://www.veeam.com/kb4483

Best,
Fabian

[rkn]

Since 19.08.2023 0:00 (CEST) there is no more unusual growth. Can someone confirm that it got better?

[Polina]

Hi rkn,

Unfortunately, I cannot officially confirm it; but glad to get at least some good news in the last few days.
We're constantly monitoring the state in support cases that are already opened, and still working closely with Microsoft to get more information on the root cause of the change.

[karsten123]

same for us. we have to take a service crash because of a full disk, but now it seems to go back to normal. very uncomfortable.
what the f*** was that?

[dasfliege]

Case 06244531

We run into the same problems since the 12th of august. As a service provider, we perform M365 backups for 35 different customers. Some of them do not have a backup since 11 days now, as the amount of changed data is just too much. Seems like it's almost performing a new initial backup for those customers.

But our biggest challenge will probably be next month's billing, as we charge our customers for used repository space, which has heavily increased now. I guess some of you guys are facing the same challenge?

[A.Rogers]

Yeah same issue of storage space usage for us as well. Completion times shot up massively although it didn't seem to hit all our jobs at the same time weirdly. Seemed to be spread out over a number of days instead of everything just being hit all at once.

[dbeerts]

Any update on this?
Is it safe to enable the backup again? I don't want to fill up the repo more as it already has grown with 6TB.. I hope there will be a fix to clear up this space again afterwards.

[hgrams]

We also have two customers who seem to have the problem. We have already had to enlarge the data disk twice and there is still no end in sight.

A fix soon would be highly appreciated

[StanO]

I received an automated message on my open case (Case #06244647) asking if the case was handled to my satisfaction and my case has been changed to a "Pending Customer Acceptance" status. It appears that Support may be waiting for me to indicate the issue has been fixed while I am waiting for Support to tell me the issue has been fixed.

The KB article (KB4483) doesn't appear to contain any additional information that steps have been taken to alleviate the issue and neither does my case log.

Stan

[RobertoM]

Case 06244531

We run into the same problems since the 12th of august. As a service provider, we perform M365 backups for 35 different customers. Some of them do not have a backup since 11 days now, as the amount of changed data is just too much. Seems like it's almost performing a new initial backup for those customers.

But our biggest challenge will probably be next month's billing, as we charge our customers for used repository space, which has heavily increased now. I guess some of you guys are facing the same challenge?

Same for us, the smaller organizations I think went back to normal simply because over the course of a few days basically everything has been downloaded again.
The result is that on snapshot-based repositories we're going to have to account for much more storage space for the complete retention period (years...)

We are in the same boat for customers billed on consumption, it's going to be a big problem and I feel for fellow service providers with way bigger customers than ours.

[agrob]

No updates so far. the last status i got is also that veeam is working with microsoft. i wondering if this has something to do with the new m365 backup service from ms... anyway, we need a solution for this soon, and the solution can not be that we have to backup all the data again.

[Polina]

Hi All,

Let me share some news, though unfortunately, it's not exactly what we all are looking for.
1) According to our observations, changes in items properties seem to have stopped on many impacted tenants a few days ago. This is good by itself, but still doesn't give us any clue on the root cause of the changes and no guarantees (sadly) that it won't happen again.

2) Microsoft is analyzing the situation, but "this is just going to take a bit more time to investigate" and "the problem hasn’t been identified yet".

3) Veeam RnD is analyzing if there's anything we can do to resolve the consequences of the issue, but this analysis doesn't look promising at this point. Items have been modified/changed and for VB365 this is a new and valid item version returned by the EWS change token during backup. We cannot simply delete it, neither change it or merge only the changed properties to some of the previous item versions. Nevertheless, the research continues.

As soon as there are any updates, Veeam support will notify you immediately, please keep being in touch with our engineers.