Maintain control of your Microsoft 365 data
SteelContainer
Service Provider
Posts: 157
Liked: 23 times
Joined: May 21, 2014 8:47 am
Location: New Zealand
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by SteelContainer » 2 people like this post

Case #07219006

I'm in the same boat as some above where issue started around 27/3 and is still persisting.
Some clients that normally do 20GB incrementals are doing 100-500GB per day
I have also logged a case with MS as per Veeam tech instructions.
If this isn't resolved soon it is really going to dig into our ability to provide a cost effective solution to our clients.
m.novelli
Veeam ProPartner
Posts: 576
Liked: 108 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by m.novelli »

I'm wondering if also other M365 Backup Vendors are in the same boat, like Acronis , Keepit, CodeTwo... Maybe Veeam could contact them to share this issue and do some brainstorming with / against Microsoft

Marco
Gostev
Chief Product Officer
Posts: 31969
Liked: 7439 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Gostev » 4 people like this post

Yes, other backup vendors are affected as well. Our partner team at Microsoft works with other backup vendors too, this is how we know it's not just us.

Of course, it's more "publicly visible" in case of Veeam simply because we protect more M365 mailboxes than most other vendors combined.
m.novelli
Veeam ProPartner
Posts: 576
Liked: 108 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by m.novelli »

Well done Gostev :D

Now I'm wondering how a Product Manager at Microsoft can give "green light" to a backend update that impact in such a HUGE way the backup ecosystem... :shock:

By the way Gostev, do you know what technology there is behind the Microsoft 365 Backup that started to appear in some Microsoft 365 tenants? https://learn.microsoft.com/en-us/micro ... -worldwide

Marco
Gostev
Chief Product Officer
Posts: 31969
Liked: 7439 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Gostev » 2 people like this post

Yes, in simple terms this will allow "entire mailbox snapshots". We will integrate with this tech once released b/c it is complementary in the same way as storage-based snapshots are complementary to proper backups for VMware protection.
NN25
Influencer
Posts: 12
Liked: 1 time
Joined: Oct 24, 2023 2:06 pm
Full Name: Nikola Nikolic
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by NN25 » 1 person likes this post

Hi,

Is there any update regarding this topic?

Backup increased now from 300GB to 1.2TB.
Gostev
Chief Product Officer
Posts: 31969
Liked: 7439 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Gostev » 2 people like this post

Microsoft is on it. They are also working to provide a statement we can share publicly, but as all such statements it needs to gather many approvals first.

We already have an excellent explanation from them about what happened and how they plan to minimize similar changes in future, but these details are provided under NDA and we're not allowed to share them publicly.
Gostev
Chief Product Officer
Posts: 31969
Liked: 7439 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Gostev » 4 people like this post

Microsoft provided us with the following statement that we're allowed to share with our customers:

Due to changes we are making to optimize search and compatibility functionality within our service, this action led to re-indexing for some of our customers' data.
The re-indexing is not indicative of a change to fields material to our BCDR and storage partners, so we will be working on a change to minimize the re-indexing that is prompting another backup of customer data.
We made changes over the weekend that should pause the unexpected behavior and those changes will be in place until Microsoft and our partners are able to make longer term enhancements to avoid future issues from taking place.
Thank you for your patience.
m.novelli
Veeam ProPartner
Posts: 576
Liked: 108 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by m.novelli »

"some" data :lol:

I think they re-indexed the same data over and over again... :roll:

Marco
VNR-LF
Enthusiast
Posts: 31
Liked: 5 times
Joined: Oct 28, 2020 1:14 pm
Full Name: Lars Freiberger
Location: Germany
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by VNR-LF »

m.novelli wrote: Apr 23, 2024 11:56 am "some" data :lol:

I think they re-indexed the same data over and over again... :roll:

Marco
yep, "some"... :lol:

it sure seems like some mailboxes have been re-indexed over and over again. for some of our mailboxes, three or four times the total mailbox size has been backed up over the past weeks. but looking at the past 3 days, backup sizes seem to be back to levels we've been used to before the issue started.

glad this is fixed and fingers crossed this issue will not resurface again.
superdevo
Influencer
Posts: 21
Liked: 6 times
Joined: Oct 16, 2019 7:43 pm
Full Name: David Torreggiani
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by superdevo » 1 person likes this post

I am still seeing huge incrementals
Polina
Veeam Software
Posts: 3293
Liked: 794 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Polina »

Hi David,

Do you have a case with our support? Also, in which region/data center are you seeing the issue?
superdevo
Influencer
Posts: 21
Liked: 6 times
Joined: Oct 16, 2019 7:43 pm
Full Name: David Torreggiani
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by superdevo » 2 people like this post

I have a case going with MS.
Their reply as of this morning:
I’m continuing to work with the product team on this. I don’t yet have an update unfortunately. This might take a little bit of time as there are multiple tenants impacted as you’ve noted. I apologize for this continued inconvenience here.
Am I the only one still dealing with this?

Will post here further updates.

Thanks for checking Polina

PS: region: Americas/US
VNR-LF
Enthusiast
Posts: 31
Liked: 5 times
Joined: Oct 28, 2020 1:14 pm
Full Name: Lars Freiberger
Location: Germany
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by VNR-LF »

since monday i have noticed another increase in the amount of data in our exchange backups. it seems to me as if once again, data that has already been backed up in the past is being backed up again. has anyone else noticed this?
MaartenA
Service Provider
Posts: 97
Liked: 34 times
Joined: Oct 31, 2021 7:03 am
Full Name: maarten
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by MaartenA »

Yes, same here.
Polina
Veeam Software
Posts: 3293
Liked: 794 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Polina »

@VNR-LF and @MaartenA

Please open cases with Veeam support (if you haven't done that yet) and provide me with your case IDs.
We were recently notified of a similar issue by another customer, and now the more data we have to provide to Microsoft, the better.

Thanks!
MaartenA
Service Provider
Posts: 97
Liked: 34 times
Joined: Oct 31, 2021 7:03 am
Full Name: maarten
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by MaartenA » 1 person likes this post

@Polina. See Case #07398831
agrob
Veteran
Posts: 395
Liked: 56 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by agrob » 1 person likes this post

see a similiar increase as well..
Case id: #07400232
VNR-LF
Enthusiast
Posts: 31
Liked: 5 times
Joined: Oct 28, 2020 1:14 pm
Full Name: Lars Freiberger
Location: Germany
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by VNR-LF »

Case #07400443
ghgard
Influencer
Posts: 11
Liked: 1 time
Joined: Feb 07, 2019 1:53 am
Full Name: Grant GArdner
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by ghgard »

Case #07420922

I started seeing this about 2 days ago. Its really random, one run the job will backup just a few hundred MB and then the next it will do 15GB. We typically never have larger than 750MB per run.
Mike Resseler
Product Manager
Posts: 8207
Liked: 1328 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Mike Resseler »

@ghgard
Thanks for the case #. To be honest, I don't understand it anymore as MSFT claims they are not making those updates anymore. I will monitor to see if it is again a batch of support cases coming our way :-(
agrob
Veteran
Posts: 395
Liked: 56 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by agrob »

Is there any update regarding this issue? tonight we again had 470GB of Data transfared and running out of Diskspace soon. I lose patient with those (put some not nice words here) at MS...
#07400232
MaartenA
Service Provider
Posts: 97
Liked: 34 times
Joined: Oct 31, 2021 7:03 am
Full Name: maarten
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by MaartenA »

For us, the problem disappeared again after a few days. EMEA region.
agrob
Veteran
Posts: 395
Liked: 56 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by agrob »

Thanks Maarten, wie had some day with normal traffic (arround 10-15GB) and now again 470GB... In the meantime i got feedback from Veeam Support that MS should have fixed this by today:
https://www.veeam.com/kb4483#:~:text=Ar ... r%203,20th.
Lets hope....
m.novelli
Veeam ProPartner
Posts: 576
Liked: 108 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by m.novelli »

Today I woke up with a Server full of VBO backup... 0 space available

It's a big customer with 700+ seat suffering this space issue since one year :(

I'm pushing my biggest customers to M365 Cloud Backup solutions, on-premises Server is no more a viable solution

Marco
RobertoM
Novice
Posts: 4
Liked: 1 time
Joined: Jul 15, 2021 4:28 pm
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by RobertoM » 1 person likes this post

We've been seeing very sporadic full mailbox downloads here and there across our customers, not sure why but probably unrelated to KB4483

I have however one customer where seemingly nobody changed or updated anything and all of a sudden Veeam for M365 decided to make a new, full backup...
There must be a reason, but since I cannot double (again) the size of the volume, I have to delete everything and start from scratch.
deduplicat3d
Expert
Posts: 119
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by deduplicat3d »

I believe I may be impacted by this issue. Is there any reason to open a support ticket, the KB seems to indicate there's no workaround? Is there anyways to tell how the storage is consumed within your backups (i.e. daily breakdown between mailboxes, or specific sharepoint sites that are using up more data?) The Storage Consumption report just shows the total storage per day.
Mildur
Product Manager
Posts: 10100
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Mildur »

Hi Corey
I believe I may be impacted by this issue. Is there any reason to open a support ticket, the KB seems to indicate there's no workaround?
Did you experience this issue recently or a few weeks ago? According to our information from Microsoft, the situation should have been resolved by now. If you're facing this issue in the last few days, I suggest opening a case with our customer support team to investigate further.
Is there anyways to tell how the storage is consumed within your backups (i.e. daily breakdown between mailboxes, or specific sharepoint sites that are using up more data?) The Storage Consumption report just shows the total storage per day.
Unfortunately not, today the product does not track "backup size" on object-level.

Best,
Fabian
Product Management Analyst @ Veeam Software
squebel
Service Provider
Posts: 162
Liked: 14 times
Joined: Sep 27, 2019 5:06 pm
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by squebel »

Similar issue. Case 07583604.

What do we do to get around this? There has to be a solution, right?
Polina
Veeam Software
Posts: 3293
Liked: 794 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: M365 Mailbox Backup increase since friday / latest August MS Patch installation

Post by Polina » 1 person likes this post

Hi squebel,

Thanks for providing your case ID. The problem from your support case does not seem to be related to the increased size of Exchange increments.
Please continue working with our support to investigate your specific case.

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests