Maintain control of your Microsoft 365 data
edh
Service Provider
Posts: 177
Liked: 44 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Contact:

Office365 performance

Post by edh »

Hi ,

Anyone got a good performance doing backups of Office365 tenants?

Even removing the Thrrothle in Office 365 Center the performance is terrible for the first full backup...

We got configured each tenant with 1 + 10 extra application for get more performance... but currently after +100 hours of backups we got just near 2TB backup done.

How do you do for backup tenants with more than 1K accounts? The first backup can take months...

I check the logs of the work and we dont have performance penally in our storage all is less than 1ms...

Regards
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Office365 performance

Post by Mildur » 1 person likes this post

Hi edh

In our latest version v6a, we have several performance optimizations.
https://www.veeam.com/kb4344

And I don‘t recommend to many application. We see performance issues if you use too many of them. We have a statement about that in our help center:

https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=60
Using multiple applications may impact the performance of your production SharePoint environment. This functionality will be deprecated in future versions of Veeam Backup for Microsoft 365.
Try to lower the applications and update to V6a.
If that doesn‘t help, please open a support case and let our support analyze the bottleneck. Please provide the case number here for our reference.

Thanks
Fabian
Product Management Analyst @ Veeam Software
johan.h
Veeam Software
Posts: 711
Liked: 178 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Office365 performance

Post by johan.h »

Just use one app registration in VB365. Microsoft has updated this page about SharePoint throttling quite a bit in the last few months. You'll find the allocated performance for SharePoint has to do with how many resources are available in the underlying components that you and several other customers may be sharing - and the throttling is designed to prevent an individual tenant from oversubscribing this, with limits allocated based on how many M365 users are licensed.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Office365 performance

Post by Polina »

Hi @edh,

A few questions in addition to my colleagues' comments above:
- What build are you running?
- Is it a SharePoint/OneDrive job or Exchange (or a mix of different types of data)?
- Are you backing up to a local Jet-based repository?
- What is your proxy sizing/configuration?
karsten123
Service Provider
Posts: 350
Liked: 79 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Office365 performance

Post by karsten123 »

just to clarify: the todays best practice is to setup no additional backup applications; just the one, that is created in the add orga wizard?
And the use of it will be deprecated in the future?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Office365 performance

Post by Mike Resseler » 1 person likes this post

@karsten123 We are not certain yet whether it will be deprecated. At this point in time MSFT advises us not to use multiple app registrations anymore (although multiple backup accounts is no problem for them).
infused
Service Provider
Posts: 178
Liked: 13 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Office365 performance

Post by infused » 1 person likes this post

run a vm in azure and backup to blobs. fast as.
edh
Service Provider
Posts: 177
Liked: 44 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Contact:

Re: Office365 performance

Post by edh »

- What build are you running? 6.1.0XXXX
- Is it a SharePoint/OneDrive job or Exchange (or a mix of different types of data)?
We got 4 diferent jobs : 1Exchange/2Teams/3OneDrive/4Sharepoint
- Are you backing up to a local Jet-based repository? Yes
- What is your proxy sizing/configuration? 8cores / 32 GB RAM / 3TB SSD , single tenant per server (aprox 200 accounts) near 800 objects

Then should we delete the extra aplications and just leave 1?

Regards
k00laid
Veeam Vanguard
Posts: 222
Liked: 51 times
Joined: Jan 13, 2011 5:42 pm
Full Name: Jim Jones
Location: Hurricane, WV
Contact:

Re: Office365 performance

Post by k00laid » 1 person likes this post

Adding in here that I would recommend that you bump the proxy (if observation warrants it and it is a VM) to 16 cores and to move away from JetDB and use object storage as your storage platform. What we see is a pretty impressive data reduction as well as a increase in speed. If your speed issues are occurring primarily in jobs 2-4 then it's still worth looking in the logs for either keyword "throttling" or "429".
Jim Jones, Sr. Product Infrastructure Architect @iland / @1111systems, Veeam Vanguard
karsten123
Service Provider
Posts: 350
Liked: 79 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Office365 performance

Post by karsten123 » 3 people like this post

Applying CP 6.1.0.254 P20220825 and reducing backup applications to zero gave me significant performance gains for larger sharepoint sites.
With 6.1.0.222 there were runtimes > 24h :(
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Office365 performance

Post by Mike Resseler »

Karsten,

Thanks for letting us know. We do have test data for the performance gains but I am interested to hear how much you are getting between 6a and 6 CP1
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Office365 performance

Post by m.novelli »

k00laid wrote: Aug 22, 2022 11:39 am Adding in here that I would recommend that you bump the proxy (if observation warrants it and it is a VM) to 16 cores and to move away from JetDB and use object storage as your storage platform. What we see is a pretty impressive data reduction as well as a increase in speed. If your speed issues are occurring primarily in jobs 2-4 then it's still worth looking in the logs for either keyword "throttling" or "429".
Now it's possible to backup directly to Azure BLOB Storage?
No need for a local HDD and then tiering?

Marco
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Office365 performance

Post by Mildur » 1 person likes this post

Hi Marco

There is no tiering with Veeam Backup for Microsoft 365 as in Veeam Backup & Replication with the SOBR.
VB365 Data flow goes from M365 Cloud to the VB365 Proxy to the Azure Blob Storage.
We just need the local HDD for the cache directory.

Thanks
Fabian
Product Management Analyst @ Veeam Software
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Office365 performance

Post by m.novelli »

Nice! I have a couple of question, off-topic. I'm planning to move to Azure VM a big customer

- data format used on Azure BLOB Storage is still Jet Database?
- for 1 TB of mixed data (Exchange / Sharepoint / OneDrive / Teams) how much cache do I need?
- what Azure BLOB Storage do you suggest to lower costs? Hot or Cool based on transaction costs?
- real monthly costs to backup 1 TB of mixed data (Exchange / Sharepoint / OneDrive / Teams)?

Thanks!
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Office365 performance

Post by Polina »

Hi Marco,

1) Object storage repositories (including Azure Blob Storage) have their own format for storing backup data. For more information, you can check out our Help Center
2) Cache is normally about 1% of the source data size
3) You can decide based on some calculations provided here
4) Real costs would greatly depend on your source data (file versioning, structure, etc); but the like above can provide you with some basic estimations.

Thanks!
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Office365 performance

Post by m.novelli »

Thanks! Super helpfull :)
k00laid
Veeam Vanguard
Posts: 222
Liked: 51 times
Joined: Jan 13, 2011 5:42 pm
Full Name: Jim Jones
Location: Hurricane, WV
Contact:

Re: Office365 performance

Post by k00laid »

Hi @mildur, correct me if I'm wrong but if you are using with Azure Blob or any other storage that isn't local to the proxy do you not still have an issue of hairpinning? The proxy does the processing and then writes to Blob and the local metadata cache at the same time but that could still be very bandwidth intensive.
Mildur wrote: Aug 29, 2022 8:31 am Hi Marco

There is no tiering with Veeam Backup for Microsoft 365 as in Veeam Backup & Replication with the SOBR.
VB365 Data flow goes from M365 Cloud to the VB365 Proxy to the Azure Blob Storage.
We just need the local HDD for the cache directory.

Thanks
Fabian
Jim Jones, Sr. Product Infrastructure Architect @iland / @1111systems, Veeam Vanguard
fractalenergy
Service Provider
Posts: 28
Liked: 7 times
Joined: May 19, 2021 1:40 pm
Full Name: Francis Brodeur
Contact:

Re: Office365 performance

Post by fractalenergy »

karsten123 wrote: Aug 26, 2022 2:58 pm Applying CP 6.1.0.254 P20220825 and reducing backup applications to zero gave me significant performance gains for larger sharepoint sites.
With 6.1.0.222 there were runtimes > 24h :(
We actually have ONE user site/teams/mailbox that never end... Only one job with 3 object on it's own local repo and took forever and never end. I've just applied that lastest patch and let's see for the performance if it will help for me. :)
karsten123
Service Provider
Posts: 350
Liked: 79 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Office365 performance

Post by karsten123 » 1 person likes this post

Mike Resseler wrote: Aug 29, 2022 5:28 am Karsten,

Thanks for letting us know. We do have test data for the performance gains but I am interested to hear how much you are getting between 6a and 6 CP1
Hi Mike,
sure.

It is a Sharepoint Online site with 1.2 TB of data and more than 300000 files.

5.0.3.1063 + 20 backup applications: 12 minutes

6.1.0.222 + 0 backup applications: 28,5 hours

6.1.0.254 CP20220825 + 0 backup applications: 30 - 60 minutes
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Office365 performance

Post by Mildur »

k00laid wrote: Aug 29, 2022 1:21 pm Hi mildur, correct me if I'm wrong but if you are using with Azure Blob or any other storage that isn't local to the proxy do you not still have an issue of hairpinning? The proxy does the processing and then writes to Blob and the local metadata cache at the same time but that could still be very bandwidth intensive.
Yes, all M365 data must still be downloaded using the proxy and uploaded to the Object Storage in a compressed format as objects.
The question was about using a local backup storage. The data will not be stored locally before offloading it to Object Storage.

Thanks
Fabian
Product Management Analyst @ Veeam Software
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Office365 performance

Post by Polina »

@karsten123,

That's very interesting information for us, thank you for sharing.
May I also ask you to check if there are any 429 errors in the logs on 6.1.0.254?
karsten123
Service Provider
Posts: 350
Liked: 79 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Office365 performance

Post by karsten123 »

Hi Polina,

there are no throttling events in the logs.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Office365 performance

Post by Polina »

Karsten,

Would it be possible to get your logs via a support case? Our QA and RnD want to take a closer look at the backups on the latest version and, if possible, compare it with your previous runs. If you open the case, please provide me its ID, and I'll immediately escalate it.

Thanks!
LeeMackie
Service Provider
Posts: 20
Liked: 2 times
Joined: May 25, 2022 12:37 am
Full Name: Lee Mackie
Contact:

Re: Office365 performance

Post by LeeMackie »

I am continuing to see poor performance with 6.1.0.254 in our own internal tenancy (I haven't upgraded our customer facing backup setup to 6a so can't comment there).

The issue is specifically around a particularly large site that never fully backed up during the initial run and has since never completed a back up at all as it just sits there seemingly doing nothing. In reality I've been told it's comparing the contents of the site against what is already backed up in object storage but a read rate of 8 KB/s and a processing rate of 2 items/s displayed in the VEEAM console does not fill me full of confidence it'll ever complete. Currently I'm up to 70 hours runtime but have left it as long as 120 hours before having to cancel for one reason or another.

We split that large site out from our main backup of Sharepoint sites, which since 6a has now been running for 54 hours (doing an incremental) at time of posting but is slowly working it's way through the sites.

For reference, we are not running any auxiliary backup applications and have re-authenticated our organization during the support ticket.

I'd love to hear any suggestions as to what I can do to get this site backed up - #02597087.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Office365 performance

Post by Polina »

Hi Lee,

First, please continue working with our support engineers. If needed they will involve the RnD team in the investigation, and I'll follow up on your case as well.
Next, from time to time we see similar issues with some sites taking really a while to be backed up, but I've never yet heard of cases when such backups hadn't finally been completed )
LeeMackie
Service Provider
Posts: 20
Liked: 2 times
Joined: May 25, 2022 12:37 am
Full Name: Lee Mackie
Contact:

Re: Office365 performance

Post by LeeMackie »

No problems Polina - just thought I'd add my 2c and see if there was any suggestions.
Mohammed Sohail
Service Provider
Posts: 36
Liked: 3 times
Joined: Nov 25, 2017 2:43 am
Full Name: Mohammed Sohail
Contact:

Re: Office365 performance

Post by Mohammed Sohail »

LeeMackie wrote: Sep 02, 2022 3:56 am I am continuing to see poor performance with 6.1.0.254 in our own internal tenancy (I haven't upgraded our customer facing backup setup to 6a so can't comment there).

The issue is specifically around a particularly large site that never fully backed up during the initial run and has since never completed a back up at all as it just sits there seemingly doing nothing. In reality I've been told it's comparing the contents of the site against what is already backed up in object storage but a read rate of 8 KB/s and a processing rate of 2 items/s displayed in the VEEAM console does not fill me full of confidence it'll ever complete. Currently I'm up to 70 hours runtime but have left it as long as 120 hours before having to cancel for one reason or another.

We split that large site out from our main backup of Sharepoint sites, which since 6a has now been running for 54 hours (doing an incremental) at time of posting but is slowly working it's way through the sites.

For reference, we are not running any auxiliary backup applications and have re-authenticated our organization during the support ticket.

I'd love to hear any suggestions as to what I can do to get this site backed up - #02597087.
Hi Lee,
Just to provide my feedback. we have been facing the same issue since two months now and the support told us to wait.
After a lot of troubleshooting, We canceled the job and run it again and we have waited for almost 350+ Hours and then it finished.
After couple of days, the problem started again. so they provided us with some proxy.xml configuration changes on that specific repository ( these changes seems related on how SharePoint data cache and Blob is interacting to each other) and magically, the job finished in 21 Minutes .
I hope this helps.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Office365 performance

Post by Polina »

Hi Mohammed,

21min vs 350+h is impressive... I'm definitely interested to learn more about it - may I ask for your support case ID?

Thanks!
dotdk
Service Provider
Posts: 30
Liked: 9 times
Joined: Jun 14, 2019 11:55 am
Full Name: Thomas Lund
Contact:

Re: Office365 performance

Post by dotdk »

Hi Mohammed and Polina

We would be very interested in what changes was made to to proxy.xml since we are facing very long processing times.

We have seen increased backup time for SharePoint and OneDrive data since version 6. We were originally running with 50 backup applications, when that was a help, but are now down to 2, but we are still facing heavy throttling. If possible please share any insights you get from the scenario Mohammed described.
Mohammed Sohail
Service Provider
Posts: 36
Liked: 3 times
Joined: Nov 25, 2017 2:43 am
Full Name: Mohammed Sohail
Contact:

Re: Office365 performance

Post by Mohammed Sohail »

Polina wrote: Sep 22, 2022 7:26 am Hi Mohammed,

21min vs 350+h is impressive... I'm definitely interested to learn more about it - may I ask for your support case ID?

Thanks!
Hi Polina,
The First ticket we opened was this 02629448 .
and then we opened this ticket 02773906 where the change was provided in the XML.
Post Reply

Who is online

Users browsing this forum: theadamlion and 10 guests