Office365 performance

[edh]

Hi ,

Anyone got a good performance doing backups of Office365 tenants?

Even removing the Thrrothle in Office 365 Center the performance is terrible for the first full backup...

We got configured each tenant with 1 + 10 extra application for get more performance... but currently after +100 hours of backups we got just near 2TB backup done.

How do you do for backup tenants with more than 1K accounts? The first backup can take months...

I check the logs of the work and we dont have performance penally in our storage all is less than 1ms...

Regards

[Mildur]

Hi edh

In our latest version v6a, we have several performance optimizations.
https://www.veeam.com/kb4344

And I don‘t recommend to many application. We see performance issues if you use too many of them. We have a statement about that in our help center:

https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=60

Using multiple applications may impact the performance of your production SharePoint environment. This functionality will be deprecated in future versions of Veeam Backup for Microsoft 365.

Try to lower the applications and update to V6a.
If that doesn‘t help, please open a support case and let our support analyze the bottleneck. Please provide the case number here for our reference.

Thanks
Fabian

[johan.h]

Just use one app registration in VB365. Microsoft has updated this page about SharePoint throttling quite a bit in the last few months. You'll find the allocated performance for SharePoint has to do with how many resources are available in the underlying components that you and several other customers may be sharing - and the throttling is designed to prevent an individual tenant from oversubscribing this, with limits allocated based on how many M365 users are licensed.

[Polina]

Hi @edh,

A few questions in addition to my colleagues' comments above:
- What build are you running?
- Is it a SharePoint/OneDrive job or Exchange (or a mix of different types of data)?
- Are you backing up to a local Jet-based repository?
- What is your proxy sizing/configuration?

[karsten123]

just to clarify: the todays best practice is to setup no additional backup applications; just the one, that is created in the add orga wizard?
And the use of it will be deprecated in the future?

[Mike Resseler]

@karsten123 We are not certain yet whether it will be deprecated. At this point in time MSFT advises us not to use multiple app registrations anymore (although multiple backup accounts is no problem for them).

[infused]

run a vm in azure and backup to blobs. fast as.

[edh]

- What build are you running? 6.1.0XXXX
- Is it a SharePoint/OneDrive job or Exchange (or a mix of different types of data)?
We got 4 diferent jobs : 1Exchange/2Teams/3OneDrive/4Sharepoint
- Are you backing up to a local Jet-based repository? Yes
- What is your proxy sizing/configuration? 8cores / 32 GB RAM / 3TB SSD , single tenant per server (aprox 200 accounts) near 800 objects

Then should we delete the extra aplications and just leave 1?

Regards

[k00laid]

Adding in here that I would recommend that you bump the proxy (if observation warrants it and it is a VM) to 16 cores and to move away from JetDB and use object storage as your storage platform. What we see is a pretty impressive data reduction as well as a increase in speed. If your speed issues are occurring primarily in jobs 2-4 then it's still worth looking in the logs for either keyword "throttling" or "429".

[karsten123]

Applying CP 6.1.0.254 P20220825 and reducing backup applications to zero gave me significant performance gains for larger sharepoint sites.
With 6.1.0.222 there were runtimes > 24h

[Mike Resseler]

Karsten,

Thanks for letting us know. We do have test data for the performance gains but I am interested to hear how much you are getting between 6a and 6 CP1

[m.novelli]

Adding in here that I would recommend that you bump the proxy (if observation warrants it and it is a VM) to 16 cores and to move away from JetDB and use object storage as your storage platform. What we see is a pretty impressive data reduction as well as a increase in speed. If your speed issues are occurring primarily in jobs 2-4 then it's still worth looking in the logs for either keyword "throttling" or "429".

Now it's possible to backup directly to Azure BLOB Storage?
No need for a local HDD and then tiering?

Marco

[Mildur]

Hi Marco

There is no tiering with Veeam Backup for Microsoft 365 as in Veeam Backup & Replication with the SOBR.
VB365 Data flow goes from M365 Cloud to the VB365 Proxy to the Azure Blob Storage.
We just need the local HDD for the cache directory.

Thanks
Fabian

[m.novelli]

Nice! I have a couple of question, off-topic. I'm planning to move to Azure VM a big customer

- data format used on Azure BLOB Storage is still Jet Database?
- for 1 TB of mixed data (Exchange / Sharepoint / OneDrive / Teams) how much cache do I need?
- what Azure BLOB Storage do you suggest to lower costs? Hot or Cool based on transaction costs?
- real monthly costs to backup 1 TB of mixed data (Exchange / Sharepoint / OneDrive / Teams)?

Thanks!

[Polina]

Hi Marco,

1) Object storage repositories (including Azure Blob Storage) have their own format for storing backup data. For more information, you can check out our Help Center
2) Cache is normally about 1% of the source data size
3) You can decide based on some calculations provided here
4) Real costs would greatly depend on your source data (file versioning, structure, etc); but the like above can provide you with some basic estimations.

Thanks!

[m.novelli]

Thanks! Super helpfull

[k00laid]

Hi @mildur, correct me if I'm wrong but if you are using with Azure Blob or any other storage that isn't local to the proxy do you not still have an issue of hairpinning? The proxy does the processing and then writes to Blob and the local metadata cache at the same time but that could still be very bandwidth intensive.

Hi Marco

There is no tiering with Veeam Backup for Microsoft 365 as in Veeam Backup & Replication with the SOBR.
VB365 Data flow goes from M365 Cloud to the VB365 Proxy to the Azure Blob Storage.
We just need the local HDD for the cache directory.

Thanks
Fabian

[fractalenergy]

Applying CP 6.1.0.254 P20220825 and reducing backup applications to zero gave me significant performance gains for larger sharepoint sites.
With 6.1.0.222 there were runtimes > 24h

We actually have ONE user site/teams/mailbox that never end... Only one job with 3 object on it's own local repo and took forever and never end. I've just applied that lastest patch and let's see for the performance if it will help for me.

[karsten123]

Karsten,

Thanks for letting us know. We do have test data for the performance gains but I am interested to hear how much you are getting between 6a and 6 CP1

Hi Mike,
sure.

It is a Sharepoint Online site with 1.2 TB of data and more than 300000 files.

5.0.3.1063 + 20 backup applications: 12 minutes

6.1.0.222 + 0 backup applications: 28,5 hours

6.1.0.254 CP20220825 + 0 backup applications: 30 - 60 minutes

[Mildur]

Hi mildur, correct me if I'm wrong but if you are using with Azure Blob or any other storage that isn't local to the proxy do you not still have an issue of hairpinning? The proxy does the processing and then writes to Blob and the local metadata cache at the same time but that could still be very bandwidth intensive.

Yes, all M365 data must still be downloaded using the proxy and uploaded to the Object Storage in a compressed format as objects.
The question was about using a local backup storage. The data will not be stored locally before offloading it to Object Storage.

Thanks
Fabian

[Polina]

@karsten123,

That's very interesting information for us, thank you for sharing.
May I also ask you to check if there are any 429 errors in the logs on 6.1.0.254?

[karsten123]

Hi Polina,

there are no throttling events in the logs.

[Polina]

Karsten,

Would it be possible to get your logs via a support case? Our QA and RnD want to take a closer look at the backups on the latest version and, if possible, compare it with your previous runs. If you open the case, please provide me its ID, and I'll immediately escalate it.

Thanks!

[LeeMackie]

I am continuing to see poor performance with 6.1.0.254 in our own internal tenancy (I haven't upgraded our customer facing backup setup to 6a so can't comment there).

The issue is specifically around a particularly large site that never fully backed up during the initial run and has since never completed a back up at all as it just sits there seemingly doing nothing. In reality I've been told it's comparing the contents of the site against what is already backed up in object storage but a read rate of 8 KB/s and a processing rate of 2 items/s displayed in the VEEAM console does not fill me full of confidence it'll ever complete. Currently I'm up to 70 hours runtime but have left it as long as 120 hours before having to cancel for one reason or another.

We split that large site out from our main backup of Sharepoint sites, which since 6a has now been running for 54 hours (doing an incremental) at time of posting but is slowly working it's way through the sites.

For reference, we are not running any auxiliary backup applications and have re-authenticated our organization during the support ticket.

I'd love to hear any suggestions as to what I can do to get this site backed up - #02597087.

[Polina]

Hi Lee,

First, please continue working with our support engineers. If needed they will involve the RnD team in the investigation, and I'll follow up on your case as well.
Next, from time to time we see similar issues with some sites taking really a while to be backed up, but I've never yet heard of cases when such backups hadn't finally been completed )

[LeeMackie]

No problems Polina - just thought I'd add my 2c and see if there was any suggestions.

[Mohammed Sohail]

I am continuing to see poor performance with 6.1.0.254 in our own internal tenancy (I haven't upgraded our customer facing backup setup to 6a so can't comment there).

The issue is specifically around a particularly large site that never fully backed up during the initial run and has since never completed a back up at all as it just sits there seemingly doing nothing. In reality I've been told it's comparing the contents of the site against what is already backed up in object storage but a read rate of 8 KB/s and a processing rate of 2 items/s displayed in the VEEAM console does not fill me full of confidence it'll ever complete. Currently I'm up to 70 hours runtime but have left it as long as 120 hours before having to cancel for one reason or another.

We split that large site out from our main backup of Sharepoint sites, which since 6a has now been running for 54 hours (doing an incremental) at time of posting but is slowly working it's way through the sites.

For reference, we are not running any auxiliary backup applications and have re-authenticated our organization during the support ticket.

I'd love to hear any suggestions as to what I can do to get this site backed up - #02597087.

Hi Lee,
Just to provide my feedback. we have been facing the same issue since two months now and the support told us to wait.
After a lot of troubleshooting, We canceled the job and run it again and we have waited for almost 350+ Hours and then it finished.
After couple of days, the problem started again. so they provided us with some proxy.xml configuration changes on that specific repository ( these changes seems related on how SharePoint data cache and Blob is interacting to each other) and magically, the job finished in 21 Minutes .
I hope this helps.

[Polina]

Hi Mohammed,

21min vs 350+h is impressive... I'm definitely interested to learn more about it - may I ask for your support case ID?

Thanks!

[dotdk]

Hi Mohammed and Polina

We would be very interested in what changes was made to to proxy.xml since we are facing very long processing times.

We have seen increased backup time for SharePoint and OneDrive data since version 6. We were originally running with 50 backup applications, when that was a help, but are now down to 2, but we are still facing heavy throttling. If possible please share any insights you get from the scenario Mohammed described.

[Mohammed Sohail]

Hi Mohammed,

21min vs 350+h is impressive... I'm definitely interested to learn more about it - may I ask for your support case ID?

Thanks!

Hi Polina,
The First ticket we opened was this 02629448 .
and then we opened this ticket 02773906 where the change was provided in the XML.