Sharepoint Online Backups are SLOW

[Unison]

Hi guys,
we are trialing Veeam Backup for o365 for a client - we have deployed VeeamBackup for o365 using the Veeam/Azure Marketplace system for o365 - i.e. VM that has Veeam for o365 already installed ready to go.
We setup a job for exchange a job for onedrive and a job for sharepoint - kicked it off and walked away.

the initial jobs completed fine over the weekend.

Problem is, we are now trying to run incremental - but the Sharepoint job is taking forever! last run it took just under 24hrs and there was only a few hundred MB change across the entire Sharepoint site (o365 usage reports show there was barely any data change between job runs!).
The sharepoint site in total only has 80gig and about 20,000 files in it - so not really that massive.

Of course i started searching on this and found quite a few reports of SLOW Sharepoint backups - even some suggestions that this slowness was resolved with V3.....but we are on V3.0.0.422 - and definitely seeing this slowness problem.

We are thinking of Splitting the sharepoint job as has been suggested in order to reduce the chance of MSFT throttling - but i suspect that the throttle will happen for one half/some of the splits anyway....not really solving the problem.

As this is a TRIAL - i cant log a support job so cant list a support ticket - but was hoping some of the 'Greens' can chime in here to advise if there is anything that can be done to improve sharepoint backup - if anything has been worked out yet with MSFT to solve the throttling problem for backup operations?

Sharepoint backups for this customer is obviously a big deal - if it is near impossible to backup with Veeam or requires the Azure resources to be powered on permanently generating $$$$$ Azure costs - then i dont think we will stick with Veeam for this and will have to look at other backup solutions - unless other solutions are all plagued with this same sharepoint problem?

[Polina]

Hi Gav,

First, can you check if there're any SharePoint/OneDrive libraries with more than 5000 items in them?
Next, even if you're using a Community Edition, you can open a support case. Please share your case ID here and we'll provide you a fix.

Thanks!

[Unison]

Hey Polina,
my post above was quite long/detailed so its understandable you missed the bit where i mentioned that the sharepoint account has about 20,000 files in it and is about 80 gig in size......MOST of those 20k files and 80GIG data is inside a single site.
There are several other sites but most of those only have a few hundred MB worth of data in them.
So the main problem is the single site that has about 20k files in it and most of the 80gig.

Ok will attempt to create a support ticket against this trial product and post the ID here - but will also standby for anything you can provide to try or to confirm this is not something that can be resolved.

[Unison]

Support case created:
Case #03740054

[Polina]

Hi Gav,

Thank you for providing the case ID number; support engineers will contact you shortly.

My question was to drill down into a bit more detailed information and to check if in that site of 20k were any libraries containing 5k+ of items. In the build 3.0.0.422, there was a bug in the processing of such libraries (parts of a site), which caused sluggishness of incremental jobs.

[Unison]

Hi Polina,
is there a fix/patch for that bug in .422? Support have not advised of a bug/fix - wondering if you can provide that or workaround/more details?

[Polina]

Hi Gav,

The patch provided to you by the support engineer already includes all the required fixes.
How much time did your incremental job take after applying the patch?

[Unison]

Hi Polina,
the incremental jobs seem to now be taking about 40mins - up to 1.5hrs......which doesn't seem that long but these are job runs where there was only a few MB transferred in total....so basically nothing changed but still takes around 1hr to complete - and the processing rate shown is in the KB space - sometimes only showing a processing rate like 3kb for an entire incremental run (hence taking an hr to complete due to the slow rate).
I have sent the job logs to support as they requested those so will see if they find anything from that.

However this being our first experience with Veeams Backup for o365 product - so far the seemingly very slow performance issues are going to make it difficult to recommend or deploy for other sites.
Is Veeam looking to establish a mechanism with Microsoft so that the Veeam product can backup the tenant data without any throttling - seems that would be the only solution to these kinds of issues.

[thoken]

Hi Unison,

Did you receive some more feedback regarding this case? I'm having the same problems when processing SharePoint teamsites and OneDrive. My backup respository is in MS Azure so normaly bandwith shouldn't be a problem.

Best regards,

Thomas

[nielsengelen]

Thomas, best would be to open up a support case so we can have some more insight in what could be going on.

[Unison]

Hey Thomas,
Unfortunately no not really - Veeam are a little vague on the reasons/details/their efforts to resolve this kind of problem (other than to say things like 'the new Veeam for o365 is not X number of times FASTER' - but not really saying how they achieve that or what they have done behind the scenes or what agreements they have created with Microsoft to somehow flag the backup activity as not being the same kind of 'access' that causes Microsoft to in turn throttle your access/backups during job runs).

Microsoft still seems to be throttling 'large' SharePoint backups (wouldn't really consider an 80gig/20,000 files SharePoint setup to be 'large' but it still results in massive throttling - in the early days. Microsoft will say they are throttling because there are too many access attempts under the one o365 login (that your backup job uses) - Veeam will turn around and say there is nothing they can do about that......so around and around in circles the problem goes......cant see this being properly resolved until Veeam and Microsoft somehow work out a way to flag the backup activity as 'backup activity' so that Microsoft can then prevent any kind of throttling on that.
That will take cooperation between Veeam and Microsoft there.....and Veeam dont seem to be doing anything about that or working towards that......well maybe they are....but Veeam are closed lip about it (probably because they dont really want to market/acknowledge that 'if your sharepoint site has a lot of data in it, the Veeam for o365 solution is going to be terrible and disappointingly slow - because Microsoft ARE going to throttle the backup activity).

How we have managed to get by up to this point - is we split up the Sharepoint backup job into a series of smaller backup jobs......say one job backing up 50 sub sites, another job for the next 50 and so on. We also made the exchange and onedrive backups run OUTSIDE of the windows of these sharepoint jobs so that no jobs ran in parallel or close to each other. Then over a period of weeks, we ran each of those jobs in their own window (never running them together). Most of them still throttled at some point - so they were slow - but eventually they all completed.
Then we started consolidating the Sharepoint jobs one by one into just the ONE single Sharepoint backup job capturing all sites - and now we just have one Sharepoint backup job and this job completes within an hour usually - backing up all 80gig of data and 20k files (the sites have very little change day to day).
So yes, a really messy and 'not as described' o365 backup solution (you wont see any mention of these requrements/issues on the Veeam for o365 marketing page!) - but it seems for now, this is the kind of thing/setup which is necessary to get a decent size sharepoint site backed up.

Hopefully Veeam and Microsoft can create a solution for this.

[Polina]

Unison, Thomas,

First, my apologies for keeping you waiting for a response.

Second, I'll try to throw some light on what's happening in the background behind the performance metrics you see.

When backing up SharePoint (or OneDrive), what matters most is not the amount of data processed, but the number of items in a site. And those are not only files themselves (file content), but many other entities, like versions, fields, properties, permissions, features, views and so on. For example, a small 3.3GB site with 8.4K files can have 11.3K items. And please note, that the site size you see in the Admin Center doesn’t count the extra amount of MBs or GBs consumed by these additional items.

To speed up incremental runs, we utilize a native Microsoft mechanism that allows detecting changes within a site. It saves time on analyzing changes per item and allows to focus on processing only the changed ones. In cases with negligible amount of changed data, we see speeds increase up to 40x.

But even with this mechanism and significantly reduced number of items to process, there’s still a risk of getting throttled. The nature of throttling limits doesn’t allow to make any exceptions (unfortunately). There’re some workarounds, and I can confirm that we’re going to deliver those in the next versions.

Currently, our recommendations for SharePoint and OneDrive backups is to split different services to different jobs (or even sites and subsites to different jobs), tweak proxy threads if needed (for example, reduce the number of threads if you see many 429 errors) and mind your backup windows to avoid ‘overheating’ by sending too many request from a single account.

Thanks!

[Unison]

Thanks for the response Polina - it is appreciated.
I for one will be very interested in seeing what the workarounds coming in the next version will be - hopefully you remember to post them here so i see an alert about them

The nature of throttling does make it difficult to add an exception - but wouldn't be impossible? Just a quick idea....couldn't Veeam engage with Microsoft to implement some kind of solution - perhaps when a veeam job starts, it first talks to the Microsoft environment to advise that "Tenant XYX" is about to be backed up by a backup solution "please avoid applying any throttling policies to this tenant" (as Microsoft would have a way to do this for large governments/organisations on their platform that demand much higher access volumes - they would have a way to disable/set different throttling policies on a per tenant basis).
Then when the Veeam job completes, Veeam advises the Microsoft environment "Ok, backup operation finished, please return the throttle policy back to normal".

Obviously this is a high level view of a possible solution - but it does seem like it would be technically possible (with some kind of agreement/solution created between Veeam and Microsoft).

Our Sharepoint backups are running now - but it took a lot of manual intervention to split the jobs, manage backup windows, avoid overheating etc etc in the first few weeks just to get a successful run (also adding to Azure costs as we have Veeam for o365 running from Azure Marketplace) - this is not a good experience for any new or even existing Veeam customers.
Hopefully the coming workarounds reduce this shock and manual overhead of the solution - until something can be done about throttling management.

[nielsengelen]

We'll make sure to follow up here Gav so you know v4 is available with the improvement. We are always engaging with Microsoft but it's always a matter of discussions and time. If Microsoft does implement something like it, it most likely won't be unique for us (sadly ).

We appreciate your feedback, workaround and final solution and hopefully, the next version eases it.

[Mike Resseler]

And to anser on your high-level overview proposal... Great minds think alike (we have done these conversations quite some time ago ). But that product team does not want to make an exception, and to be honest, I kind of understand why they do it... I would do the same if I were in their shoes. The risk of lots of vendors asking for exceptions would basically make throttling not used anymore at all

[Unison]

Thanks for advising that you/veeam have already been trying to talk to them about creating a solution like this.
I too understand why they might be doing it for 'all' - specifically not trying to create exceptions for backup vendors (because we know they would have the ability to change throttling per tenant because of gov/large sites etc) - but throttling is a solution/method of preventing/blocking unplanned/excessive requests and activity.........however backup is not really unplanned or excessive as it is the tenants owner who is responsible for backing up that data (correct Microsoft?).
Microsoft will not 'backup' the data for any tenant and it is openly known that Microsoft will point the finger back at the Tenant owner if the worst kind of disaster happened and all data was lost for a tenant in the Microsoft eco system ("Why didnt you have a backup of your data?" Microsoft would just say in a total loss situation).
Microsoft offers no kind of backup (replication/redundancy offered by MS is not backup) for o365 - therefore they should not throttle/penalise in any way those organisations who are filling the gap by protecting their own data with 'backups'.....any backup solution of their choosing.

So while i can understand why MS might not be allowing a throttling solution for backup vendors (maybe more because its difficult for them or throttling helps to reduce their own bandwidth/throughput costs).....i just don't think its acceptable because i understand more that it is a tenant owners responsibility to backup that o365 data......why would MS penalise customers for backing up their own data (given backup operation are not unplanned or excessive - they are just completely necessary.....a requirement of MS....a requirement of the tenant for having your data in o365).

I think it will just be a matter of time before a backup vendor cracks this throttling policy issue with MS - hopefully its Veeam first - because it doesn't make sense why backup is categorised as excessive/unnecessary traffic - therefor throttled - when it is absolutely necessary and vital traffic to ANY tenant)

Creating exceptions ONLY for backup vendors would not reduce the effectiveness of throttling policies either - because throttling policies would still be effective for all the other reasons why throttling policies are implemented int he first place......but they would just no longer penalise/impact legitimate USER or BACKUP activities. So yes bandwidth/throughput demands would go up but that is only because legitimate backup operations would no longer be slammed by these 'catch all' careless throttling policies.

[Mike Resseler]

Well, I certainly don't disagree with you. And maybe these things will improve in the future. In the meantime, as we stated above, we continue to work on finding new ways of making the backup faster, and our next version has again such an improvement. The beta will be here soon and we are hoping that a lot of you (through our normal beta policies in cooperation with your SE) will try it out in a test environment and see the improvement (and report on it )