The OneDRive backup takes 3-4 days to complete

[sumeet]

Hello,

Case # 04850232 : The OneDRive backup takes 3-4 days to complete and the CPU remains at 100% utilization when OneDrive backup runs.

This setup is at version 5.0.1.225. No changes in the time to backup even after the upgrade to 5b.

The server has 14 CPUs and 24 GB RAM.

We have other clients for whom more data gets downloaded for OneDrive backup and still the duration is within 12 hours.

The incremental run that completed earlier today took 61 hours 53 minutes and processed 2689 objects of 2793 and transferred 40.1 GB of data for 93832 items processed.
Currently the retry run for OneDrive is running and for the last 7 hours 48 mins and has completed 24 objects of 104 with transferred 2GB for 4573 items processed. The CPU is at 100% and Memory utilization is at 91%

There are 10 backup accounts and I have checked that throttling is minimum
The number of threads is set to 64 (it was 48 until today morning). Last week I had reduced this value from 64 to 48, hoping it might help.

The other backups on this server complete within 24 hours, allowing us to have daily restore points.

Any help?

[Mike Resseler]

Hi Sumeet,

Thanks for letting us know the case number. Please continue to work with our engineers. Based on the info there is not much I can tell you. I do have some questions... You speak about other clients... Does this mean you are using this installation as a service provider? You have multiple organizations protected? And only this particular organization is having issues?

[sumeet]

Hi Mike,

For each client we have a dedicated windows server. So just one Organization protected in this server. All the CPU and memory is dedicated to this organization.

By referring too other clients, I mean other organizations on other servers, which do not have as many objects as this client has, but still the amount of time this takes way too much. The other clients have less objects, but so the amount of CPU and memory is less, the number of backup accounts are less, the number of threads are less, but they complete in 12 hours.

Few months ago (while at version 4), I had opened a case for this same client. The case concluded that more CPU is required. More CPU would mean to best have a full physical host and not waste on VMware license. Not sure how to go ahead. Will deploying another windows server and move just the OneDrive job help? I doubt, because on the current server, once the other backup jobs (Archive, Exchange, SharePoint) complete, I do not see much improvements in the pace of OneDrive job. You guys understand better, maybe something from the logs will help.

-Sumeet.

[sumeet]

Hello all,

Support asked me to install the latest update 5.0.1.252, I could do that yesterday as the OneDrive backup had completed yesterday morning after 83 hours of running (processed 2743 objects of 2793 with 51.6 GB transferred and 112541 items transferred). The 46 errors are due to the item changed during backup, which is expected as the backup keeps running during business hours.

Even after latest patch (which I knew would not make any difference, but as it is with every case, the first thing I'm asked to do is install latest patch), there is no change. The backup is been running for 22hours with 1090 objects processed of 2790 and transferred 21.2 GB for 37187 items.

I have informed this to the support, waiting for more updates.

Does anyone in the forum have similar size of users in OneDrive to process? What is the performance? Any pointers?

Thanks,
-Sumeet.

[Mike Resseler]

Let them continue to investigate, from my side I will take a look into the case also but I am not going to make you any promises

[sumeet]

The case I had opened did not help much. I was suggested to setup a different proxy for onedrive and sharepoint backup. Do not see much value a different proxy would provide if all the resources on the box is dedicated to VBO.

Anyways, hoping that more CPU might help, we have moved the VM on a dedicated ESX host (no other VM on the host) and 54 vCPUs (have kept 2 vCPUs for the host) have been allocated to the VM and still the backup has just processed 764 objects of 2799.

Does anyone know a way to check if the OneDrive backup is running delta discovery and not a full sync. Or to put the question in another way, the backup run has found the change token and only doing an incremental run for every object, how do I check this?

In the history tab, the backup has "Incremental" against the job, so it is not a full sync. But is there a way to know VBO has a fully functional change token in place for the backup job?
Also, does the change token work at each object level or is it at the job level? What if a single/few objects in the job do not have a change token - is this possible?

The reason I ask this is because, almost every back up run, I see few users that have duration more than 20+ hours. In the current backup run, which is still running, the duration is 20:49, total amount of data transferred 10.8 GB and items processed are 16818, but there are four users that have duration as 20+ hours, three user has 18+ hours and so on.
Since the data transferred is not much, it makes me feel that every job run, VBO is doing a internal full sync and is unable to get a change token generated.

Any suggestions??

Thanks,
-Sumeet.

[A.Rogers]

Hi Sumeet,

Did you manage to get anywhere with this issue at all? I am struggling with Sharepoint and Onedrive speeds myself on a new tenant we recently set up. I haven't been able to get to the bottom of it yet with Veeam also looking. Were you using modern authentication with legacy by any chance? Something that seemed to help with the speed of the incremental file checking (not downloading new data) was switching over to pure modern authentication only.

[sumeet]

Hi Rogers,

My client requires me to use basic authentication. I have asked them several times if we can switch to modern authentication, as I suspected that this should speed up the backup process (though I have not found any official documentation that confirms this). Hopefully in the next few days, my client confirms this switch.

Regarding the issue, some progress that shows change token is missing for few objects.

In the first 15mins of OneDrive job with 80+ users in processing, I see the following entries (snippet from logs pasted below) that says full backup will be performed.

The logs shows "No web changes to analyze. Full backup will be performed".
Meaning that for few objects the change token is not available.

I have changed config.xml (and restarted VBO services) so to not rotate the logs, and at the end of day 4, when OneDrive completes, I have full logs available to scan through on how many objects have "Full backup will be performed".

Then the next steps will be for Veeam to check why change token is not generated.

Some level of debugging that Veeam be doing is being done by me.



3/08/2021 5:12:10 PM 155 (3908) Synchronizing web changes...
3/08/2021 5:12:10 PM 155 (3908) No web changes to analyze. Full backup will be performed
3/08/2021 5:12:10 PM 148 (13752) Changed views: 0
3/08/2021 5:12:10 PM 148 (13752) Deleted views: 0



3/08/2021 5:12:51 PM 99 (9152) Synchronizing list changes...
3/08/2021 5:12:51 PM 155 (3908) Synchronizing list changes...
3/08/2021 5:12:51 PM 155 (3908) No list changes to analyze. Full backup will be performed.
3/08/2021 5:12:51 PM 155 (3908) Loading list details...


3/08/2021 5:14:34 PM 117 (2832) Site collection ID: 1235849d-3ca7-4c0c-9c84-d1d884753136
3/08/2021 5:14:34 PM 117 (2832) Web ID: 92d3c6de-7ac0-452b-a4d5-704d3bb5d84c, Title: Ana Rita Pires
3/08/2021 5:14:34 PM 164 (13920) Synchronizing list changes...
3/08/2021 5:14:34 PM 164 (13920) Synchronization completed. Elapsed time: 00:00:00.0498897
3/08/2021 5:14:34 PM 164 (13920) No failed items from the last run
3/08/2021 5:14:34 PM 164 (13920) Analyzing list changes...
3/08/2021 5:14:34 PM 164 (13920) Changed views: 0
3/08/2021 5:14:34 PM 164 (13920) Deleted views: 0
3/08/2021 5:14:34 PM 164 (13920) Changed items: 1
3/08/2021 5:14:34 PM 164 (13920) Deleted items: 0
3/08/2021 5:14:34 PM 164 (13920) List details changed: False
3/08/2021 5:14:34 PM 164 (13920) List permissions changed: False
3/08/2021 5:14:34 PM 164 (13920) List changes analysis completed. Elapsed time: 00:00:00.0000188
3/08/2021 5:14:34 PM 164 (13920) Loading list details...
3/08/2021 5:14:34 PM 82 (15224) Personal site found: https://yarratrams-my.sharepoint.com/pe ... ams_com_au
3/08/2021 5:14:34 PM 82 (15224) Checking site collection administrator privileges on site: https://yarratrams-my.sharepoint.com/pe ... ams_com_au...
3/08/2021 5:14:34 PM 148 (13752) Changed item: /personal/ahfaz_rehman_yarratrams_com_au/Documents/1. OCMS Projects/Projects/DMP Project/DMPSR004_ Change Request CR_04.xlsx, version: 22, label: 9.0
3/08/2021 5:14:34 PM 117 (2832) Synchronizing web changes...
3/08/2021 5:14:34 PM 117 (2832) No web changes to analyze. Full backup will be performed
3/08/2021 5:14:34 PM 164 (13920) List details not changed


3/08/2021 5:15:01 PM 117 (2832) Saving web details...
3/08/2021 5:15:02 PM 117 (2832) Looking for OneDrive document library, template ID: MySiteDocumentLibrary...
3/08/2021 5:15:02 PM 148 (13752) Changed item: /personal/ahfaz_rehman_yarratrams_com_au/Documents/PM Meeting 2-8-2021.docx, version: 42, label: 5.0
3/08/2021 5:15:02 PM 117 (2832) OneDrive document library (ID: a863b3c8-39bc-4079-952b-1d81ec46c5d7, title: Documents, version: 12)
3/08/2021 5:15:02 PM 117 (2832) Synchronizing list changes...
3/08/2021 5:15:02 PM 117 (2832) No list changes to analyze. Full backup will be performed.
3/08/2021 5:15:02 PM 117 (2832) Loading list details...

[A.Rogers]

This is the Microsoft document about throttling that pushed me into moving over to modern authentication only in the end. If you do move to pure modern then it is actually pretty easy to set up. You would need a global admin to log in for you but Veeam handles the rest.

https://docs.microsoft.com/en-us/sharep ... int-online

Microsoft Graph is cloud born APIs that have the latest improvements and optimizations. In general, Microsoft Graph consumes less resource than CSOM and REST to achieve the same functionality. Hence, adopting Microsoft Graph can improve application's performance and reduce throttling.

[sumeet]

Hi all,

Going back to this old thread.
It took us more than a year to get this resolved, below are the details on what the issue was.
Now the OneDrive backups run normal, instead of 6-7 days (as the duration kept increasing)

=================

Our client stores user profiles in the users OneDrive, including their favourites.
They have a group policy which pushes out 5 or so standard favourites to all users.
The group policy uses the “Update” Function to create the favourites and every time the Group Policy refreshes it checks each favourite file to see if there is anything to update.
Not sure why, but every time it does this check it changes the “date modified” flag on the file, which Veeam is seeing as a changed file and flags them to be backed up.
This group policy will reapply every 90 mins or so and generate a new change every time.

The fix is to change the Group Policy to use the “Create” function, instead of “Update”.
Create checks to see if the file exists, if it does, it does nothing. If the files doesn’t exist it creates it.

=================

In case if someone runs into this issue, hope this helps.

Thanks,
-Sumeet.

[Mike Resseler]

Sumeet,
Extremely appreciated that you come back to us with this information. Honestly, I am completely surprised by this and would have never thought on something like this. You might call this misconfiguration, but in the end, it existed. I'm wondering now how did you came up that this could be the issue?