Extremely slow reading from Office 365 cloud

[MILJW002]

Hi Polina - you are just a bit ahead of Peter (the engineer helping me). I’ve provided the logs etc, but I don’t have the fix yet. Yep, I’ll happily share the results once I’ve been able to test the fix.

[MILJW002]

Hi Polina,

Just a quick update. I received the fix this afternoon, so have been testing. The OneDrive for Business backup (with no changed data) has gone from 3.5 - 4 hours to between 1:40 hours to 2 hours. So, first impressions are it looks better.

I did however notice that the bandwidth consumption keeps bursting, so working with support to see if I can tune any other settings.

So, def an improvement, but still investigating.

J

[Polina]

Thank you for this news, James!
VBO proxy settings allow for limiting the bandwidth consumption if needed. Nevertheless, please continue working with support to optimize your setup.

[MILJW002]

Hi Polina,

I don’t need to limit the bandwidth consumption as it’s not saturating the link (yet).

This is running on an ADSL connection (I know, slow) that is capable of 23 Mbps down and 1 Mbps up.

Before the patch it was barely parsing traffic, but now it is saturating the upstream (approx 1 Mbps) for a couple of seconds then dipping down to around 200 Kbps. Before it didn’t really get above about 200 Kbps. My thought was if the Microsoft throttling was now not kicking in then it should saturate the 1 Mbps upstream while the job is running.

If I can get to the stage of it saturating the link I can then look at limiting the bandwidth.

As it is bursting, I’m wondering if the throttling is still kicking in, or if I need more/less worker threads (tuning). This is what I’m currently working with support on.

Overall it is a lot better, but I don’t think it is completely right yet.

J

[Polina]

James,

How exactly are you tracking traffic/bandwidth saturation? Are you looking at the Read/Processing rates in the VBO console?

[MILJW002]

I’ve been using Task Manager and PerfMon on the server and net flow logs from the router to monitor the actual data throughput. The Read/Processing rates in the VBO console are much lower, so not really reporting how much data is going over the wire for a backup.

Overall things are looking a lot better.

[Mike Resseler]

Thanks for the update James.

For all on this thread, besides HAPPY new year , the MSP with the fixes is out: please go and visit https://www.veeam.com/kb2809 to get a cumulative update pack with hotfixes.

[roelvdw]

Hi Mike

The KB article says you need Veeam B&R 9.5 U4 in combination with the new update with office 365 v2.
But how can we get the update 4 for Veeam backup and replication ?

Kind regards

Roel

[Mike Resseler]

You don't actually need it. It is necessary on top of update 4 if you are running VBR and VBO together. The patch is also intended to make sure that the explorers from VBR 9.5U4 and VBO 2.0 are the same (the products share the Veeam explorers)

[roelvdw]

Hi Mike

I have installed VBR & VBO365 on the same physical server .
Where can i find VBR 9.5 U4 ?

Kind regards

Roel

[Mike Resseler]

Roel,

VBR 9.5 U4 will be out very soon.

[roelvdw]

Hi Mike

Do i have to wait on VBR 9.5 U4 before installing VBO365 update ?

Kind regards

Roel

[Mike Resseler]

Roel,

Yes, they are on the same server so please hold off installing it

[heliosmedcomms]

One of my clients has their Office 365 backup hosted with iland. They really struggled to do the first backup of about 1TB in SharePoint/Office 365 groups. Took about a week. As mentioned above this appears to be a combination of throttling and poor performance APIs.

It's not IMO a great solution - restoring files using the SharePoint Explorer (VEX) is pretty technical and incredibly slow. I restored a folder today from May that was about 100MB big - bigger though in the backup due to versions. But still took nearly an hour over a 1Gbps internet link...

A product manager at ILand asked me for a critique of the service. It was pretty damning. Although I'm very aware that this is neither ILand or Veeam's fault.

But at least we have ticked the box around having a 3rd party backup. Just hope we never have to test it in earnest as it's just not suitable for disaster recovery. Hell would have frozen over by the time we restored the data if something like Wannacry for hold of SharePoint. Enough to give you nightmares!

Plus these VEX components - hardly user friendly. Kind of expected some kind of web interface to restore data.

[nielsengelen]

Rob, the product offers a RESTful API that allows service providers to build there own offering (web-based) and iLand is one of these looking and implementing it.

Additionally, there is a community-driven project available via GitHub which I've created to get service providers going and gets regular updates.

[heliosmedcomms]

Hi Neil - thanks for the update. Maybe my critique did hit home I'm okay with using the VEX tools because I'm IT but restoring backups is something that one would like to make more accessible to the end-user. For me it's not a problem to remote into the laptop sat on my client's network to initiate a restore. It's still slow but fine for the biggest requirement - restoring files due to user error (accidental delete, corrupt file etc).

Part of this performance hit will be the requirement for the restoring data to travel from IIand's servers across the internet to my client's laptop (albeit over a pretty fast connection https://i.imgur.com/VuHM224.png), into the local Veeam B&R system and then back the other way as it's restored/uploaded to SharePoint.

So if one could trigger the restore from within ILand's infrastructure, then the job would pull data across (one assumes) a very high speed internal network and then across (again one assumes) very big pipe to Office 365. And it sounds like the REST API will be a way to implement some kind of web based interface.

Although one suspects that the biggest bottleneck will still be Veeam @ ILand -> SharePoint...

[AEJ]

I know this is an old post, so forgive me if this has been solved already. We are new to Veeam for Office 365 backup, and have finally gotten everything setup and running.

However, we're seeing incredibly slow backup times sometimes (10-20K per second). This primarily occurs when trying to back up our main sharepoint site which is about 65GB and about 75K items. It took the first backup around 28 hours to complete, and I assumed that future jobs would be much much quicker as it would be incremental at that point. However, the speed appears like another full backup. There aren't many settings to change in the GUI, so I'm wondering if there is anything via powershell that would help to speed things up?

We run the user backup separately, and to backup 10 users' email and one drive took around 6 hours.

Our environment is a WIndows server VM hosted in Azure created specifically for this task, and nothing else runs on the machine. It has a managed premium ssd disk attached to the VM directly, and the repository lives there.

Thanks for any insight in advance!

[ITENOS-Kai]

AEJ,
in the Archiver.Proxy Log you should see around the time the backupjob has finished something like this in the log:
Accounts throttling statistics:
27.07.2021 10:47:27 93 (5588) VBO@example.com: requests 18586, throttled 425

If you're using modern authentication with legacy, you should check out auxiliary backup accounts (https://www.veeam.com/blog/automate-aux ... ffice.html).
If you're using modern authentication only, you have to create more apps through GUI (https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=50)

[nielsengelen]

Extra backup accounts does not assist with Exchange though so if you are seeing slow speeds for this, please open a support case for more insight. Please let us know the case ID for future reference once done.