Maintain control of your Microsoft Office 365 data
MILJW002
Enthusiast
Posts: 36
Liked: 3 times
Joined: Apr 29, 2017 2:26 pm
Full Name: James Miller
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by MILJW002 »

Hi Polina - you are just a bit ahead of Peter (the engineer helping me). I’ve provided the logs etc, but I don’t have the fix yet. Yep, I’ll happily share the results once I’ve been able to test the fix.

MILJW002
Enthusiast
Posts: 36
Liked: 3 times
Joined: Apr 29, 2017 2:26 pm
Full Name: James Miller
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by MILJW002 »

Hi Polina,

Just a quick update. I received the fix this afternoon, so have been testing. The OneDrive for Business backup (with no changed data) has gone from 3.5 - 4 hours to between 1:40 hours to 2 hours. So, first impressions are it looks better.

I did however notice that the bandwidth consumption keeps bursting, so working with support to see if I can tune any other settings.

So, def an improvement, but still investigating.

J

Polina
Veeam Software
Posts: 1904
Liked: 395 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by Polina »

Thank you for this news, James!
VBO proxy settings allow for limiting the bandwidth consumption if needed. Nevertheless, please continue working with support to optimize your setup.

MILJW002
Enthusiast
Posts: 36
Liked: 3 times
Joined: Apr 29, 2017 2:26 pm
Full Name: James Miller
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by MILJW002 »

Hi Polina,

I don’t need to limit the bandwidth consumption as it’s not saturating the link (yet).

This is running on an ADSL connection (I know, slow) that is capable of 23 Mbps down and 1 Mbps up.

Before the patch it was barely parsing traffic, but now it is saturating the upstream (approx 1 Mbps) for a couple of seconds then dipping down to around 200 Kbps. Before it didn’t really get above about 200 Kbps. My thought was if the Microsoft throttling was now not kicking in then it should saturate the 1 Mbps upstream while the job is running.

If I can get to the stage of it saturating the link I can then look at limiting the bandwidth.

As it is bursting, I’m wondering if the throttling is still kicking in, or if I need more/less worker threads (tuning). This is what I’m currently working with support on.

Overall it is a lot better, but I don’t think it is completely right yet.

J

Polina
Veeam Software
Posts: 1904
Liked: 395 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by Polina »

James,

How exactly are you tracking traffic/bandwidth saturation? Are you looking at the Read/Processing rates in the VBO console?

MILJW002
Enthusiast
Posts: 36
Liked: 3 times
Joined: Apr 29, 2017 2:26 pm
Full Name: James Miller
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by MILJW002 »

I’ve been using Task Manager and PerfMon on the server and net flow logs from the router to monitor the actual data throughput. The Read/Processing rates in the VBO console are much lower, so not really reporting how much data is going over the wire for a backup.

Overall things are looking a lot better.

Mike Resseler
Product Manager
Posts: 7095
Liked: 973 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by Mike Resseler »

Thanks for the update James.

For all on this thread, besides HAPPY new year ;-), the MSP with the fixes is out: please go and visit https://www.veeam.com/kb2809 to get a cumulative update pack with hotfixes.

roelvdw
Enthusiast
Posts: 60
Liked: 6 times
Joined: May 11, 2012 1:38 pm
Full Name: Vandewalle Roel
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by roelvdw »

Hi Mike

The KB article says you need Veeam B&R 9.5 U4 in combination with the new update with office 365 v2.
But how can we get the update 4 for Veeam backup and replication ?

Kind regards

Roel

Mike Resseler
Product Manager
Posts: 7095
Liked: 973 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by Mike Resseler »

You don't actually need it. It is necessary on top of update 4 if you are running VBR and VBO together. The patch is also intended to make sure that the explorers from VBR 9.5U4 and VBO 2.0 are the same (the products share the Veeam explorers)

roelvdw
Enthusiast
Posts: 60
Liked: 6 times
Joined: May 11, 2012 1:38 pm
Full Name: Vandewalle Roel
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by roelvdw »

Hi Mike

I have installed VBR & VBO365 on the same physical server .
Where can i find VBR 9.5 U4 ?

Kind regards

Roel

Mike Resseler
Product Manager
Posts: 7095
Liked: 973 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by Mike Resseler »

Roel,

VBR 9.5 U4 will be out very soon.

roelvdw
Enthusiast
Posts: 60
Liked: 6 times
Joined: May 11, 2012 1:38 pm
Full Name: Vandewalle Roel
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by roelvdw »

Hi Mike

Do i have to wait on VBR 9.5 U4 before installing VBO365 update ?

Kind regards

Roel

Mike Resseler
Product Manager
Posts: 7095
Liked: 973 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by Mike Resseler »

Roel,

Yes, they are on the same server so please hold off installing it

heliosmedcomms
Influencer
Posts: 13
Liked: never
Joined: Dec 22, 2018 4:03 pm
Full Name: Rob Nicholson
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by heliosmedcomms »

One of my clients has their Office 365 backup hosted with iland. They really struggled to do the first backup of about 1TB in SharePoint/Office 365 groups. Took about a week. As mentioned above this appears to be a combination of throttling and poor performance APIs.

It's not IMO a great solution - restoring files using the SharePoint Explorer (VEX) is pretty technical and incredibly slow. I restored a folder today from May that was about 100MB big - bigger though in the backup due to versions. But still took nearly an hour over a 1Gbps internet link...

A product manager at ILand asked me for a critique of the service. It was pretty damning. Although I'm very aware that this is neither ILand or Veeam's fault.

But at least we have ticked the box around having a 3rd party backup. Just hope we never have to test it in earnest as it's just not suitable for disaster recovery. Hell would have frozen over by the time we restored the data if something like Wannacry for hold of SharePoint. Enough to give you nightmares!

Plus these VEX components - hardly user friendly. Kind of expected some kind of web interface to restore data.

nielsengelen
Veeam Software
Posts: 4508
Liked: 926 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by nielsengelen »

Rob, the product offers a RESTful API that allows service providers to build there own offering (web-based) and iLand is one of these looking and implementing it.

Additionally, there is a community-driven project available via GitHub which I've created to get service providers going and gets regular updates.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

heliosmedcomms
Influencer
Posts: 13
Liked: never
Joined: Dec 22, 2018 4:03 pm
Full Name: Rob Nicholson
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by heliosmedcomms »

Hi Neil - thanks for the update. Maybe my critique did hit home :-) I'm okay with using the VEX tools because I'm IT but restoring backups is something that one would like to make more accessible to the end-user. For me it's not a problem to remote into the laptop sat on my client's network to initiate a restore. It's still slow but fine for the biggest requirement - restoring files due to user error (accidental delete, corrupt file etc).

Part of this performance hit will be the requirement for the restoring data to travel from IIand's servers across the internet to my client's laptop (albeit over a pretty fast connection https://i.imgur.com/VuHM224.png), into the local Veeam B&R system and then back the other way as it's restored/uploaded to SharePoint.

So if one could trigger the restore from within ILand's infrastructure, then the job would pull data across (one assumes) a very high speed internal network and then across (again one assumes) very big pipe to Office 365. And it sounds like the REST API will be a way to implement some kind of web based interface.

Although one suspects that the biggest bottleneck will still be Veeam @ ILand -> SharePoint...

AEJ
Lurker
Posts: 1
Liked: never
Joined: Jul 23, 2021 4:16 pm
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by AEJ »

I know this is an old post, so forgive me if this has been solved already. We are new to Veeam for Office 365 backup, and have finally gotten everything setup and running.

However, we're seeing incredibly slow backup times sometimes (10-20K per second). This primarily occurs when trying to back up our main sharepoint site which is about 65GB and about 75K items. It took the first backup around 28 hours to complete, and I assumed that future jobs would be much much quicker as it would be incremental at that point. However, the speed appears like another full backup. There aren't many settings to change in the GUI, so I'm wondering if there is anything via powershell that would help to speed things up?

We run the user backup separately, and to backup 10 users' email and one drive took around 6 hours.

Our environment is a WIndows server VM hosted in Azure created specifically for this task, and nothing else runs on the machine. It has a managed premium ssd disk attached to the VM directly, and the repository lives there.

Thanks for any insight in advance!

ITENOS-Kai
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 17, 2020 10:13 am
Full Name: Kai Weldert
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by ITENOS-Kai »

AEJ,
in the Archiver.Proxy Log you should see around the time the backupjob has finished something like this in the log:
Accounts throttling statistics:
27.07.2021 10:47:27 93 (5588) VBO@example.com: requests 18586, throttled 425

If you're using modern authentication with legacy, you should check out auxiliary backup accounts (https://www.veeam.com/blog/automate-aux ... ffice.html).
If you're using modern authentication only, you have to create more apps through GUI (https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=50)

nielsengelen
Veeam Software
Posts: 4508
Liked: 926 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Extremely slow reading from Office 365 cloud

Post by nielsengelen »

Extra backup accounts does not assist with Exchange though so if you are seeing slow speeds for this, please open a support case for more insight. Please let us know the case ID for future reference once done.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

Post Reply

Who is online

Users browsing this forum: Majestic-12 [Bot] and 12 guests