Extremely slow reading from Office 365 cloud

[Robvil]

Hi,

I have opened a case, 02394834, on this.

I just noticed my O365 backup takes forever now .... as in 17B/s in transfer speed ... so a simpel incremental never completes.
It has been running fine until 10/13/17. After this date the transfer speed is nearly zero and has never completet as session.

Ohh, it´s not a local bandwidth issue.

Anyone seeing this?

Is Microsoft somehow inforcing speed transfer limits?

Regards
Robert

[foggy]

Hi Robert, what Veeam Backup for Office 365 version are you using? Any chance this started to occur after the upgrade? Here's another topic with similar issue for more information.

[Robvil]

1.5

It helped to lower the active threads from 64 to 10.
Now 47GB data has been backup.

[Mike Resseler]

Robert,

What is the speed that you see now?

Mike
(PS: you might want to play a bit with those active threads. It is difficult to tell you a formula what the ideal thread # is but playing a bit with it until you find the best setting for your environment is certainly my advice)

[Robvil]

well, i have a 100Mbit/s internet connection and it is using all bandwidth now, so it looks dine.

[Mike Resseler]

Good to hear, although using all the bandwidth is maybe something you also don't want

Note that in 1.5 you can set limits to your bandwidth per proxy: https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=15

Cheers
Mike

[Robvil]

I know ... i limited it to 80Mbit/s after testing with the new thread count.

[ketan.patel]

I am experiencing the same issue. Veeam backup for o365 is extremely slow. its backing up hardly 1.5 MB/sec. we have a case 03138546 open with Veeam and they want me to test the bandwwith betweeen the veeam proxy and o365. I played around with threads but no luck.

has anyone beeen experiencing thee same isssue and any work around? Thanks in advance.

[Polina]

Hi Ket,

If you have a minute to share the outcomes of your case investigation and tests, it'd be much appreciated. Was it because of the bandwidth?
Thanks!

[ketan.patel]

Hi Polina,

Bandwidth between EOL and Veeam proxy aappears to be fine. we have a case open and its not resolved yet. we are still running very low throughput. we have currently 7 jobs running and we are getting only 4-5MB/sec processsing rate. also confirmed with MS and EWS throtteling for our tenant is currently at rest. throttling policy has been disabled for o365 tenant by MS.

[mbraak]

Any update for us?
i also experience very slow backup performance (sharepoint backup)
Working with veeam and MS support but it seems that it isn't possible to speed up Office 365 Sharepoint backups. This makes Sharepoint useless for larger sites for now.

Looking forward to your reply.

MBR

[Mike Resseler]

Hey MBR,

SharePoint indeed cannot be changed from the MSFT side (maybe it can but they won't). We are actively seeking to improve this and are researching different things within R&D. Unfortunately we haven't found a good solution for now. We are using all the best practices from MSFT around connecting to SharePoint Online and downloading data, but as you state yourself, sometimes it is extremely slow.

Do you already have a support case? We can use more logs to investigate and to see what is going on. From the logs we already have, it seems that there probably isn't a one-fit solution. It depends on the situation.

[vipthomps]

Ketan,

What did you put in as a request with Microsoft to have the throttling policy disabled? Thanks!

[Mike Resseler]

Eric,

This works only for Exchange and requires a support call. The problem we see is that MSFT does not want to do this for everyone. We first thought it was depending on your O365 size but that doesn't seem to be the case. What exact parameters they are using to allow or disallow it is not known (and they won't tell us )

[tbeekmans]

I'm encountering the same problem, SharePoint backups are extremely slow between 30KB/s to 100KB/s, we just transferred 9 GB in 3 days, I opened a case at MS to see if there is any throttling (keep you all updated), bandwidth or resources aren't any issues on our backup server, seems more the limitation on the source or the API, I really think it's really necessary that Veeam and Microsoft get together for this problem, product isn't ready for full backup of large Office 365 SharePoint organisations yet, I tried to backup or complete SharePoint Office 365 organisation around 500GB after 2 weeks I stopped it didn't work.

[Polina]

Hi Tom and welcome to the community!

Thank you for sharing this feedback. Please let us know what MS responds. Our team is researching possible options to address the challenge with SharePoint throttling and it's helpful to know the specific use cases.
Meanwhile, may I ask if you opened any cases with Veeam Support on this? and also did you try to play with the number of threads in proxy settings to improve the backup performance?

Thanks

[mbraak]

I had a case at Microsoft but that didn't gave me anything. Sharepoint Throttling is fixed and wont be release anywhere near future.
Veeam Support also officially stated Sharepoint Backups are currently not there best implementation and a lot of customers are complaining about it.
Development is researching this problem and are searching for improvementste be make but this wont be anytime near future also.
I do have one other contact (Veeam System Engineer) which is informing internally to see if there is anything left to improve things for us.
I hope to hear from him anywhere next week.

In advance i am evaluating an other promising Office 365 product atm of which the first performance tests are very promising.
As soon as i have more concrete information i will let you all know.

[Mike Resseler]

Hi Mbraak,

Allow me to correct one thing.

1. Official support statement: (Please DM me your support case because it is not correct). We are using the correct implementation per Microsoft recommendation (and request) to connect to SharePoint.

2. We are indeed working on improvements and getting it faster. We are also working together with MSFT around this. Not anytime near future might be faster than you think. (Hence the reason why we ask logs from everyone who has very large document libraries and is suffering from this. Because that seems to be the root cause). I do not expect it to take 6 months to build improvements.

3. Your Veeam System Engineer will need to find me

Cheers
Mike

[whimark]

I was told by Microsoft support that by using several accounts could increase performance, so I would suggest that it would be possible to assign more Office365 accounts to the Veeam proxies for the same organisation like below.

Also they said that if to many request comes from the same account to sharepoint they will cut the performance for that user account

Proxy one: Veeam1@zxczc.onmicrosoft.com
Proxy two: Veeam2@zxczc.onmicrosoft.com
Proxy three: Veeam3@zxczc.onmicrosoft.com

[Polina]

Hi Markus,

That's correct, as when you're using multiple accounts you can "distribute requests load" on the service and "postpone" throttling. However, there are both pros and cons to consider.
As Mike wrote earlier, we're currently working on some enhancements to speed up processing.

[tbeekmans]

Good news, a good improvement on the backup time for SharePoint backup, Veeam provided a hotfix (Fix_149892_211cc6a369) this fix is still not released officially but will be soon for more customers hopefully.
The speed isn’t extremely high however it seems that the problem more was the high amount of items in SharePoint that needed to be backup-ed at the end.
The backup did run normally 2-3 days with the fix 1 hour, big improvement.
Again Veeam R&D anticipates very fast on problems with products, thanks for that.
I also did create a case at Microsoft however this moves very slowly to get something done at the throttling and I don’t expect many improvement on this topic.

[Polina]

Hi Tom,

Thanks for this feedback!
You're right, the fix is currently provided upon request only, as we would like to look more closely on the results in real-life environments. And it's great to know you're satisfied with the outcome. Could you please share your case ID here? this would help us to keep in sync with the support team

[whimark]

Hi Polina,

Is it possible to send the hotfix to me?

//Markus

[Polina]

Hi Markus,

Fixes can only be provided by our support team; please create a support call and post your ID in this thread, so that we could track it.
Thanks!

[tbeekmans]

The case number regarding the hotfix was, 03279696

[Polina]

Thanks, Tom!

[whimark]

There was a slight performance gain with the new hotfix, but in my case being able to assign different accounts to the proxies would probably solve most of the issues I have with the performance when backing up Sharepoint sites.

[Polina]

Thank you for the feedback, Markus. Our R&D team continues their work on performance enhancements and all the possible options are being tested now.

[MILJW002]

Hi,

I’ve had slow backups for SharePoint and OneDrive since first installing version 2. I’ve played with reducing the thread count, but it hasn’t made a significant difference.

I’ve just logged a case as I’d love to try the fix to see if it helps.

03338555

James

[Polina]

Hi James,

It looks that the fix has been already provided to you. Can you post some updates on performance improvements once it's applied?

Thanks