Maintain control of your Microsoft 365 data
Post Reply
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Is there a concrete fix to the throttling issue?

Post by cparker4486 »

Hello,

I reached out to support (#03134789) to report that the OneDrive repository for my own account was not finishing and would always end with a 500 error. After many days of troubleshooting, and back and forth, support said the problem is on my end (MS's to be specific). That MS is probably throttling my account and that Veeam is unable to detect this exactly but that it's the most likely explanation.

I came to the forum to see what I could find but I found nothing concrete (I don't claim to have found and/or read every single post on the topic). There was talk of changing some settings on the proxy (tried that didn't work) and calling MS to "unthrottle" the tenant (haven't tried yet).

What's Veeam's current best course of action for this?
-- Chris
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler »

Hi Chris,

The "unthrottling" of the tenant works for exchange online (but Microsoft decided on whether to do it or not, not sure what the parameters are for receiving a yes or no...) but not for SharePoint Online. And OneDrive for Business is part of SharePoint Online.


I tried looking at your case but I couldn't find really good information in it so I am hoping that you can help me a bit here.
1. What changes did you make with support (for example, did you change the threads in the proxy?)
2. That specific OneDrive for Business account, does it have a lot of (most probably small) files?
3. Can you give me the exact 500 error?
4. Are the logs in the support call (can't see it for some reason, might be because I am outside the network...)
Thanks
Mike
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

Hi Mike,

Thanks for looking into my issue. To be honest, I don't feel like support is really aware of this issue fully. It took a long time for the rep to come back with the suggestion that it may be MS doing throttling. They also didn't mention that it didn't apply to OneDrive for Business because I don't think they knew about it.

I came to the forum on my own to see if there was any other information out there. The rep didn't suggest changing threads on the proxy. I found that suggestion in the forum. I believe I changed it to 10. There was no change.

I believe the only other change I made was to increase the resources on the proxy but that didn't make any difference. We're a very small network and the proxy is 2016 Server Core that originally had 4GB of memory. I gave it 8GB.

2. Yes. Almost 39,000 with an average of ~1.5MB each.
3. "9/21/2018 9:24:25 AM :: Processing OneDrive username (https://domain-my.sharepoint.com/personal/username) finished with warning: Failed to backup item: , The remote server returned an error: (500) Internal Server Error. :: 3:29:40"

The message makes me think there might be a file that the job can't backup. But that could be a red herring.

4. Yes, I've uploaded logs a few times. I will do it again.


Thank you!
-- Chris
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler »

For 4. I was just asking, if they are there, no need to upload again. (I just can't see them but that is not important ;-))

For the others, let me try and hear if there is already something else known for this
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler »

By the way, is it always the same file or files?
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

I can't say for certain about the file(s) but the error is always the same: "Failed to backup item: ,".
-- Chris
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler »

By the way Chris,
I see the support call is closed? Is that correct? Any reason for it?
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

The last thing support suggested was contrary to guidance they'd given me earlier so I didn't write back and the case closed automatically.
-- Chris
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler »

Hmmm. OK. I will ask someone to look into it. Give me a bit of time (I am at Ignite at the moment)
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

Much appreciated.
-- Chris
Fatboy40
Influencer
Posts: 17
Liked: 1 time
Joined: Mar 02, 2016 10:22 am
Full Name: Clive Arnold
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Fatboy40 »

I use a product, which I won't name, to sync a local folder to OneDrive for Business. It performs a nightly backup of various documents so that we have them for disaster recovery.

However recently I've been getting an increased amount of OneDrive / SharePoint throttling issues as well, causing the sync to fail. Maybe Microsoft have been altering their throttling thresholds recently?
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Polina »

Hi Clive,

It's not that I have heard of. I googled it a bit and it seems that the similar issues with SharePoint Online throttling and the corresponding 429 errors are splashing to multiple customers simultaneously from time to time.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler » 1 person likes this post

@cparker4486
I talked to support and they asked if you could open a request again. Because of privacy regulations and such we have to destroy your logs after the case is closed. (Which is a good thing actually...). But because of that, our higher tier can't dive into the case anymore. Can you open again and let me know. I will then inform the teamlead of the case number so they can check if my feeling is correct...
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

Sorry for the late reply. Thank you, Mike. I will do it today.
-- Chris
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

The case # is 03230102.
-- Chris
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by cparker4486 »

Support responded with a CU download that has a bugfix for the error I'm getting. The job is rerunning now.
-- Chris
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Mike Resseler »

Thanks Chris,

I just passed the case to our support team lead

Brgds,
Mike
mlange
Enthusiast
Posts: 42
Liked: 3 times
Joined: Sep 02, 2014 3:06 pm
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by mlange »

Is this download available? I'm getting the same 500 error on one user.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Polina »

Hi Matt,

Which build are you running? Did you install the latest KB2809?

Thanks!
mlange
Enthusiast
Posts: 42
Liked: 3 times
Joined: Sep 02, 2014 3:06 pm
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by mlange »

Hi Polina,

I'm on 2.0.0.594. I have not installed the latest patch as I didn't know there was one. I'll install it and see what happens. Thanks!
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Polina »

Great, I'm looking forward to getting the news from you. And please make sure prior to installing this patch that you don't have Veeam Backup & Replication 9.5 Update 3 or 3a running on the same server as your VBO.

Thanks!
mlange
Enthusiast
Posts: 42
Liked: 3 times
Joined: Sep 02, 2014 3:06 pm
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by mlange » 1 person likes this post

Still having the same 500 error on one particular user. VBR and VBO are on separate boxes.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Polina »

Matt,

Please open a support case and let our engineers investigate the logs. Also, take a minute to share your case ID here.

Thanks!
mlange
Enthusiast
Posts: 42
Liked: 3 times
Joined: Sep 02, 2014 3:06 pm
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by mlange »

Case ID 03384922.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by Polina »

Hi Matt,

Did you have a chance to follow the support team recommendations? It's indeed likely that you're experiencing the issue covered in KB2745.
mlange
Enthusiast
Posts: 42
Liked: 3 times
Joined: Sep 02, 2014 3:06 pm
Contact:

Re: Is there a concrete fix to the throttling issue?

Post by mlange »

I got in contact with the user today, and while she wasn't having any issues with the file being blocked from download or opening, we have removed the file and am waiting for the next run to see what happens.
Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests