Maintain control of your Microsoft 365 data
dali@iae.nl
Enthusiast
Posts: 74
Liked: 15 times
Joined: Jan 17, 2022 10:31 am
Full Name: Da Li
Contact:

VB365: The remote server returned an error: (429)

Post by dali@iae.nl »

After installing version 7 I get some new nearly the same errors, not 410 but now 429:
Failed to process site: <user> (https://<domain>-my.sharepoint.com/personal/<user+domain>). The remote server returned an error: (429).

Is this also a known issue?
Mildur
Product Manager
Posts: 8913
Liked: 2357 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Failed to process team: <team name> UnknownError. The remote server returned an error: (410

Post by Mildur »

Hi Da Li

Error 429 means, you are getting throttled by Microsoft. Unfortunately a hotfix won't solve this issue.
Please open a case with our support team and let them confirm it by the logs.

Best,
Fabian
Product Management Analyst @ Veeam Software
dali@iae.nl
Enthusiast
Posts: 74
Liked: 15 times
Joined: Jan 17, 2022 10:31 am
Full Name: Da Li
Contact:

Re: Failed to process team: <team name> UnknownError. The remote server returned an error: (410

Post by dali@iae.nl » 1 person likes this post

Now created Case #05911452 - The remote server returned an error: (429).
Mildur
Product Manager
Posts: 8913
Liked: 2357 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Mildur »

Thanks,
I split this topic from the other one because of the different cause of the issue.

Best,
Fabian
Product Management Analyst @ Veeam Software
dali@iae.nl
Enthusiast
Posts: 74
Liked: 15 times
Joined: Jan 17, 2022 10:31 am
Full Name: Da Li
Contact:

Re: VB365: The remote server returned an error: (429)

Post by dali@iae.nl »

Solution was to add more Backup Applications (have now 8 instead of 1) because it seems like a throttle issue at Microsoft site.
(But why this was no issue in version 6 and now popped up in v7 keeps unsure).
jfrobs
Novice
Posts: 5
Liked: never
Joined: Oct 15, 2019 12:52 pm
Full Name: jfrobs
Contact:

Re: VB365: The remote server returned an error: (429)

Post by jfrobs »

Hello Same problem here.

Upgrade to V7 and : The remote server returned an error: (429) for some users during the first backup. And for the last backup launched I have only one more with this error.

I don't understand the solution.
How to increase the Backup Application?

thx
Mildur
Product Manager
Posts: 8913
Liked: 2357 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Mildur »

Hi jfrobs

Yes, the steps are in our user guide. But please be careful and first read our statement:
Impact of Multiple Backup Applications on Performance

Microsoft restrictions allow us to us multiple applications only for an initial backup. Incremental backups should not use multiple backup applications. If you get the 429 error, you should open a support case and create the accounts under supervision of our Support specialists.

The steps on how to create such applications are documented here:
https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=70

Best,
Fabian
Product Management Analyst @ Veeam Software
jfrobs
Novice
Posts: 5
Liked: never
Joined: Oct 15, 2019 12:52 pm
Full Name: jfrobs
Contact:

Re: VB365: The remote server returned an error: (429)

Post by jfrobs »

hello,

I have only configured one connector since veeam365 went live in version 6 so I am within the requirements for version 7.

Maybe the error will disappear at the next incremental backup?
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina »

Hi,

First full backups are affected by throttling much heavier, but it may occur on incremental runs as well. It depends on the number of changes and the amount of new data that appeared between backup job runs. By adding a couple of auxiliary applications you will distribute requests to SharePoint between the apps, and hence reduce the number of 429 errors.
renatorichina
Influencer
Posts: 10
Liked: 2 times
Joined: Mar 06, 2015 1:55 pm
Full Name: Renato Richina
Contact:

Re: VB365: The remote server returned an error: (429)

Post by renatorichina »

I have the same issue and it just start to appear after upgrading from V6 to V7. I have currently an open support case #05956916. The recommendation was to split jobs and add 5-10 application object. Done both (added 10 AO) but the 429 still appears constantly. Looks really like an issue in V7, as in V6, i didn't have the problem
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina »

Hi Renato,

We currently see an increased number of issues with heavier throttling, and it's now being investigated by RnD. I cannot yet confirm if the problem is on Veeam's side or if something has unexpectedly changed on the M365 side, but I hope to get the updates in the next few days. For now, please follow the recommendations provided by our support team.

Thanks!
jeff.kuhn
Lurker
Posts: 1
Liked: never
Joined: Sep 26, 2022 7:11 pm
Full Name: Jeff Kuhn
Contact:

Re: VB365: The remote server returned an error: (429)

Post by jeff.kuhn »

Recently upgraded to 7 and seeing 429 errors. Following this thread.
TheBrain82
Influencer
Posts: 14
Liked: never
Joined: Mar 01, 2022 1:53 pm
Full Name: Oleg the Brain
Contact:

Re: VB365: The remote server returned an error: (429)

Post by TheBrain82 »

Same here, seeing hundrets of "The remote server returned an error: (429)" for all of my customer.
This comes up addiionally to the known. "Warning: The remote server returned an error: (503) Server Unavailable." and "Server does not provide the recommended delay" and "Warning: Web request failed" and "Warning: Unable to process request. Server cannot service the request".

ButI do also confirm, that in v7 "The remote server returned an error: (429)" shows up much more than in V6 for any reason!!
lasseoe
Service Provider
Posts: 76
Liked: 7 times
Joined: Dec 17, 2012 4:39 pm
Full Name: Lasse Osterild
Location: Denmark
Contact:

[MERGED] V7 throttling leading to errors

Post by lasseoe »

Hi,

I've had a case open with support for a bit now and their suggestions have made little to no difference, so I'm posting here in case someone's experiencing similar issues and have an idea what might be the cause of this.

After upgrading to V7 a single tenant is experiencing 429 throttling and 503 errors, Veeam support suggested lowering proxy threads from 64 to 32 which made a slight difference but nothing that really mattered. The tenant has 16 aux backup applications and support thought that was plenty. Looking at the logs only 0,3-0,5% of requests are throttled.
Lowering threads doesn't feel like a proper way to handle this as it's for the entire proxy and not just a single tenant, and again.. it worked before.

The thing that strikes me the most, is that this appeared after the V7 upgrade, and only for one particular tenant. While the tenant is rather large, it did work just fine before the upgrade.

I have a few tenants without aux accounts, they get throttled but report no issues at all even though they are throttled a lot more (~4% of requests).

Case ID: 05962737
Mildur
Product Manager
Posts: 8913
Liked: 2357 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Mildur » 1 person likes this post

Hi Guys

RnD is investigating this issues.
Please log a support case if you don't have done so already.

@lasseoe
I moved your post to this topic.

Best,
Fabian
Product Management Analyst @ Veeam Software
Nossa30
Lurker
Posts: 1
Liked: never
Joined: Mar 29, 2023 12:25 pm
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Nossa30 »

Case # 05975783
We are having the exact same problem. ALL jobs being throttled with the same error. I've created support case with Veaam already.

I've tried adding my own throttling by reducing threads all the way down to 16 threads. I've tried throttling network traffic all the way down to 3Mb/s but nothing seems to work so far since upgrading to V7.

Veeam rep suggested that:

"Please increase retries timeout by adding this key SPDownloadMaxRetryWait="30" to the Proxy.xml file."
Tried that fix, but also didn't seem to resolve the issue.
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina » 1 person likes this post

Hi All,

The hotfix is now available via support and soon will be delivered in a new patch.
Trillian
Lurker
Posts: 1
Liked: 1 time
Joined: Apr 03, 2023 6:56 am
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Trillian » 1 person likes this post

We also had this problem after upgrading from v6 to v7. Opened a support Case (# 05988181) and received a fix within 10 minutes. Replaced 3 dll files and everything works again now.
juerg.hirschi
Service Provider
Posts: 21
Liked: 2 times
Joined: Jan 09, 2023 6:41 pm
Full Name: Jürg
Location: Switzerland
Contact:

Re: VB365: The remote server returned an error: (429)

Post by juerg.hirschi » 1 person likes this post

Can confirm as well, that the hotfix resolves the issue. all good again.

here it took only 3min to receive the instructions ;-)
nicolajnielsen
Service Provider
Posts: 1
Liked: never
Joined: Dec 14, 2022 7:43 am
Full Name: Nicolaj Nielsen
Contact:

Re: VB365: The remote server returned an error: (429)

Post by nicolajnielsen »

@Polina would you be able to share some details about what the issue were and how it was fixed?

I've seen a pattern in the logs (ProgramData\Veeam\Backup365\Logs\) which looks like that vb365 on V6, doesn't adhere back off requests (Retry-After HTTP Header), and seem to implement own desired cool down when experiencing an 427 and 503 but nothing shown in the task log which is found quite odd, such things should properly be an warning.

Could you demystify this?
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina »

Hi Nicolaj,

The problem was trivial but not that easy to find - the default wait time before sending the next request in case of throttling was decreased, and instead of 'cooling down' we were overheating the server.
cerberus
Expert
Posts: 155
Liked: 15 times
Joined: Aug 28, 2015 2:45 pm
Full Name: Mirza
Contact:

Re: VB365: The remote server returned an error: (429)

Post by cerberus »

We are seeing the same issue "The remote server returned an error. (429)" when backing up SharePoint sites, any ETA on patch?
Oerten
Service Provider
Posts: 12
Liked: 1 time
Joined: Nov 13, 2018 12:27 pm
Full Name: Fredrik Örtenholm
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Oerten »

Same issue here. Especially when performing initial full backups.
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina »

All,

You get the hotfix immediately via a support case.
A patch is on also its way but we will need a few more days to prepare it.
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina »

A small update
We published a KB for this specific issue: https://www.veeam.com/kb4442
sumeet
Service Provider
Posts: 125
Liked: 20 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: VB365: The remote server returned an error: (429)

Post by sumeet »

Shouldn't this have been caught in your testing.

I understand that basic testing would happen on a M365 tenancy that does not have much data, so issue not observed.

But mostly Veeam Engg and over all Veeam Organization should have their mailboxes/onedrive/sharepoint on M365 and hopefully be using Veeam backup for M365 to backup the data. Unless you do not follow - eat your own cookie and do not trust your own product. If your own data is in M365, then prior to the release of the product, definitely you should be deploying a fresh installation and backing up the full data. If this is done, such issues should be caught before reaching out to us.

There can be a possibility that your veeam Org does not use M365 and instead use on-premise exchange or some other mail server. In such cases, you should be setting up a dummy M365 tenancy with few hundred dummy mailboxes/onedrive/sharepoint, which have few hundred GBs of data, or maybe few TBs. This much you should be able to put in the cost if you are having a product like this. And as mentioned earlier, prior to release of the product run a full first backup run.

This is the basic level of testing that should happen to catch a problem that you say was trivial. Such trivial problem of yours causes us to loose out on our SLA's and the time wasted to work with your support to get this resolved.

I upgraded to version 6 in Oct last year (after having waited for many months) and still have issues that I'm working with your support to resolve (current case #05929428). The thought of going to v7 is scary for me, especially looking at this thread.

Specially when you say - the problem was trivial - done and dusted.
Mike Resseler
Product Manager
Posts: 8048
Liked: 1264 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Mike Resseler »

Our Veeam org is actually using our own product :-). And we do test with tenants that are far larger than a few hundred mailboxes and so on. What happened is that a retry timeout was changed which initially in our testing proved to be a performance improvement in a lot of our tenants. And even with some tenants now, we still have seen performance improvements until we hit a certain threshold which causes the 429 errors. Unfortunately, hitting that threshold is not always from our solution but can also be because other 3rd party solutions or tools are getting or putting information on SPO. I certainly wished we had discovered this earlier, but unfortunately we did not. And we do apologize for this.
LeeMackie
Service Provider
Posts: 20
Liked: 2 times
Joined: May 25, 2022 12:37 am
Full Name: Lee Mackie
Contact:

Re: VB365: The remote server returned an error: (429)

Post by LeeMackie »

Does this issue cause/fix also apply for Exchange backups? I note that the hotfix seems to apply specifically for Sharepoint and Teams.

Seeing the same issue with a single tenant and the support case indicated it was a throttling issue and to look into opening a case with Microsoft.
Polina
Veeam Software
Posts: 3014
Liked: 714 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VB365: The remote server returned an error: (429)

Post by Polina »

Hi Lee,

Both the issue and the fix were specific to SharePoint processing.
With Exchange, it's possible to temporarily release throttling as described here: https://www.veeam.com/kb4198. This way, you'll be able to do a heavy job of the first full backup and then, when policies are back, continue doing incremental runs that typically are less impacted by throttling.
LeeMackie
Service Provider
Posts: 20
Liked: 2 times
Joined: May 25, 2022 12:37 am
Full Name: Lee Mackie
Contact:

Re: VB365: The remote server returned an error: (429)

Post by LeeMackie »

Thanks for confirming Polina, I've been through that article and the tenancy doesn't report that throttling is in effect, and it is throwing the error on incremental run not a full.

I guess this issue is something else and I'll have to go back down the support path.
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests