VB365: The remote server returned an error: (429)

[dali@iae.nl]

After installing version 7 I get some new nearly the same errors, not 410 but now 429:
Failed to process site: <user> (https://<domain>-my.sharepoint.com/personal/<user+domain>). The remote server returned an error: (429).

Is this also a known issue?

[Mildur]

Hi Da Li

Error 429 means, you are getting throttled by Microsoft. Unfortunately a hotfix won't solve this issue.
Please open a case with our support team and let them confirm it by the logs.

Best,
Fabian

[dali@iae.nl]

Now created Case #05911452 - The remote server returned an error: (429).

[Mildur]

Thanks,
I split this topic from the other one because of the different cause of the issue.

Best,
Fabian

[dali@iae.nl]

Solution was to add more Backup Applications (have now 8 instead of 1) because it seems like a throttle issue at Microsoft site.
(But why this was no issue in version 6 and now popped up in v7 keeps unsure).

[jfrobs]

Hello Same problem here.

Upgrade to V7 and : The remote server returned an error: (429) for some users during the first backup. And for the last backup launched I have only one more with this error.

I don't understand the solution.
How to increase the Backup Application?

thx

[Mildur]

Hi jfrobs

Yes, the steps are in our user guide. But please be careful and first read our statement:
Impact of Multiple Backup Applications on Performance

Microsoft restrictions allow us to us multiple applications only for an initial backup. Incremental backups should not use multiple backup applications. If you get the 429 error, you should open a support case and create the accounts under supervision of our Support specialists.

The steps on how to create such applications are documented here:
https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=70

Best,
Fabian

[jfrobs]

hello,

I have only configured one connector since veeam365 went live in version 6 so I am within the requirements for version 7.

Maybe the error will disappear at the next incremental backup?

[Polina]

Hi,

First full backups are affected by throttling much heavier, but it may occur on incremental runs as well. It depends on the number of changes and the amount of new data that appeared between backup job runs. By adding a couple of auxiliary applications you will distribute requests to SharePoint between the apps, and hence reduce the number of 429 errors.

[renatorichina]

I have the same issue and it just start to appear after upgrading from V6 to V7. I have currently an open support case #05956916. The recommendation was to split jobs and add 5-10 application object. Done both (added 10 AO) but the 429 still appears constantly. Looks really like an issue in V7, as in V6, i didn't have the problem

[Polina]

Hi Renato,

We currently see an increased number of issues with heavier throttling, and it's now being investigated by RnD. I cannot yet confirm if the problem is on Veeam's side or if something has unexpectedly changed on the M365 side, but I hope to get the updates in the next few days. For now, please follow the recommendations provided by our support team.

Thanks!

[jeff.kuhn]

Recently upgraded to 7 and seeing 429 errors. Following this thread.

[TheBrain82]

Same here, seeing hundrets of "The remote server returned an error: (429)" for all of my customer.
This comes up addiionally to the known. "Warning: The remote server returned an error: (503) Server Unavailable." and "Server does not provide the recommended delay" and "Warning: Web request failed" and "Warning: Unable to process request. Server cannot service the request".

ButI do also confirm, that in v7 "The remote server returned an error: (429)" shows up much more than in V6 for any reason!!

[lasseoe]

Hi,

I've had a case open with support for a bit now and their suggestions have made little to no difference, so I'm posting here in case someone's experiencing similar issues and have an idea what might be the cause of this.

After upgrading to V7 a single tenant is experiencing 429 throttling and 503 errors, Veeam support suggested lowering proxy threads from 64 to 32 which made a slight difference but nothing that really mattered. The tenant has 16 aux backup applications and support thought that was plenty. Looking at the logs only 0,3-0,5% of requests are throttled.
Lowering threads doesn't feel like a proper way to handle this as it's for the entire proxy and not just a single tenant, and again.. it worked before.

The thing that strikes me the most, is that this appeared after the V7 upgrade, and only for one particular tenant. While the tenant is rather large, it did work just fine before the upgrade.

I have a few tenants without aux accounts, they get throttled but report no issues at all even though they are throttled a lot more (~4% of requests).

Case ID: 05962737

[Mildur]

Hi Guys

RnD is investigating this issues.
Please log a support case if you don't have done so already.

@lasseoe
I moved your post to this topic.

Best,
Fabian

[Nossa30]

Case # 05975783
We are having the exact same problem. ALL jobs being throttled with the same error. I've created support case with Veaam already.

I've tried adding my own throttling by reducing threads all the way down to 16 threads. I've tried throttling network traffic all the way down to 3Mb/s but nothing seems to work so far since upgrading to V7.

Veeam rep suggested that:

"Please increase retries timeout by adding this key SPDownloadMaxRetryWait="30" to the Proxy.xml file."
Tried that fix, but also didn't seem to resolve the issue.

[Polina]

Hi All,

The hotfix is now available via support and soon will be delivered in a new patch.

[Trillian]

We also had this problem after upgrading from v6 to v7. Opened a support Case (# 05988181) and received a fix within 10 minutes. Replaced 3 dll files and everything works again now.

[juerg.hirschi]

Can confirm as well, that the hotfix resolves the issue. all good again.

here it took only 3min to receive the instructions

[nicolajnielsen]

@Polina would you be able to share some details about what the issue were and how it was fixed?

I've seen a pattern in the logs (ProgramData\Veeam\Backup365\Logs\) which looks like that vb365 on V6, doesn't adhere back off requests (Retry-After HTTP Header), and seem to implement own desired cool down when experiencing an 427 and 503 but nothing shown in the task log which is found quite odd, such things should properly be an warning.

Could you demystify this?

[Polina]

Hi Nicolaj,

The problem was trivial but not that easy to find - the default wait time before sending the next request in case of throttling was decreased, and instead of 'cooling down' we were overheating the server.

[cerberus]

We are seeing the same issue "The remote server returned an error. (429)" when backing up SharePoint sites, any ETA on patch?

[Oerten]

Same issue here. Especially when performing initial full backups.

[Polina]

All,

You get the hotfix immediately via a support case.
A patch is on also its way but we will need a few more days to prepare it.

[Polina]

A small update
We published a KB for this specific issue: https://www.veeam.com/kb4442

[sumeet]

Shouldn't this have been caught in your testing.

I understand that basic testing would happen on a M365 tenancy that does not have much data, so issue not observed.

But mostly Veeam Engg and over all Veeam Organization should have their mailboxes/onedrive/sharepoint on M365 and hopefully be using Veeam backup for M365 to backup the data. Unless you do not follow - eat your own cookie and do not trust your own product. If your own data is in M365, then prior to the release of the product, definitely you should be deploying a fresh installation and backing up the full data. If this is done, such issues should be caught before reaching out to us.

There can be a possibility that your veeam Org does not use M365 and instead use on-premise exchange or some other mail server. In such cases, you should be setting up a dummy M365 tenancy with few hundred dummy mailboxes/onedrive/sharepoint, which have few hundred GBs of data, or maybe few TBs. This much you should be able to put in the cost if you are having a product like this. And as mentioned earlier, prior to release of the product run a full first backup run.

This is the basic level of testing that should happen to catch a problem that you say was trivial. Such trivial problem of yours causes us to loose out on our SLA's and the time wasted to work with your support to get this resolved.

I upgraded to version 6 in Oct last year (after having waited for many months) and still have issues that I'm working with your support to resolve (current case #05929428). The thought of going to v7 is scary for me, especially looking at this thread.

Specially when you say - the problem was trivial - done and dusted.

[Mike Resseler]

Our Veeam org is actually using our own product . And we do test with tenants that are far larger than a few hundred mailboxes and so on. What happened is that a retry timeout was changed which initially in our testing proved to be a performance improvement in a lot of our tenants. And even with some tenants now, we still have seen performance improvements until we hit a certain threshold which causes the 429 errors. Unfortunately, hitting that threshold is not always from our solution but can also be because other 3rd party solutions or tools are getting or putting information on SPO. I certainly wished we had discovered this earlier, but unfortunately we did not. And we do apologize for this.

[LeeMackie]

Does this issue cause/fix also apply for Exchange backups? I note that the hotfix seems to apply specifically for Sharepoint and Teams.

Seeing the same issue with a single tenant and the support case indicated it was a throttling issue and to look into opening a case with Microsoft.

[Polina]

Hi Lee,

Both the issue and the fix were specific to SharePoint processing.
With Exchange, it's possible to temporarily release throttling as described here: https://www.veeam.com/kb4198. This way, you'll be able to do a heavy job of the first full backup and then, when policies are back, continue doing incremental runs that typically are less impacted by throttling.

[LeeMackie]

Thanks for confirming Polina, I've been through that article and the tenancy doesn't report that throttling is in effect, and it is throwing the error on incremental run not a full.

I guess this issue is something else and I'll have to go back down the support path.