[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

[scidoner]

Just came here to say I had the same problem with Wasabi S3, which is a supported object storage target.

No one (even Wasabi support) could resolve this issue. In the end, I put it down to some kind of problem with WinHTTP and switched to a Linux gateway to proxy the S3 repository data and have not had an issue since.

[Mildur]

Hi scidoner

Did you have opened a case with us? Which I can review?

Best,
Fabian

[scidoner]

@Mildur - Nah, we weren't convinced it was a Veeam root cause at the time, more likely that Windows HTTP implementation is rubbish.

[MrSpock]

Since we moved our backup offloading from Amazon S3 to Wasabi, we have seen a significant number of the following errors in our Veeam logs during cleanup:

Code: Select all

[01.07.2024 04:56:26.063] < 16156> aws      | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [8932] seconds
[01.07.2024 04:56:26.063] < 16156> aws      | >>  |WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

[01.07.2024 04:56:26.063] < 16156> aws      | Retrying with AccessKey [<removed>]
[01.07.2024 04:56:27.391] <  4860> net      |   Sending request. URI: [/<removed>?delete]. Method: [POST]. Request: [DeleteObjects]
[01.07.2024 04:56:27.391] <  4860> net      |   Headers:
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Length: 336036
[01.07.2024 04:56:27.391] <  4860> net      |     Content-MD5: <removed>
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Type: text/plain; charset=utf-8
[01.07.2024 04:56:27.391] <  4860> net      |     Host: s3.eu-central-2.wasabisys.com
[01.07.2024 04:56:27.391] <  4860> net      |     User-Agent: APN/1.0 Veeam/1.0 Backup/12.0
[01.07.2024 04:56:27.391] <  4860> net      |   Body:

This error is logged when Veeam is unable to delete objects during cleanup.

We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:

Code: Select all

S3ConcurrentTaskLimit
S3MultiObjectDeleteLimit
CloudConcurrentMultiDeleteTaskLimit
S3RequestTimeoutSec
S3RequestRetryTotalTimeoutSec

We observed that the number of errors increases with higher limit settings. Performance during upload operations is perfectly fine; only delete operations are affected.

We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.

A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:13:23 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head2)
x-amz-id-2:<removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked

74
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>
0

But Wasabi doesn't send complete responses every time. The return code is still 200 (OK), but the body is missing. When this happens, the full body is lost. Not a single chunk is received. Veeam detects the response as invalid or unrecognized, and schedules a retry.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:12:33 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head3)
x-amz-id-2: <removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked

Veeam will make multiple retries until a valid response is received. This process can take a long time and may exceed the timeout value, causing the offload session to fail. The timeout can be increased using the registry values mentioned above.

I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.

Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.

Veeam support case ID: #07185215
put on hold, waiting for Wasabi response

Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks

Thank you,

Johan

[Mildur]

Hi Johan

I merged your topic with the existing one. Unfortunately we never got a case number from our two other customers.
Please give me some time to discuss this issue with our alliance team.

Best,
Fabian

[MikeAtWasabi]

Hi Johan,

Thank you for being a valued Wasabi customer, and we appreciate your patience while our engineering team investigates the issue on our end. We understand that this is an issue that needs to be resolved to speed up the cleanup process. Thanks for your assistance with the logs and Fiddler traces you provided. We will update the Wasabi ticket as soon as we have feedback from our engineering team.

Best Regards,
Mike

[pirx]

Since we moved our backup offloading from Amazon S3 to Wasabi, we have seen a significant number of the following errors in our Veeam logs during cleanup:

Code: Select all

[01.07.2024 04:56:26.063] < 16156> aws      | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [8932] seconds
[01.07.2024 04:56:26.063] < 16156> aws      | >>  |WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

[01.07.2024 04:56:26.063] < 16156> aws      | Retrying with AccessKey [<removed>]
[01.07.2024 04:56:27.391] <  4860> net      |   Sending request. URI: [/<removed>?delete]. Method: [POST]. Request: [DeleteObjects]
[01.07.2024 04:56:27.391] <  4860> net      |   Headers:
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Length: 336036
[01.07.2024 04:56:27.391] <  4860> net      |     Content-MD5: <removed>
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Type: text/plain; charset=utf-8
[01.07.2024 04:56:27.391] <  4860> net      |     Host: s3.eu-central-2.wasabisys.com
[01.07.2024 04:56:27.391] <  4860> net      |     User-Agent: APN/1.0 Veeam/1.0 Backup/12.0
[01.07.2024 04:56:27.391] <  4860> net      |   Body:

This error is logged when Veeam is unable to delete objects during cleanup.

We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:

Code: Select all

S3ConcurrentTaskLimit
S3MultiObjectDeleteLimit
CloudConcurrentMultiDeleteTaskLimit
S3RequestTimeoutSec
S3RequestRetryTotalTimeoutSec

We observed that the number of errors increases with higher limit settings. Performance during upload operations is perfectly fine; only delete operations are affected.

We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.

A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:13:23 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head2)
x-amz-id-2:<removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked

74
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>
0

But Wasabi doesn't send complete responses every time. The return code is still 200 (OK), but the body is missing. When this happens, the full body is lost. Not a single chunk is received. Veeam detects the response as invalid or unrecognized, and schedules a retry.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:12:33 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head3)
x-amz-id-2: <removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked

Veeam will make multiple retries until a valid response is received. This process can take a long time and may exceed the timeout value, causing the offload session to fail. The timeout can be increased using the registry values mentioned above.

I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.

Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.

Veeam support case ID: #07185215
put on hold, waiting for Wasabi response

Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks

Thank you,

Johan

Cleanup/Delete is also a big issue for us since migrating from AWS to Wasabi. I'd cases open on both ends but with no clear result. Same Veeam regkey for tuning, but still long or just failing cleanups. I can post case numbers next week.

[pirx]

Our case ID is 129372

[MrSpock]

Hi Johan

I merged your topic with the existing one. Unfortunately we never got a case number from our two other customers.
Please give me some time to discuss this issue with our alliance team.

Best,
Fabian

Hello Fabian,

I have asked my friendly support representative at Wasabi if they have received any communication from your alliance team, and this is his response:

As far as I can tell, we (Wasabi support) have NOT received communication from the alliance team. If they reached out to Wasabi via a different channel, I unfortunately will not be able to tell.

Is there any reason to believe they may have reached out so we can attempt to follow up on this?
..
If they've reached out to you regarding the issue at hand and shared any new information, please don't hesitate to let us know.

Could you please check with the alliance team to see how things are progressing?

Thank you,

Johan

[Mildur]

Hi Johan

Our alliance team reached out to Wasabi.
Unfortunately I'm not personally aware about the exact state of the issue.
I will check with our team if we have an update.

Best,
Fabian

[MrSpock]

Hello,

I just wanted to inform you that this issue has been resolved by Wasabi on their end. They implemented a fix in their production environment around August 7th. Since then, we have gradually increased the performance limit settings in Veeam Backup, and we have been operating on the default settings since August 14th without any issues during delete requests.

I am grateful for the assistance provided by both Veeam and Wasabi during these investigations.

Johan

[Mildur]

Hi Johan

I apologize for the late answer. I was on vacation.
Thank you for your feedback. I'm glad it could be solved.

Best,
Fabian