Discussions related to using object storage as a backup target.
Post Reply
scidoner
Enthusiast
Posts: 37
Liked: 8 times
Joined: Jul 07, 2013 4:43 am
Contact:

[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by scidoner »

Just came here to say I had the same problem with Wasabi S3, which is a supported object storage target.

No one (even Wasabi support) could resolve this issue. In the end, I put it down to some kind of problem with WinHTTP and switched to a Linux gateway to proxy the S3 repository data and have not had an issue since.
Mildur
Product Manager
Posts: 9525
Liked: 2524 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by Mildur »

Hi scidoner

Did you have opened a case with us? Which I can review?

Best,
Fabian
Product Management Analyst @ Veeam Software
scidoner
Enthusiast
Posts: 37
Liked: 8 times
Joined: Jul 07, 2013 4:43 am
Contact:

Re: HTTP exception: WinHttpReceiveResponse_callback_timeout: 12152: The server returned an invalid or unrecognizedrespon

Post by scidoner »

@Mildur - Nah, we weren't convinced it was a Veeam root cause at the time, more likely that Windows HTTP implementation is rubbish.
MrSpock
Service Provider
Posts: 49
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

[MERGED] [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by MrSpock »

Since we moved our backup offloading from Amazon S3 to Wasabi, we have seen a significant number of the following errors in our Veeam logs during cleanup:

Code: Select all

[01.07.2024 04:56:26.063] < 16156> aws      | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [8932] seconds
[01.07.2024 04:56:26.063] < 16156> aws      | >>  |WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

[01.07.2024 04:56:26.063] < 16156> aws      | Retrying with AccessKey [<removed>]
[01.07.2024 04:56:27.391] <  4860> net      |   Sending request. URI: [/<removed>?delete]. Method: [POST]. Request: [DeleteObjects]
[01.07.2024 04:56:27.391] <  4860> net      |   Headers:
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Length: 336036
[01.07.2024 04:56:27.391] <  4860> net      |     Content-MD5: <removed>
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Type: text/plain; charset=utf-8
[01.07.2024 04:56:27.391] <  4860> net      |     Host: s3.eu-central-2.wasabisys.com
[01.07.2024 04:56:27.391] <  4860> net      |     User-Agent: APN/1.0 Veeam/1.0 Backup/12.0
[01.07.2024 04:56:27.391] <  4860> net      |   Body:
This error is logged when Veeam is unable to delete objects during cleanup.

We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:

Code: Select all

S3ConcurrentTaskLimit
S3MultiObjectDeleteLimit
CloudConcurrentMultiDeleteTaskLimit
S3RequestTimeoutSec
S3RequestRetryTotalTimeoutSec
We observed that the number of errors increases with higher limit settings. Performance during upload operations is perfectly fine; only delete operations are affected.

We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.

A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:13:23 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head2)
x-amz-id-2:<removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked

74
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>
0
But Wasabi doesn't send complete responses every time. The return code is still 200 (OK), but the body is missing. When this happens, the full body is lost. Not a single chunk is received. Veeam detects the response as invalid or unrecognized, and schedules a retry.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:12:33 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head3)
x-amz-id-2: <removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked
Veeam will make multiple retries until a valid response is received. This process can take a long time and may exceed the timeout value, causing the offload session to fail. The timeout can be increased using the registry values mentioned above.

I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.

Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.

Veeam support case ID: #07185215
put on hold, waiting for Wasabi response

Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks

Thank you,

Johan
Mildur
Product Manager
Posts: 9525
Liked: 2524 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by Mildur »

Hi Johan

I merged your topic with the existing one. Unfortunately we never got a case number from our two other customers.
Please give me some time to discuss this issue with our alliance team.

Best,
Fabian
Product Management Analyst @ Veeam Software
MikeAtWasabi
Technology Partner
Posts: 1
Liked: never
Joined: Jul 05, 2024 5:20 pm
Full Name: Mike
Contact:

Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by MikeAtWasabi »

Hi Johan,

Thank you for being a valued Wasabi customer, and we appreciate your patience while our engineering team investigates the issue on our end. We understand that this is an issue that needs to be resolved to speed up the cleanup process. Thanks for your assistance with the logs and Fiddler traces you provided. We will update the Wasabi ticket as soon as we have feedback from our engineering team.

Best Regards,
Mike
pirx
Veteran
Posts: 598
Liked: 87 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: [MERGED] [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by pirx »

MrSpock wrote: Jul 03, 2024 7:27 am Since we moved our backup offloading from Amazon S3 to Wasabi, we have seen a significant number of the following errors in our Veeam logs during cleanup:

Code: Select all

[01.07.2024 04:56:26.063] < 16156> aws      | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [8932] seconds
[01.07.2024 04:56:26.063] < 16156> aws      | >>  |WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

[01.07.2024 04:56:26.063] < 16156> aws      | Retrying with AccessKey [<removed>]
[01.07.2024 04:56:27.391] <  4860> net      |   Sending request. URI: [/<removed>?delete]. Method: [POST]. Request: [DeleteObjects]
[01.07.2024 04:56:27.391] <  4860> net      |   Headers:
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Length: 336036
[01.07.2024 04:56:27.391] <  4860> net      |     Content-MD5: <removed>
[01.07.2024 04:56:27.391] <  4860> net      |     Content-Type: text/plain; charset=utf-8
[01.07.2024 04:56:27.391] <  4860> net      |     Host: s3.eu-central-2.wasabisys.com
[01.07.2024 04:56:27.391] <  4860> net      |     User-Agent: APN/1.0 Veeam/1.0 Backup/12.0
[01.07.2024 04:56:27.391] <  4860> net      |   Body:
This error is logged when Veeam is unable to delete objects during cleanup.

We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:

Code: Select all

S3ConcurrentTaskLimit
S3MultiObjectDeleteLimit
CloudConcurrentMultiDeleteTaskLimit
S3RequestTimeoutSec
S3RequestRetryTotalTimeoutSec
We observed that the number of errors increases with higher limit settings. Performance during upload operations is perfectly fine; only delete operations are affected.

We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.

A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:13:23 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head2)
x-amz-id-2:<removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked

74
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>
0
But Wasabi doesn't send complete responses every time. The return code is still 200 (OK), but the body is missing. When this happens, the full body is lost. Not a single chunk is received. Veeam detects the response as invalid or unrecognized, and schedules a retry.

Code: Select all

HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:12:33 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head3)
x-amz-id-2: <removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked
Veeam will make multiple retries until a valid response is received. This process can take a long time and may exceed the timeout value, causing the offload session to fail. The timeout can be increased using the registry values mentioned above.

I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.

Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.

Veeam support case ID: #07185215
put on hold, waiting for Wasabi response

Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks

Thank you,

Johan
Cleanup/Delete is also a big issue for us since migrating from AWS to Wasabi. I'd cases open on both ends but with no clear result. Same Veeam regkey for tuning, but still long or just failing cleanups. I can post case numbers next week.
pirx
Veteran
Posts: 598
Liked: 87 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by pirx »

Our case ID is 129372
MrSpock
Service Provider
Posts: 49
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by MrSpock »

Mildur wrote: Jul 03, 2024 7:58 am Hi Johan

I merged your topic with the existing one. Unfortunately we never got a case number from our two other customers.
Please give me some time to discuss this issue with our alliance team.

Best,
Fabian
Hello Fabian,

I have asked my friendly support representative at Wasabi if they have received any communication from your alliance team, and this is his response:
As far as I can tell, we (Wasabi support) have NOT received communication from the alliance team. If they reached out to Wasabi via a different channel, I unfortunately will not be able to tell.

Is there any reason to believe they may have reached out so we can attempt to follow up on this?
..
If they've reached out to you regarding the issue at hand and shared any new information, please don't hesitate to let us know.
Could you please check with the alliance team to see how things are progressing?

Thank you,

Johan
Mildur
Product Manager
Posts: 9525
Liked: 2524 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by Mildur »

Hi Johan

Our alliance team reached out to Wasabi.
Unfortunately I'm not personally aware about the exact state of the issue.
I will check with our team if we have an update.

Best,
Fabian
Product Management Analyst @ Veeam Software
MrSpock
Service Provider
Posts: 49
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by MrSpock »

Hello,

I just wanted to inform you that this issue has been resolved by Wasabi on their end. They implemented a fix in their production environment around August 7th. Since then, we have gradually increased the performance limit settings in Veeam Backup, and we have been operating on the default settings since August 14th without any issues during delete requests.

I am grateful for the assistance provided by both Veeam and Wasabi during these investigations.

Johan
Mildur
Product Manager
Posts: 9525
Liked: 2524 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response

Post by Mildur » 1 person likes this post

Hi Johan

I apologize for the late answer. I was on vacation.
Thank you for your feedback. I'm glad it could be solved.

Best,
Fabian
Product Management Analyst @ Veeam Software
Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests