-
- Enthusiast
- Posts: 37
- Liked: 8 times
- Joined: Jul 07, 2013 4:43 am
- Contact:
[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Just came here to say I had the same problem with Wasabi S3, which is a supported object storage target.
No one (even Wasabi support) could resolve this issue. In the end, I put it down to some kind of problem with WinHTTP and switched to a Linux gateway to proxy the S3 repository data and have not had an issue since.
No one (even Wasabi support) could resolve this issue. In the end, I put it down to some kind of problem with WinHTTP and switched to a Linux gateway to proxy the S3 repository data and have not had an issue since.
-
- Product Manager
- Posts: 9557
- Liked: 2532 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hi scidoner
Did you have opened a case with us? Which I can review?
Best,
Fabian
Did you have opened a case with us? Which I can review?
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Enthusiast
- Posts: 37
- Liked: 8 times
- Joined: Jul 07, 2013 4:43 am
- Contact:
Re: HTTP exception: WinHttpReceiveResponse_callback_timeout: 12152: The server returned an invalid or unrecognizedrespon
@Mildur - Nah, we weren't convinced it was a Veeam root cause at the time, more likely that Windows HTTP implementation is rubbish.
-
- Service Provider
- Posts: 49
- Liked: 3 times
- Joined: Apr 24, 2009 10:16 pm
- Contact:
[MERGED] [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Since we moved our backup offloading from Amazon S3 to Wasabi, we have seen a significant number of the following errors in our Veeam logs during cleanup:
This error is logged when Veeam is unable to delete objects during cleanup.
We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:
We observed that the number of errors increases with higher limit settings. Performance during upload operations is perfectly fine; only delete operations are affected.
We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.
A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.
But Wasabi doesn't send complete responses every time. The return code is still 200 (OK), but the body is missing. When this happens, the full body is lost. Not a single chunk is received. Veeam detects the response as invalid or unrecognized, and schedules a retry.
Veeam will make multiple retries until a valid response is received. This process can take a long time and may exceed the timeout value, causing the offload session to fail. The timeout can be increased using the registry values mentioned above.
I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.
Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.
Veeam support case ID: #07185215
put on hold, waiting for Wasabi response
Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks
Thank you,
Johan
Code: Select all
[01.07.2024 04:56:26.063] < 16156> aws | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [8932] seconds
[01.07.2024 04:56:26.063] < 16156> aws | >> |WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
[01.07.2024 04:56:26.063] < 16156> aws | Retrying with AccessKey [<removed>]
[01.07.2024 04:56:27.391] < 4860> net | Sending request. URI: [/<removed>?delete]. Method: [POST]. Request: [DeleteObjects]
[01.07.2024 04:56:27.391] < 4860> net | Headers:
[01.07.2024 04:56:27.391] < 4860> net | Content-Length: 336036
[01.07.2024 04:56:27.391] < 4860> net | Content-MD5: <removed>
[01.07.2024 04:56:27.391] < 4860> net | Content-Type: text/plain; charset=utf-8
[01.07.2024 04:56:27.391] < 4860> net | Host: s3.eu-central-2.wasabisys.com
[01.07.2024 04:56:27.391] < 4860> net | User-Agent: APN/1.0 Veeam/1.0 Backup/12.0
[01.07.2024 04:56:27.391] < 4860> net | Body:
We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:
Code: Select all
S3ConcurrentTaskLimit
S3MultiObjectDeleteLimit
CloudConcurrentMultiDeleteTaskLimit
S3RequestTimeoutSec
S3RequestRetryTotalTimeoutSec
We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.
A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.
Code: Select all
HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:13:23 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head2)
x-amz-id-2:<removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked
74
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>
0
Code: Select all
HTTP/1.1 200 OK
Content-Type: application/xml
Date: Wed, 22 May 2024 07:12:33 GMT
Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head3)
x-amz-id-2: <removed>
x-amz-request-id: <removed>
x-wasabi-cm-reference-id: <removed>
Transfer-Encoding: chunked
I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.
Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.
Veeam support case ID: #07185215
put on hold, waiting for Wasabi response
Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks
Thank you,
Johan
-
- Product Manager
- Posts: 9557
- Liked: 2532 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
[Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hi Johan
I merged your topic with the existing one. Unfortunately we never got a case number from our two other customers.
Please give me some time to discuss this issue with our alliance team.
Best,
Fabian
I merged your topic with the existing one. Unfortunately we never got a case number from our two other customers.
Please give me some time to discuss this issue with our alliance team.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Technology Partner
- Posts: 1
- Liked: never
- Joined: Jul 05, 2024 5:20 pm
- Full Name: Mike
- Contact:
Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hi Johan,
Thank you for being a valued Wasabi customer, and we appreciate your patience while our engineering team investigates the issue on our end. We understand that this is an issue that needs to be resolved to speed up the cleanup process. Thanks for your assistance with the logs and Fiddler traces you provided. We will update the Wasabi ticket as soon as we have feedback from our engineering team.
Best Regards,
Mike
Thank you for being a valued Wasabi customer, and we appreciate your patience while our engineering team investigates the issue on our end. We understand that this is an issue that needs to be resolved to speed up the cleanup process. Thanks for your assistance with the logs and Fiddler traces you provided. We will update the Wasabi ticket as soon as we have feedback from our engineering team.
Best Regards,
Mike
-
- Veteran
- Posts: 598
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: [MERGED] [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Cleanup/Delete is also a big issue for us since migrating from AWS to Wasabi. I'd cases open on both ends but with no clear result. Same Veeam regkey for tuning, but still long or just failing cleanups. I can post case numbers next week.MrSpock wrote: ↑Jul 03, 2024 7:27 am Since we moved our backup offloading from Amazon S3 to Wasabi, we have seen a significant number of the following errors in our Veeam logs during cleanup:
This error is logged when Veeam is unable to delete objects during cleanup.Code: Select all
[01.07.2024 04:56:26.063] < 16156> aws | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [8932] seconds [01.07.2024 04:56:26.063] < 16156> aws | >> |WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response [01.07.2024 04:56:26.063] < 16156> aws | Retrying with AccessKey [<removed>] [01.07.2024 04:56:27.391] < 4860> net | Sending request. URI: [/<removed>?delete]. Method: [POST]. Request: [DeleteObjects] [01.07.2024 04:56:27.391] < 4860> net | Headers: [01.07.2024 04:56:27.391] < 4860> net | Content-Length: 336036 [01.07.2024 04:56:27.391] < 4860> net | Content-MD5: <removed> [01.07.2024 04:56:27.391] < 4860> net | Content-Type: text/plain; charset=utf-8 [01.07.2024 04:56:27.391] < 4860> net | Host: s3.eu-central-2.wasabisys.com [01.07.2024 04:56:27.391] < 4860> net | User-Agent: APN/1.0 Veeam/1.0 Backup/12.0 [01.07.2024 04:56:27.391] < 4860> net | Body:
We have been working on this issue with Veeam and Wasabi for more than three months, but it remains unresolved. Several theories have been proposed during this period, mostly related to network issues on the client side and performance bottlenecks in the Wasabi S3 API. We have ruled out all possible network-related issues and have also tried various combinations of the following registry values to find a sweet spot that the API can handle:
We observed that the number of errors increases with higher limit settings. Performance during upload operations is perfectly fine; only delete operations are affected.Code: Select all
S3ConcurrentTaskLimit S3MultiObjectDeleteLimit CloudConcurrentMultiDeleteTaskLimit S3RequestTimeoutSec S3RequestRetryTotalTimeoutSec
We have finally concluded that Wasabi indeed returns invalid responses to delete requests. We confirmed this by intercepting the traffic with Fiddler Classic.
A valid response to the delete request looks like the one below (API reference). The return code is 200 (OK), and the body contains the delete result in valid XML. Note that Wasabi transfers the body in chunks.
But Wasabi doesn't send complete responses every time. The return code is still 200 (OK), but the body is missing. When this happens, the full body is lost. Not a single chunk is received. Veeam detects the response as invalid or unrecognized, and schedules a retry.Code: Select all
HTTP/1.1 200 OK Content-Type: application/xml Date: Wed, 22 May 2024 07:13:23 GMT Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head2) x-amz-id-2:<removed> x-amz-request-id: <removed> x-wasabi-cm-reference-id: <removed> Transfer-Encoding: chunked 74 <?xml version="1.0" encoding="UTF-8"?> <DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult> 0
Veeam will make multiple retries until a valid response is received. This process can take a long time and may exceed the timeout value, causing the offload session to fail. The timeout can be increased using the registry values mentioned above.Code: Select all
HTTP/1.1 200 OK Content-Type: application/xml Date: Wed, 22 May 2024 07:12:33 GMT Server: WasabiS3/7.20.2895-2024-04-24-09eaa42f6e (head3) x-amz-id-2: <removed> x-amz-request-id: <removed> x-wasabi-cm-reference-id: <removed> Transfer-Encoding: chunked
I am quite sure that I am not the only one experiencing these problems. I have read posts about similar issues related to slow and time-consuming cleanups. I can only imagine how much unnecessary load these retries put on the Wasabi infrastructure, probably making the problem even worse.
Neither Veeam nor Wasabi have been able to tell if these problems (error 12152) are common among Veeam customers, so I would like to ask all other Veeam users offloading to Wasabi to search your SOBR logs (*.log + *.log.gz) for "WinHttpQueryDataAvaliable: 12152" after cleanups have been made. Please let me know your findings.
Veeam support case ID: #07185215
put on hold, waiting for Wasabi response
Wasabi support case ID: #127596 and #124314
no valuable updates regarding the case since three weeks
Thank you,
Johan
-
- Veteran
- Posts: 598
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
-
- Service Provider
- Posts: 49
- Liked: 3 times
- Joined: Apr 24, 2009 10:16 pm
- Contact:
Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hello Fabian,
I have asked my friendly support representative at Wasabi if they have received any communication from your alliance team, and this is his response:
Could you please check with the alliance team to see how things are progressing?As far as I can tell, we (Wasabi support) have NOT received communication from the alliance team. If they reached out to Wasabi via a different channel, I unfortunately will not be able to tell.
Is there any reason to believe they may have reached out so we can attempt to follow up on this?
..
If they've reached out to you regarding the issue at hand and shared any new information, please don't hesitate to let us know.
Thank you,
Johan
-
- Product Manager
- Posts: 9557
- Liked: 2532 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hi Johan
Our alliance team reached out to Wasabi.
Unfortunately I'm not personally aware about the exact state of the issue.
I will check with our team if we have an update.
Best,
Fabian
Our alliance team reached out to Wasabi.
Unfortunately I'm not personally aware about the exact state of the issue.
I will check with our team if we have an update.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Service Provider
- Posts: 49
- Liked: 3 times
- Joined: Apr 24, 2009 10:16 pm
- Contact:
Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hello,
I just wanted to inform you that this issue has been resolved by Wasabi on their end. They implemented a fix in their production environment around August 7th. Since then, we have gradually increased the performance limit settings in Veeam Backup, and we have been operating on the default settings since August 14th without any issues during delete requests.
I am grateful for the assistance provided by both Veeam and Wasabi during these investigations.
Johan
I just wanted to inform you that this issue has been resolved by Wasabi on their end. They implemented a fix in their production environment around August 7th. Since then, we have gradually increased the performance limit settings in Veeam Backup, and we have been operating on the default settings since August 14th without any issues during delete requests.
I am grateful for the assistance provided by both Veeam and Wasabi during these investigations.
Johan
-
- Product Manager
- Posts: 9557
- Liked: 2532 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: [Wasabi] WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response
Hi Johan
I apologize for the late answer. I was on vacation.
Thank you for your feedback. I'm glad it could be solved.
Best,
Fabian
I apologize for the late answer. I was on vacation.
Thank you for your feedback. I'm glad it could be solved.
Best,
Fabian
Product Management Analyst @ Veeam Software
Who is online
Users browsing this forum: No registered users and 14 guests