Checkpoint deletion error on Wasabi capacity tier

[StephanF]

Hi,

Support case #07249962

For some time now we are getting repeatedly error during the checkpoint removal process from our Wasabi capacity tier. The main error messages are (either one or the other):
Failed to delete checkpoint. Details: HTTP exception: Retrieving message chunk header, error code: 110
Failed to delete checkpoint. Details: Several pending checkpoints found 97,109,

According to the support engineer this should be related to rate limiting of Delete operations on Wasabi's end and he was referring to this URL: https://docs.wasabi.com/docs/setup-limi ... rity%20and
Now I found this post stating their is no such throttling from Wasabi. I guess the support engineer was looking at the wrong API in Wasabi's docs.

Nevertheless we implemented registry keys to reduce S3ConcurrentTaskLimit and limited parallel jobs on the cloud repository to 4. Now the offload performance got bad and I am getting also failed jobs due to resource unavailable timeouts.

I keep working with support on this but nevertheless want to ask, if anybody has seen such error messages before?

Best regards,
Stephan

[david.domask]

Hi Stephan,

Thank you for the detailed description and for sharing the case number.

I checked the case briefly and the registry values and adjustments Veeam Support advised are fairly common for such situations, as such errors may occur for reasons such as network interruption, load on S3 target, etc; Veeam Backup and Replication has quite a few "levers" to accommodate for this which the Engineer recommended. Please continue with the Support Engineer as based on the notes I saw, it looks like they are narrowing down on the issue, so let's see the results of their update after your most recent information. The resource unavailable timeouts should be checked, as likely there are more adjustments that can be set.

If you have further concerns on the handling of any case, reach out to Support Management and explain the situation, and they will review the case.

> According to the support engineer this should be related to rate limiting of Delete operations on Wasabi's end and he was referring to this URL: https://docs.wasabi.com/docs/setup-limi ... rity%20and
Now I found this post stating their is no such throttling from Wasabi. I guess the support engineer was looking at the wrong API in Wasabi's docs.

Yes, regrettably it seems to be the case. I will address this with the Support Team as the link is clearly about Wasabi's account control API, not the S3 one, so likely just a misunderstanding.

[StephanF]

Hi David,

thanks for your answer.
Currently I have no concerns in the handling of our support case. Indeed your engineer is doing good work and I have no reason to complain.

What made me concern a little bit is the fact, that we see such issues with Wasabi storage and that these "non-standard" settings are needed to make it work flawlessly in our case. As Wasabi is recommended by Veeam, our environment is rather small and our Internet connection is quite good, maybe I expected that it "just works" out of the box.
That's mainly the reason I asked here if other users are seeing the same issues.

BR
Stephan

[david.domask]

Hi Stephan,

Glad to hear that the Engineer is meeting your expectations; sorry if I seemed too pre-emptive with my statement, more just wanted to ensure you had the options available if there were concerns.

As for the adjustments, completely understood; while very often it is the out of box experience, depending on the workload being process, performance in a specific s3 region, etc, adjustments may need to be made; this is one reason we do want to see cases where it's not working out of box as it does help us to better understand what adjustments (if any) can be made if it's not a temporary issue; so I do appreciate you sharing the case and what you've been experiencing during these issues.

Based on the most recent update from Support, it does look like some progress was made as it looks like previously there were unrecoverable errors being returned from the S3 endpoint which now look to be mitigated by the changes, and the newest errors appear to be related elsewhere which the Engineer is attempting to address. Let's see the results of the most recent steps and hopefully all green offloads soon.

[admd]

Hey, I got this error since the update to Veeam 12.2.0.334 on a client of mine.
The new checkpoint removal process is always failed.

"Several pending checkpoints found 324,326,
Failed to cleanup V2 storages
Failed to cleanup archive"

Would you share the solution you found ?

Regards,

[david.domask]

Hi admd,

In this case, the issue will need to be checked by Support, so open a Support Case and allow Veeam Support a chance to review the behavior. Be sure to include logs for Support to review. (Use the 3rd radio button to select a host to export logs from, and select the Veeam server itself).

Thanks!

[ratkinsonuk]

I'm also seeing these errors since upgrading to v12.2. We use AWS S3 as our capacity tier.

The errors aren't frequent enough to become annoying, but I would like to know if the issue is self-correcting. Will Veeam remove the checkpoints on it's next offload pass/backup or do we need to manually intervene for each occurrence?

Thanks, Rob.

[david.domask]

Hi Rob,

Unfortunately will be difficult to tell without a review of the logs as to why the error is happening. In the topic creator's (tc) case, Veeam Support intervention was required, however, the same messages could appear due to load on the S3 provider or for other reasons, so a Support Case is the way to go here.

Please share your case number once created, thanks!