Checkpoint removal process info

[david.domask]

Hi Benjaminlee,

Thank you for sharing the cases and the extensive details. If I get it right from the case notes, the operations you were attempting were being done on backups that were being processed by the retention/checkpoint deletion job. Normally, these processes should be pretty fast and you would avoid the lock conflicts, but seems like due to the difficulties with the Checkpoint Removal/Retention, the lock was held on the backups.

I would advise continue working with Support; likely the changes here are more related to an existing issue that was "uncovered" by the introduction of the background checkpoint removal in 12.2, not directly related to the change itself, and lowering the concurrent tasks from unlimited to a limited amount makes sense as the solution here, but please continue working with Support.

[vashthestampede]

Here we are, same issue of the other with a s3-compatible capacity tier (BackBlaze).
Case #07475556 Checkpoint Removal job Failes

In my case if i look at the job logs i see the error but a the same time the succesfull row:

Code: Select all

22/10/2024 03:00:07          Job started at 22/10/2024 03:00:07
22/10/2024 03:00:23 Error    Failed to remove checkpoint e3290e58-e4ab-45e1-990e-dc2b25c9ba55 from capacity extent.
22/10/2024 23:42:49 Error    Failed to remove checkpoint e3290e58-e4ab-45e1-990e-dc2b25c9ba55 from capacity extent.
23/10/2024 01:37:00          Checkpoints have been successfully removed from the backup repository B2 Repository.
23/10/2024 03:00:57          Job finished at 23/10/2024 03:00:57

Then, the Job seem to fail two times, but only one mail received as notify.

At the moment one "Chekpoint Removal" job running with the following details

Code: Select all

23/10/2024 03:00:57          Job started at 23/10/2024 03:00:57
23/10/2024 04:51:25          Checkpoints have been successfully removed from the backup repository B2 Repository.

[daviek]

I've just started seeing errors in Veeam B&R in the below format. From what I've read, the Retention/checkpoint piece in v12.2 has changed quite a bit. This is a little different than what you're seeing, but I wanted to put it out there anyway since I'm new to the R&D Forums. Hope this helps!
 
Failed to remove checkpoints from the backup repository .............
Failed to remove checkpoints from the backup repository ............
Failed to remove checkpoints for the backup .......to ......... from the repository ..........
Failed to remove checkpoints for the backup .......to ......... from the repository ..........
Job has failed unexpectedly

[Gostev]

The retention/checkpoint logic did not change at all in v12.2, however we added reporting for when this process fails (whereas previously it would just fail quietly). When it fails, it's always due to an issue connecting to object storage or due to object storage itself experiencing issues (e.g. overload).

[veremin]

As Anton mentioned, most issues are not directly related to the background checkpoint removal process itself but rather to misconfigured infrastructure, often including the network.

I have merged your responses into the general thread where this problem is being discussed.

In any case, we recommend reaching out to the technical support department to determine the actual issue, as the mention of it may be hidden deep within the debug logs.

Thanks!

[talltim]

Is there a way to exclude the Checkpoint removal job from email notifications? I have a call logged with support but I have a feeling they are going to say it is network issues. Every one that has failed so far seems to be removed on the next run of the job.

[hayliz]

Hello everyone, we have several large Veeam environments with scaleout repository. We also get constant chepoint removal messages every day. We have been working with the Veeam support colleagues for a long time.

Our setup:
VMware Environment: 7.0.3, 23794027
Veeam: v12.2.
Repository: Scaleout Repository with capacity tier (copy)
Veeam case: 07407985

We were informed that it could possibly be due to the performance tier. But unfortunately, we cannot be given a precise statement as to the exact cause or it is not clear.

[NeilColyn]

For us, the issue was our Veeam object storage repository (Azure Storage Account) had Versioning enabled which is not supported.

https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Data in an object storage bucket or container must be managed solely by Veeam Backup & Replication, including retention (in case you enable Object Lock and Versioning features on an S3 bucket or version-level WORM on an Azure container) and data management. Enabling lifecycle rules is not supported, and may result in backup and restore failures.

[m.novelli]

Wow, this is a good catch

I should control my old Azure Storage Accounts created before v12.2, I'm sure I have some with versioning and Veeam didn't check that before 12.2 or 12.1 I'm pretty sure

Marco

[m.novelli]

Nope, I got confused with "soft delete" option on Azure Storage Account, checked right now

Marco

[veremin]

Is there a way to exclude the Checkpoint removal job from email notifications? I have a call logged with support but I have a feeling they are going to say it is network issues. Every one that has failed so far seems to be removed on the next run of the job.

This will be possible in the next version of our product, Veeam Backup & Replication 12.3. There we've improved the background checkpoint removal notifications by including the underlying error and downgraded the severity of issues from "failure" to "warning." Additionally, we have introduced a couple of registry keys that allow you to further reduce the severity to an informational message.

Thanks!

[sfirmes]

Veeam Backup & Replication 12.3 is now available for download and What's New in v12.3 document has details on all the enhancements around background checkpoint removal.

[veremin]

Yeah, 12.3 has just been released.

With it you can control the severity of background checkpoint removal event by creating ObjectStorageCheckpointRemovalSeverity
(DWORD, 0 — info, 1 — warning, 2 — error) registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server.

Thanks!

[Wallachuck]

Seeing this same error here on around 10 or so sites. Searched the term "Failed to remove a checkpoint during a background job" and this thread popped up.


All using Wasabi and Veeam 12.2.0.334
Previously running 12.1.2.172 without the error

Going to try upgrade to 12.3.0.310 to see if it fixes it and will post back if it does to help out the next person searching the net.

[veremin]

The majority of issues stem not from the background checkpoint removal process itself, but from improperly configured infrastructure, which often involves the network.

Upgrading to 12.3 will not resolve these issues; rather, it will reduce their severity level and provide a low-level error that will give a clearer understanding of what is wrong with the infrastructure or network.

Thanks!

[m.novelli]

Yesterday I've updated a dozen of customers to v12.3, one Server send me this warning every 30 minutes - 1 hour:

Job has been stopped with failures. Name: [Checkpoint Removal] Details: Job aborted due to server termination

Veeam and the Server are up and running, in the "Background Retention" I can see "Checkpoint removal: working" , no sign of errors

If it's not a known issue I'm going to open a support ticket just to get rid of the email every 30 minutes

Marco

[m.novelli]

Case #07539611 

Marco

[veremin]

I suspect the root error might be linked to server termination, which can occur during upgrades, reboots, or similar actions.

The email warning spam could be due to SMTP issues, such as an exhausted SMTP server cache causing repeated send attempts.

However, the situation seems unusual since we've specifically implemented an exception to prevent failure reports from being sent with every VBR server reboot.

So, kindly, continue your investigation with our support team.

Thanks!