Checkpoint removal process info

[ifi-deda]

Since I upgraded to Veeam B&R 12.2.0.334 I noticed that every night an additional background retention process is started. It is called "Checkpoint Removal". It starts at 3:00am and run continuously for 24 hours.
It seems related to object storage repositories but I could not find info about it in the official documentation.
Can someone explain how it works, what its purpose is and point me to the documentation?

Thanks

[david.domask]

Hi ifi-deda,

Sure, please check our "What's New" document here for 12.2: https://www.veeam.com/kb4600

Background checkpoint removal — checkpoint removal operation has been decoupled from the backup
job and will now run as a system session after backup jobs and offload processes finish and there are
no conflicting tasks (checkpoint removal has the lowest priority for the task scheduler). In addition, this
process is triggered daily at 3 a.m. to perform the same activity. This change will remove additional load
from object storage during the time when it is already extremely busy accepting incoming backups, and
also prevent backup jobs from appearing to “hang” at the end while processing checkpoints removal.

Basically, as it says, the checkpoint removal for the checkpoints on Object Storage are now detached from the jobs themselves as this removal can be fairly busy work for the S3 storage. The checkpoints are an internal mechanism for Object Storage Repositories to help manage the data blocks -- checkpoint removal removes the no longer necessary data as per retention. Previously the removal might occur while other backup operations are utilizing the Object Storage repository and it could cause slowness for all operations, often resulting in the jobs "hanging" at 99% when in reality the jobs were busy with the checkpoint removal operations, and now the jobs will run without having to compete for resources with the checkpoint removal.

[ReKe]

Any Reg-Key to change the starttime? Most of the Copy to Objectstorage jobs run at this time for us?

[david.domask]

Hi ReKe,

The registry values are:

CheckpointRemovalJobStartTimeHours
CheckpointRemovalJobStartTimeMins

Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Type: DWORD
Values: Integer (default 3 and 0 respectively for Hours/Mins)

[ReKe]

Hi David,

thanks for the reply. Will try it.

Gretings Rene

[mpm@aramark]

We upgraded to v12.2 over the weekend and I just received an email. Subject: Failed to remove a checkpoint during a background job on <vbr server name>.
What actions need to be taken?

[sfirmes]

mpm@aramark, you should contact Veeam Support and open a support case so that they can help you troubleshoot the issue.

[DShoemaker]

We are experiencing a similar trouble. I have opened a support case with Veeam.

[Gostev]

In addition, this process is triggered daily at 3 a.m. to perform the same activity.

BTW some text got lost here during PDF conversion, this was meant to say "to perform the same activity for backups which are no longer associated with any jobs, as well as VeeamZIP and exported backups".

[sfirmes]

We are experiencing a similar trouble. I have opened a support case with Veeam.

Please post or PM me your support case and I will look into it.

Thanks

[efd121]

Is there a reg-key to disable the check point removal from starting at all or limiting how many hours it runs?

I have a need to temporarily stop it and/or limit the hours it runs. When offloads and the checkpoint removal are running at the same time both processes cause severe slowness. If they are allowed to run by themselves they process a lot faster since the S3 appliance is not trying to manage both random read and random writes at the same time
Dave

[Gostev]

If you were to limit the run time or disable this completely, your storage appliance would eventually overfill. So instead it's better to change the start time for after the offload ends.

[admd]

Thanks for the registry key !

We have two clients which have the same errors. One is everyday, the other is more random.
Case #02833866 ongoing.
Support asked me to gather logs from Object Storage side (Wasabi)
I am waiting for the error to appear again.

[efd121]

I agree that disabling or limiting the hours is not a good way to manage the backups. I would not intend to leave the system in this configuration. In my current situation I've been told when both offloads and deletes occur at the same time both suffer severe performance issues and if I could let them run on their own both would run significantly faster. I'm not convinced that would be the case but I'm seeing that my offloads are falling further and further behind each day. In some severe cases data is expiring faster than I can get the next offload completed for large VMs.
My short-term thought process would be to allow offloads to run while disabling deletions and monitor space on the appliance. When the space gets to a point where I have to run deletes, I would temporarily stop offloads to let them process. Its not a solution I would accept long term, I'm trying to find a way, any way to get the system caught up, I'm the equivalent of 6+ days of backups behind, some systems fully caught up and others without a single offload on the capacity tier.
Dave

[tomers@y-tech.net]

Hi,
I am in the same position. After upgrading to the latest hotfix, the checkpoint removal jobs keeps running 24/7. The load on the object storage is huge, it reaches 2000 and 3000 request rate only for the objectPutRetention requests, which are the checkpoint removal as much as I understand. This keeps the backups from running, everything is hang for a week now.
I have an opened case number 07430504 but I don't think the support guys understand the urgency of that issue, calling it "performance degradation" instead of full service outage.

Nothing works. We don't have backups. We have something like 500 servers waiting for a week now. Nothing to do from our side.
We are checking other vendors at the moment as a backup, maybe we will just switch everything if we will not have answers in the next hours.
we are Veeam Cloud Connect providers and our service is degraded because of that. It is sad that we can't have the service we need from your support in this bad situation.

[david.domask]

Hi Tomer,

Thank you for your post and for sharing the case number, and sorry to hear about the challenges.

Checking the case and with Support Management, the case has been prioritized and the situation is being prioritized given the impact of the issue and should be reaching out soon.

[Gostev]

In my current situation I've been told when both offloads and deletes occur at the same time both suffer severe performance issues and if I could let them run on their own both would run significantly faster.

Please note that vast majority of deletes should still happen around same time as before, right after the backup jobs finish. As per my earlier comment above, the checkpoint removal process that starts at night (3am by default) is only there to take care of a few "special" backup types. So unless you're heavy user of VeeamZIP or Export functionality, or have tons of abandoned backup (detached from their original jobs and just left to slowly expire) then I would not expect much activity happening after 3am in the first place.

[Gostev]

The load on the object storage is huge, it reaches 2000 and 3000 request rate only for the objectPutRetention requests

Then this is not caused by the Checkpoint removal process discussed in this topic... putObjectRetention is from extending immutability on existing objects.

[tyler.jurgens]

Is PutObjectRetention also used to set the initial retention policy for immutable backups as well, not just extending immutability on existing objects? I've seen this heavily used for first time backups as well.

[Gostev]

Yes, of course. Immutability is set when a new unique object first appears, and it is extended when existing objects are being reused in a newly created restore point.

While checkpoint removal operation discussed here works with metadata that represents a restore point.

[tyler.jurgens]

@tomer - This may not help for the situation you are in now, but we've avoided most of those problems by increasing the block size on jobs targeting our S3. Larger blocks means fewer objects to set the retention policy on when it lands on our Object Storage arrays. This will have the net benefit of shorter backup times, shorter time frame for all the PutObjectRetention processing and shorter time frame for removing older restore points. However, it does come at a cost of larger incremental backups.

Checkpoint removal is a thing for us as well. One specific customer removed the backup copy jobs and replaced them with backup jobs, since he could not alter block size on backup copy jobs directly. This resulted in backups that are left behind to be cleaned up with the checkpoint removal process. The concern was not around backup size for him, but rather backups completing within the allotted time - larger objects helped immensely. Once those older backups finally age out we should be in a great spot moving forward.

[veremin]

Is PutObjectRetention also used to set the initial retention policy for immutable backups as well, not just extending immutability on existing objects? I've seen this heavily used for first time backups as well.

Yes, of course. Immutability is set when a new unique object first appears, and it is extended when existing objects are being reused in a newly created restore point.

That is not entirely correct. When creating an object, we do not use the "putobjectretention" call. It is only used for extending immutability. In the case of object creation, we utilize the "putobject" request, where we can immediately set the desired lock period.

So, the Tyler's case looks a bit strange, and I believe our support team is already investigating it.

Thanks!

[m.novelli]

Also I’ve started to see many random “Failed to remove a checkpoint” from different customers after 12.2 upgrade.

We use Azure as Object Repository

Marco

[veremin]

Hi, Marco,

Do you happen to have a support ticket created with the debug logs attached, so we can ask the R&D team to take a look at it?

Thanks!

[m.novelli]

Sure, I need a couple of days, now I'm out of office

Marco

[m.novelli]

Case #07445010 

I'm uploading logs from different customers

Marco

[veremin]

Thank you, Mario, for opening the case with us and providing the logs. I have passed the ticket on to the R&D team. If necessary, they will join the investigation process.

[m.novelli]

Hi guys, the support engineer suggested me to choose a different time for background checkpoint removal , to avoid conflict with the backup copy to Object Storage

My 2 cents

1) time schedule is only by reg key
2) I'm expecting from Veeam "It just works"

I'm raising a feature request to make "background checkpoint removal" more intelligent, to detect when the copy job run and schedule itself according, or go back to previous implementation in v12.1 , or some more magic

Thanks,

Marco

[Gostev]

At the time of reg key only minimal activity should happen in any case, as only a handful of backups is processed then at best. The bulk of work still happens after the backup jobs, so not much different from "previous implementation in 12.1" with the exception of waiting for all backup jobs to finish running.

So I would double check if you have many backups of types I listed in my post in the first place (which is more likely since you're primarily dealing with SMB customer), and if not then question support engineer's suggestion as there's nothing for Veeam to do at the time of the reg key in principle.

@veremin this does bring the question when did start processing the retention of VeeamZIP/Exported/Abandoned backups in 12.1? Because I assume it had to be started at some specific time as well even in 12.1?

[veremin]

Hi guys, the support engineer suggested me to choose a different time for background checkpoint removal , to avoid conflict with the backup copy to Object Storage

Upon reviewing the provided debug logs, the R&D team identified errors related to the SSL certificate retrieval failure on the VeeamAgent side. There were no other errors or warnings found in the log bundle. The issue does not appear to be linked to the background checkpoint removal job, but rather to potential network connectivity or infrastructure problems. Additionally, there were multiple http retries observed in the VeeamAgent communication with Azure, supporting this theory.

Therefore, it seems that the support engineer's suggestions regarding the root cause of the issue and potential remedial actions are incorrect.

We recommend sharing this information with the engineer and considering escalating the ticket if necessary.

@veremin this does bring the question when did start processing the retention of VeeamZIP/Exported/Abandoned backups in 12.1? Because I assume it had to be started at some specific time as well even in 12.1?

I will double-check the information with the QA team and will get back later once I have additional details. Thanks!

Thanks!