Second opinion on C drive filling up over time

[jcofin13]

Case #07844016
Veeam 12.3.2.3617

Im getting some feedback on the case but no clear answers to my questions below. Just looking for a second opinion so i dont make a mess of the server by purging or modifying something i shouldnt.

Anyway.....
I have a veeam server where the C drive continues to fill up. As best i can tell its related to the C:\ProgramData\Veeam\Backup\System\CheckpointRemoval folder. This was troubleshot with Veeam support in April 2025. Case #07599135.

This folder contains dated folders that go back a couple of years.

In each, they can range from very small to a few hundred mb per day. Over the course of a month they are 10 - 12gb currently. WE had this issue once prior and it was causing the folders to grow to many gb per nigh in some instances.

Anyway.....if i go into a dated folder and review a log file (Agent.Cleanup.Blob.REpo.log.Index.log) i see the following (C:\ProgramData\Veeam\Backup\System\CheckpointRemoval\2025-10-06\<AWS-S3-REPO-Name>):

[05.10.2025 19:28:25.034] < 33064> aws | WARN|HTTP request failed, retry in [4] seconds, attempt number [4], total retry timeout left: [1784] seconds
[05.10.2025 19:28:25.034] < 33064> aws | >> |S3 error: Please reduce your request rate.
[05.10.2025 19:28:25.034] < 33064> aws | >> |Code: SlowDown

When we had this issue in the past (april 2025), the ticket i had stated we reduced the concurrent operations to S3 by modifying/adding a regkey entry:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

S3ConcurrentTaskLimit
dword: 15

As i mentioned, this seemed to help but its doing it again. It doesn't seem to be as drastic or creating as many logs and using space as it was prior (april 2025).

My questions are:

1. How do i clean this up so i can get space back especially from the older folders? Can anyone confirm the dated folders in the checkpointremoval folder are ok to delete?
2. Is the error im seeing in these agent clean up logs shown above the correct place to see whats going on?
3. Is the correct action to reduce the regkey from 15 to 10 or a lesser number to reduce the aws s3 request denials and just continually monitor it over time?
4. Is it normal for the dated folders in the checkpointremoval folder to be 10-12gb per month?
5. Is it possible to monitor for this behavior inside of veeam rather than looking the the txt logs on the server and spot checking it over time? Can VeeamONE monitor for this type of thing?

My guess is the AWS message is driving the log growth and we need to throttle down the request rate as suggested but would like to know if im on the correct path in my troubleshooting.


All files dated prior to 4/30/2025 show a size of ~68gb. (per the previous ticket)
All files dated 5/1/2025 - current show a size of ~58tb

The folder in total ~126gb which seems excessive.....and appears to be growing 10-12gb per month due to the logs in this folder.

[david.domask]

Hi jcofin13,

Thank you for sharing the case numbers and for the detailed write up.

I'll go through your questions one by one, then a bit more commentary:

1. Debug logs are always safe to delete, just means it might be difficult to troubleshoot some issues if the corresponding logs have been cleared. In this case, if it's really the S3 "Slow down" response filling the logs, it's fine to delete the older ones
2. Likely yes, but without a review of the debug logs and the environment will be difficult to give a more assertive answer (see below commentary)
3. Potentially can help, yes. This config parameter changes the number of concurrent HTTP requests made during S3 operations; however, it works in conjunction with the total concurrent tasks allowed for the repository (a concurrent task can have up to (S3ConcurrentTaskLimit) HTTP connections at once, so if you have 10 tasks and S3ConcurrentTaskLimit is set to 10, potentially 100 concurrent HTTP requests)
4. Depends on the environment -- if you have many S3 repositories and a lot of jobs using them, potentially, but at first blush this seems a bit high
5. Can you be more specific on this one? You mean the log folder size or the S3 "SlowDown" responses? For the latter, Veeam utilizes an exponential 'back-off' mechanism whenever an S3 system returns a SlowDown response; during these periods, Veeam slows the number of requests

As for the issue itself, as alluded to above, while the config parameter may assist here, depending on how heavily S3 is being utilized there may be other places to check.

I would advise continue with Support and push for a closer review of the behavior here (e.g., does it occur only during deletes, only during specific time periods, etc) -- if it's really about Checkpoint removal then there may be other options to adjust, but best to wait for Support's review.

If there are concerns on the handling of the case, please use the Talk to a Manager button in the case portal; this will connect you with Support Management and they will review the case and allocate additional resources if necessary.

[jcofin13]

Thank you. I will continue to wait for supports response and work with them but i appreciate this.

I will clean up some of the older logs in the checkpointremoval/system folder prior to april of this year (which was our original ticket on this).

For 5., im specifically looking for alerting on the Slowdown message in the logs with an ouside tool like veeamone or vbr itself so it alerts on it and we can get it looked at before the drive becomes full.

I wonder if the read requests its complaining about are per bucket or per source ip or per aws subscription or what the actual metric is that causes it to request slow down. The subscription itself is a single bucket. Im going to look at the aws side today to see if there is any indication of this error.

[david.domask]

Happy to shed some light on the matter, let's see what Support comes up with.

> For 5., im specifically looking for alerting on the Slowdown message in the logs with an ouside tool like veeamone or vbr itself so it alerts on it and we can get it looked at before the drive becomes full.

To my knowledge, "No".

Probably for the meantime you can use Windows File and Folder auditing to check folder size.

I believe the API throttling is done per bucket.

[jcofin13]

Thanks for the info. IS there a schedule to the checkpointremoval process that can be set or is it done when a backup finishes? I will say that there is a lot of activity on this VBR server as it backs up a lot of SQL DBs and is constantly running the tlog backups as well. This is the only server where i have seen this behavior pop up so im wondering if its more tied to the tlog backups or the sql part of the backups thats causing the addtional requests. The dbs are quite large as well.

[jcofin13]

also....upon review today, it looks like the dated folder in this folder from 10/8 used over 4gb in just one day. That seems excessive. I have updated the ticket with this asking why. I do see the slow down message in the base agent log folder quite a few times.

[jthornton]

Pardon the necro, but we're having the same issue on a Cloud Connect VM; support suggested it's safe to delete the old folders manually or via some custom script.

I'd love to see a built-in feature (just a simple registry key, really) that lets us configure auto-pruning logs older than a specific date. To my understanding, the registry keys in this KB (such as MaxLogCount and MaxLogSize) can be used to help manage the per-day logs: https://www.veeam.com/kb1825

[jcofin13]

Our team deletes them with a script on a regular basis keeping 3 to 6 month if i remember correctly.

WE were able to get our logs to quiet down quite a bit once we stopped S3 Errors using a combination of registry entries to limit the concurrent sessions to aws.

That said we stll get some larger log nights. ON a good night we get 15 to 50mb and maybe a log file count of 10 -15 per night. When month end or gfs happens there is a lot more. At month end we get a few gb of logs and anywhere between 400 - 800 separate log files in that nights dated checkpoint removal folder.

Its not as bad as when we were showing errors in the logs from s3. That seemed to be sporadic. Some nights were error free and small and some here many gb and had the errors in the logs repeating over and over generating the size.

The errors we were seeing were generated in the Agent.Cleanup.BlobRepo.log.Index.Gate file (the base file). Opening that file and searching for "S3 Error" or "Slow Down" showed the exact errors form AWS calling for you to slow down your send rate to them. I imagine Azure blob would be the same sort of error but i have not seen an AZ Blob instance to be able to check it. That said i have used azcopy with azure blob (outside of Veeam) and it will generate all kinds of odd errors...that are misleading. Setting environmental variables on the number of threads or sessions would cure that..so i assume if that same sort of thing is an issue with azure blob and veeam, there are probably reg entries you could add to limit the session counts to azure. Im sure it logs in the same way but again i have not been able to confirm. I know the regkeys we added were "s3" specific so i dont know that they would be universal for azure blob. Just food for thought.

Yeah it would be kind of nice if there was some tweaks in the ui for logs and even some of these cloud provider errors to throttle back if the dont play nice. The best thing i did was open a case on this to clean up the error portion of my issue.

[jthornton]

Right, that makes sense; we'll probably set up a similar cleanup task.

It looks like our flood of logs is also related to S3, Wasabi, in our case, so we'll investigate that angle next to see if we can make some optimizations. At first glance, the biggest source of noise for us appears to be when deleting old data in object storage that is no longer part of the backup chain and outside of retention and immutability timeframes.

Thank you for the further insight on the issue!

[jcofin13]

I believe we had to create an entry for S3ConcurrentTaskLimit in the registry. IF your case is the same or similar, maybe open a support case and mention this setting to see if it might be related. Once we got that added and tweaked to the right value our errors went away without affecting performance.