SQL transaction logs jobs prevent other backup jobs from running

[FFulleringer]

Hello everyone! I'm hoping to get a little help. I'm having trouble when scheduling SQL Server logs backup (I set them to run every 15 minutes), if these jobs are active every other backup job won't start, they keep forever waiting for backup infrastructure resources availability and eventually fail with the error: Backup proxy not ready.
My backup server is Veeam Backup and Replication 11 running on Windows Server 2019, it is also the backup proxy since I only have 1 server dedicated to this. My repository is an HPE StoreOnce. Both server and storage were recently installed.
Before this I had a Windows Server 2012 and an IBM v7000 lun for repository, never had an issue like this (the number of virtual machines and databases haven't changed). The jobs were configured the same way as now.

The concurrent tasks for the backup proxy is set to 6, since the server has 6CPU and the concurrent tasks for the StoreOnce repository is unlimited.

Since the transaction logs jobs run constantly the resources never become available for the backup jobs. the SVC.Veeambackup log file won't display any error but it shows how the tasks remain sort of idle:

>[RTS] Tasks count statistics: [Completed: 0], [Pending: 13], [Preparing: 0], [Ready: 0], [ResponseGiven: 0], [WaitingForComplete: 27]


Like I said, it's the first time I have this problem but also the first time I use a Storeonce repository, maybe something is misconfigured.

Will appreciate any clue or idea provided.

Thanks a lot!

[HannesK]

Hello,

and the concurrent tasks for the StoreOnce repository is unlimited.

that's a bad idea. As you have an all-in-one system with only 6 CPU cores, it should be maximum 6. Probably you are already "overbooked" and below minimum system requirements if you sum up all your roles.

I currently cannot see, how you run out of proxy tasks if everything is configured exactly the same as before. Normally, I would just say "increase the proxy tasks", but the environment is already overbooked.

I see that you ran Server 2012 before: did you run an older VBR version on that server, or was it also V11?

Best regards,
Hannes

[PetrM]

Hello,

Just one thing:

'm having trouble when scheduling SQL Server logs backup (I set them to run every 15 minutes), if these jobs are active every other backup job won't start, they keep forever waiting for backup infrastructure resources availability and eventually fail with the error: Backup proxy not ready.

It cannot be related to T-log backup jobs as they do not consume task slots: log backup is a critical operation for a database that must be performed according to the strictly defined schedule to prevent the complete filling up of the production disk, so we never wait for infrastructure resources availability in this case. I guess there are some "orphaned" tasks pending somewhere in our service cache and most probably the issue will disappear for some time after the Veeam Backup Service restart. Anyway, I recommend opening a support case for a proper technical analysis.

Thanks!

[FFulleringer]

Hello,

that's a bad idea. As you have an all-in-one system with only 6 CPU cores, it should be maximum 6. Probably you are already "overbooked" and below minimum system requirements if you sum up all your roles.

I currently cannot see, how you run out of proxy tasks if everything is configured exactly the same as before. Normally, I would just say "increase the proxy tasks", but the environment is already overbooked.

I see that you ran Server 2012 before: did you run an older VBR version on that server, or was it also V11?

Best regards,
Hannes

Thanks for your reply. The 2012 was running Veeam B&R 9.5. Not 11. I just don't know how many CPUS the Storeonce has but the HPE engineer who installed the storage told me to keep it unlimited (tasks for the repository). Tasks for the Proxy are in 1xCPU.

[FFulleringer]

Hello,

Just one thing:


It cannot be related to T-log backup jobs as they do not consume task slots: log backup is a critical operation for a database that must be performed according to the strictly defined schedule to prevent the complete filling up of the production disk, so we never wait for infrastructure resources availability in this case. I guess there are some "orphaned" tasks pending somewhere in our service cache and most probably the issue will disappear for some time after the Veeam Backup Service restart. Anyway, I recommend opening a support case for a proper technical analysis.

Thanks!

Today I checked the backups from the weekend, one of them was still "running". It backed up 3 VM's and stopped, waiting for resources availability. Backup Proxy Not Ready for 20 hours. Edited the job, disabled guest proccesing and the job started running again, finished in about 5 minutes.

That's why I think the problem is Transaction logs / Application Awareness related.

Thank you.

[FFulleringer]

Opened support case. Case # 05688419.

[PetrM]

HI Cosme,

It's an interesting observation but anyway I would treat it as a technical problem because I don't see any objective or design-related reasons for the jobs waiting for T-logs backup completion. I believe our engineers should have a look at debug logs and probably Veeam Backup Service trace or dump to sort out the issue.

Thanks!