Veeam agent t-log backup causes availability group to go offline.

[FrancWest]

Hi,

case #07063030.

After upgrading to V12 we notice that our SQL Always-On Availability group becomes inaccessible during a transaction log backup job after running a database maintenance plan which optimize the indexes. This produces 100+ GB worth of transaction log backups. When the centrally managed Windows Agent job backs-up these transaction logs it consumes 99% of CPU on the secondary node of our cluster which brings the cluster down due to insufficient CPU resources for SQL server.

I've noticed that the priority of the veeamagent.exe process is set to normal. When manually setting it to 'Below normal', the cpu usage of veeamagent.exe drops to 60% which is enough to keep the cluster online.

When we run a full backup using the Windows agent, then the veeamagent.exe process is set to 'low' priority when it's running. I assume this is because I've configured the 'Throttle Agent backup activity' under the 'I/O Backup control' option on the protection group the SQL cluster is in.

The issue here seems to be that the transaction log backup job doesn't honour this throttling setting and always sets it to normal priority. In general this isn't an issue since transaction log backups are usually small, especially when set to 15 minutes as in our case. However, after running the maintenance job we have 100+ GB worth of transaction log backups and this causes the veeamagent.exe process to consume all CPU when setting to normal priority. I've no idea why though.

I've tried setting the process priority to ' below normal' or 'low' through the registry so it's persistent, but this is being ignored, Probably because it's set by the process itself.

Anyone else notice this behaviour?

[PetrM]

Hi Franc,

This is the first time when I've heard about the issue. I will discuss this with my colleagues from QA, maybe they saw something similar in our labs during performance testing. Please continue working with our support team, I guess process profiling and debug logs analysis are required to reveal the source of the problem.

Thanks!

[FrancWest]

Ticket has been raised to higher support level. As a temporary solution I've created a scheduled task on the SQL server node that runs every 5 minutes and sets the priority of the veeamagent.exe process to low.
Today after the optimization job ran, the SQL cluster stayed online. So that workaround works. However, Veeam should set the process priority correctly by itself.

[PetrM]

Hi Franc,

In the meantime, my colleagues from QA have confirmed the issue in our labs. I will ask our support team to check whether your problem is the same as the one that was found by our QA. Basically, I agree with your proposal to set a lower process priority.

Thanks!

[FrancWest]

Thanks for the update!