Extreme long merge times since Veeam v11

[bryanvaneeden]

Hi everybody,

I figured I would share this with the community. We've had multiple Veeam support cases but not really a final solution for the problems.

Problems:
- Since we've upgraded our environments (multiple isolated environments) we have been getting enormous merge times in comparison with Veeam v10. The merge times were around 1-4 hours before the Veeam v11 upgrade. After the Veeam v11 upgrade the merge times have increased to over 30 or 50 hours! The same amount of increments are being reported. The environment itself also hasn't changed since then.

- Since Veeam v11 we have also had several issues with the following message: "7-9-2021 06:58:39 :: Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.". This occurs completely at random on a couple of backup jobs. Other backup jobs to the same repository, or even other VM's in the same job, work without any issues.

Things we have done:
- For the merge times:
Compact files (that works in some environments) but not really a final solution, since this wasn't required in Veeam v10.
Remove "corrupted" files in some backup chains. This also worked for the merge times, but this was only for a couple of VMs.
Synthetic fulls on ReFS. This also works good, but this increases the reported backup file size in our billing systems so this isn't usable.

- For the AgentClosedException:
Veeam Support mentioned it was probably due to a failed amount of retries and suggested to try: https://www.veeam.com/kb1781. We are in the process of trying this but no final solution yet. This doesn't feel like a particular stable solution since this wasn't needed before the v11 upgrade.

I am hoping that there are other people around here that also noticed this after the Veeam v11 upgrade so that we can maybe come to a solution together.

Current running cases: Case #04917735, Case #04963265, Case #05009634

[bytewiseits]

We had an issue with ReFS merge times after upgrading from V10 to V11 with one particular VM backup job. It went from an average of 25mins to merge hourly increments for 4x VMs to over a day. Other larger backup jobs were merging fine though post upgrade.

It was odd as the repository itself was idle while the VBR server maxed out a CPU core per VM for the majority of the merge process. For each increment needing to be merged, the Veeam.Backup.Manager.exe process CPU usage would eventually drop then the merge would actually take place on the repository only taking a few minutes. From what we could see in the VBR logs it was recreating encryption keys over and over and over. Health checks all passed fine. Compacts also succeeded but took days instead of the usual few hours on V10.

Unfortunately after months of back and forth with support with debug logs and test builds we gave up trying to work out the why. We removed the job's backup files entirely (including .VBM) then forgot all unavailable backups in VBR. After running the job and recreating the backup chain & VBM it has been working fine since.

Maybe check and see if any VBR CPU cores are maxing out from the Veeam.Backup.Manager.exe process while the repository server idles? Would be interested to see if it was at all similar to what we saw.

[bryanvaneeden]

Hi bytewiseits,

Thank you for the reply. I've actually already checked the repositories but none of them are having CPU issues so far. Our repositories (on all seperate environments) are large physical boxes with loads of storage that should be able to handle the workloads a lot quicker like it did on v10.

Compacting does work to some extent like I mentioned in the first post, but this is not a viable solution for the long run. I am now in the process of adding the following registry keys to the environment:

VBR Server:
Value: UseUnbufferedAccess
Type: REG_DWORD
Value data: 0

Repository:
Value: DisableHtAsyncIo
Type: REG_DWORD
Value data: 1

Veeam support suggested this should probably fix some things. They have seen a performance increase (not quite like v10) but near v10 when applying these values. We actually did apply these already on another environment and that didn't have any effect at all.

I will come back once I added the keys again on the other environment.

The most probable reason that the performance on ReFS specific environments has decreased is that in Veeam v11 they have made unbuffered access and async I/O a default. In some scenario's this degrades the performance, hence the need for the registry keys.