Performance SOBR after Upgrade v10 to v11

[wane]

Hi all,
we have upgraded our backup landscape Enterprise Plus from v10 to v11. We are backing up to a SOBR consisting of 4 extents (Windows 2019, ReFS) with the following each server:

- 512 GB RAM
- 2x AMD EPYC 7252 8-Core Processor 3.10GHz, in total 16 Cores

We write the backups with backup copy jobs to another SOBR, also consisting of 4 extents (Windows 2019, ReFS) with the following equipment:

1x Server with the following:
- 2x AMD EPYC 7252 8-Core Processor 3.10GHz, in total 16 Cores
- 256GB RAM

3x Server with the following each server:
- 2x AMD EPYC 7282 16-Core Processor 2.80GHz, in total 32 Cores
- Fibre Channel connection
- Additional proxy for storage snapshots

The repositories are all set with a "Limit maximum concurrent tasks to: 14", i.e. the SOBR for Backups and Backup Copies then has 4x14= 56 tasks available to each SOBR.
The 3 hardware servers with fibre channel are additionally set as proxies with "Max concurrent tasks: 12" each.

In addition to the 3 hardware proxies, we have a total of 16 virtual proxies (Linux and Windows mixed). The virtual proxies each deliver 4 "Max concurrent tasks" with 6vCPUs per virtual proxy.

In total, we have (16 virtual * 4 tasks) + (3 hardware * 12 tasks) = 64 + 36 = 100 concurrent tasks for the proxies. Approximately 1600 VMs are backed up.

After the upgrade to v11, however, we have extreme performance problems. The runtimes of both the backups and the copies are very long and no longer tolerable. The jobs stop with the message:

Code: Select all

20.03.2021 06:25:31 :: Waiting for backup infrastructure resources availability  

Does anyone have an idea what the problem could be or suggestions for a solution (Maybe some settings concurrent tasks or whatelse)? Or does anyone have the same problem?

[HannesK]

Hello,
if your backups got slower overall, what is your support case number? We expected faster backups with the V11 improvements.

In General: if you have 100 proxy tasks and only 56 repository tasks, then it's expected that the system is waiting for resources. BUT, it should write which resources are missing. The general "waiting for backup infrastructure resources" message might have other reasons (support can figure out).

I don't know how the load is on your repository servers, but I would at least increase tasks by factor 1.5 (even 2.0 should be fine). I also assume, that you are using per-machine backup files. https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Best regards,
Hannes

[FeragTec]

We have the same problem. After the update to V11, the backups no longer work properly.

Direct backup to a repository is still possible to some extent.
Backing up on SOBR is practically no longer possible.
I'm already longer with the support.
Support ID: # 02262938

[wane]

Hi Hannes,
Thanks for your reply. The case number is #04711828.

Yes, you are right: per vm-backup-files.

Our VMs have often more than one disk. Therefore we have more proxy tasks than repo tasks because in my mind, one proxy task stands for a VMDK, however a repository task stands for a VM

[HannesK]

Hello,
@FeragTec: your ticket is processing into R&D, so thanks for insisting on "I want my performance back"

@wane: I asked support to check / link your case with the other one. I assume that you are not using Windows deduplication in combination with REFS (because that would have failed much earlier I assume). If you could keep the settings as they are for a little longer, that would help the engineers.

Side note: tasks are per-disk on repositories and proxies except for merges where it is one task per chain.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

For every disk of VMs added to the job, Veeam Backup & Replication creates a new task.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

For per-machine backup chains, Veeam Backup & Replication creates 1 task per every VM chain (that is, every VM added to the job).

Best regards,
Hannes

[wane]

FYI: the first patch for v11 soled the performance problem.

Check this link out: https://www.veeam.com/kb4126