Backup Copy slow after V9 upgrade

[evansdp]

There's a thread about regular backup slowness in V9 (veeam-backup-replication-f2/veeam-v9-ba ... 35255.html) and we'll be testing some of the approaches described there. But our situation is slightly different and I'm curious if anyone else is seeing this:

Our Backup Copy jobs (one in particular) have slowed to a crawl (80 hours in one case) since the V9 upgrade. Our regular Backup jobs are doing fine.

We have a single Veeam Server that also serves as a proxy plus a separate proxy server. Both Veeam servers are 2012.R2. We have two repositories on a Linux-based array with 10Gb links between the Veeam servers and repositories. All of the Backup Copy jobs point to the proxy as the bottleneck but none of the regular backup jobs do.

We're preparing to add another proxy (or 2).

Anyone else seeing this behavior?

Dan Evans
Pacific Lutheran University

[foggy]

Dan, how repositories involved in the backup copy job are added to Veeam B&R?

[evansdp]

Foggy-- I'm not sure of your exact question. There are 2 repositories and both were added through the Backup Infrastructure tools on the Veeam server. Only one of the repositories is involved in the backup copy jobs (we use one repository for daily, "regular" backups and the other for backup copy, "archive" jobs).

[tsightler]

Any chance this is a fairly large job with a lot of VMs/disks? If so, you might want to open a case as there are a few known regressions in BCJ performance in v9 including one with an existing hotfix.

Also, it's important to note that proxies aren't really used for Backup Copy Jobs so the bottleneck stats can be a little bit misleading. Backup copy jobs are performed "repo-to-repo" so the "proxy" referred to in a BCJ is actually the source repostory.

[alanbolte]

Also make sure you don't have parallel processing disabled in the global settings.

[evansdp]

Thanks for the feedback so far.

@tsightler-- There's a mix of jobs-- some have only 1 or 2 VMs while others have 5-9. Sizes of VMs range from 80GB to 4TB but the amount of data processed by Veeam ranges from 2GB to 126GB. So yes, there are some "fairly large" jobs with many VMs or disks and I will be putting in a ticket. Thanks for the comment about the BCJ proxy being the source repository. That suggests we might benefit by moving our BCJ timing from ~midnight (when many of our regular backup jobs are running) to mid-day (when we have very few regular jobs running).

@alanbolte-- I checked and parallel processing is enabled.

> Dan

[foggy]

I was after the repository type used while adding the storage in Veeam B&R - Linux or CIFS? This defines where the source and target data movers are running and might explain why the proxy (effectively the source repository data mover) is reported as bottleneck.

[evansdp]

Both repositories are Linux (CentOS 7) and sit on the same physical box (32GB RAM with 45x3TB drive array configured with mdadm as 2 RAID6 LVMs). One volume is mounted as /raid6a and used for daily backups. Second volume is mounted as /raid6b and used to store the BCJs. Source servers are mix of Linux and Windows.

> Dan

[foggy]

As Tom has already mentioned, there were some issues with backup copy job performance after the upgrade (specifically, when target cache stopped to be used), so I recommend to contact support for verification whether it applies to your situation. Also, in your case both source and target data movers are running on the same server, this might be the reason of showing this server as the bottleneck.