Performance Testing with and without Proxy VM

[Frosty]

Just for the possible benefit of others, I thought I would post some details of testing that I did, comparing backup performance with and without a Veeam Proxy running on a VM.

ENVIRONMENT

3 x Dell R740 hosts (512GB RAM, 2 x 12-core CPU) running vSphere 6.5 Update 1
-- shared DAS storage: Dell EMC SC5020 storage array loaded with ~30TB of SSD RAID10 and RAID5
1 x Dell R740xd (32GB RAM, 1 x 12-core CPU) running Windows Server 2012 R2 and VBR 9.5
-- local storage: 40TB of 7200rpm nearline SAS (8 x 10TB RAID10)
10GbE networking to both the ESXi hosts and VBR backup server

PROXY VM

4 x vCPU
8GB RAM
1 x 40GB C: drive (normal Windows operating system files)

TEST VM

1 x 40GB C: drive (normal Windows operating system files)
1 x 100GB D: drive (99% full with mostly uncompressible data)

TESTING METHOD and KEY OBSERVATIONS

VBR 9.5 was installed on the R740xd physical server and backup was taken using the default proxy installed with VBR, so it was using NBD transport. Network throughput was observed to peak at 200MB/sec.

Windows Server VM was deployed and Veeam Proxy role was pushed out to it. Quick test was done to ensure that it could HotAdd disks and thus avoid NBD transport. A series of test backups (Active Full) of the test VM were performed, varying the number of CPUs in the Proxy VM to see their impact on throughput:

4 x vCPU = 100% CPU utilisation and 500MB/sec network throughput (primary bottleneck = Proxy)
6 x vCPU = 100% CPU utilisation and 510MB/sec network throughput (primary bottleneck = Proxy)
8 x vCPU = 67% CPU utilisation and 540MB/sec network throughput (primary bottleneck = Source)
12 x vCPU = 50% CPU utilisation and 575MB/sec network throughput (primary bottleneck = Source)

PRELIMINARY CONFIGURATION

We started with a Proxy VM with 4 x vCPUs and just 8GB RAM and then allowed our normal nightly backup cycle to run, with allocation of the Proxy set to Automatic Selection within VBR.

We expected a small improvement due to the 200MB/sec to 500MB/sec increased network throughput.

ACTUAL RESULTS

We were a bit surprised to find that all our backup jobs actually ran SLOWER, by a factor of 1.5x slower to as much as 5x slower.

We attribute this to the time taken within the backup job to actually perform the HotAdd of the disks, as VMs with fewer disks were processed faster and VMs with a LOT of disks had their backup times blow out the most. I recall reading something along these lines recently, but cannot remember where I read it.

MY CONCLUSIONS

* possible to get relatively large performance gains by switching from NBD Transport to using a Proxy+HotAdd (from 200MB/sec to around 500MB/sec)
* possible to get relatively small backup performance gains by dramatically increasing CPU to Proxy VM (5% to 10%)
* for many typical VMs with small disks and low rates of change, or VMs with a LOT of disks, it is probably better just to go with the default VBR proxy and with NBD transport

Interested to know whether people agree or disagree with those conclusions. I am by no means an expert and this is pretty rudimentary testing, so it is entirely possible that I have overlooked something.

FINAL CONFIGURATION for PRODUCTION USE

We have settled on a Proxy VM with 4 x vCPUs and just 4GB RAM (because memory usage seems pretty low) and will continue to monitor whether it is effective or not in our environment.

We will probably NOT allow VBR to automatically allocate the Proxy, but will instead set up nearly all our jobs to use the default VBR proxy role installed on the VBR physical server.

Scenarios where we might consider using the Veeam Proxy will include:
* taking initial Active Full backups of new VMs (roughly a 2x performance improvement)
* backups of VMs with large disks which also have a high rate of change

[nitramd]

Frosty,

Thanks for testing and posting the results, good information.

* possible to get relatively large performance gains by switching from NBD Transport to using a Proxy+HotAdd (from 200MB/sec to around 500MB/sec)

I can confirm the performance improvement for HotAdd. Most all of our proxies are virtual and I've seen up to 750 MB/sec with HotAdd; CBT activated. With update 3a, NBD performance is suppose to improve markedly. Believe the other "transports" will also get a performance boost.

* possible to get relatively small backup performance gains by dramatically increasing CPU to Proxy VM (5% to 10%)

Confirmed - having more CPUs is not necessarily better for performance.

[foggy]

More CPU's allow to run more concurrent tasks (process more VM disks) in parallel (provided the corresponding setting value is set accordingly).