Hyper-V job scheduling and configuration

[Jonathan]

The problem we’re facing, is that Veeam B&R spends an enormous amount of time waiting for the Hyper-V hosts (resource availability, snapshots, et cetera), despite the fact that the hosts have a huge amount of spare horsepower and bandwidth and the storage is 30 percent flash-based. Our hosting provider has opened tickets with Veeam on more than one occasion, but no clear culprit was ever found. This might be a result of inefficiency on the Hyper-V part, but my guess is there’s something to be won by constructing the right chain of jobs.

Currently, we have 4 Hyper-V hosts, serving a little over 100 VM’s. All VM’s are placed on shared volumes on an iSCSI storage array. Only these Hyper-V hosts have access to this array, so the only proxy option available at the moment is on-host. Total size is about 15TB, daily changed blocks about 800GB before compression.

Our backup server has an end-to-end 10GB ethernet connection to the Hyper-V hosts, as well as two replication target servers with local storage. Backup storage is local to the backup server. We run reverse incremental backups of each VM, configured as one job per shared volume (about 15 of them). Max concurrent tasks per host is set to 4. VM’s are mostly Windows Server 2012 R2 domain joined machines, but there’s also about a dozen linux servers. Each of these VM’s must be backed up as well as replicated within a time frame of between 22:00 and 6:00 hrs. I like the idea of selecting shared volumes instead of individual VM’s, because it requires no additional configuration when you create a new machine in VMM for it to be backed up.

Available options are for example one or two large backup jobs, containing all shared volumes, and a replication job with a similar setup. We could run these simultaneously, or one after the other. We could also run replication jobs off the backup files.

What would you do?

[Shestakov]

Hello Jonathan,

I would start from bottleneck analysis. Looks like your performance can be improved in a multiple ways, but we need to know where the bottleneck is. Could you provide jobs statistics?

Currently, we have 4 Hyper-V hosts, serving a little over 100 VM’s. All VM’s are placed on shared volumes on an iSCSI storage array. Only these Hyper-V hosts have access to this array, so the only proxy option available at the moment is on-host. Total size is about 15TB, daily changed blocks about 800GB before compression.

In general off-host processing provides better results and shifts workload from the production host. So if you want to improve the performance consider off-host proxy.

Thanks!

[foggy]

This thread might be useful.

[Jonathan]

I would start from bottleneck analysis. Looks like your performance can be improved in a multiple ways, but we need to know where the bottleneck is. Could you provide jobs statistics?

In general off-host processing provides better results and shifts workload from the production host. So if you want to improve the performance consider off-host proxy.

To me, it's clear the current Hyper-V/VMM configuration IS the bottleneck. My preferred solution would be to give the Veeam B&R server direct access to the storage API, bypassing the Hyper-V hosts. Unfortunately, as this is a shared storage platform, our hosting provider is reluctant to do this. Also, I don't have any access to the Hyper-V hosts other than a limited VMM console. This leaves me half-blind. The only way I can influence backup performance is through the B&R management console.

A few days ago, I changed all of my replication jobs to use the backups as a source, rather than production storage. This had a dramatic effect on the overall replication time window. Individual throughput is roughly the same as before, but there were no traffic jams where one VM in a job would be waiting for hours to be processed.

Using the backup as a source for replication has its drawbacks. Ideally, I would like to be able to do replication multiple times a day for selected VMs. This is impossible if you don't use the actual production VM as a source. Also, the state of the replicated VM now depends on job options selected in the backup job (eg crash consistent, indexing, log flushing). But for now, I'm glad my replication jobs are finished some 9 hours earlier than before.

[Jonathan]

This thread might be useful.

I agree, that's why I changed my VM selection to one single volume per job. In theory, this would make sure all of the VM's in this job would fall under the same snapshot, locked by one Hyper-V host. This has had virtually no effect on the 'waiting for resources' issue. It's just a little more convenient.

[Shestakov]

Statement that bottleneck is configuration can mean several unrelated things.
In general when you try to improve the performance you need to know what is the slowest part of your backup job.
Could you provide you jobs` stats for us to understand which part of configuration needs to be improved first (Source/Proxy/Network/Target)?
Thanks!

[Jonathan]

For a typical backup job, statistics look like this:
Source 99% > Proxy 7% > Network 25% > Target 0%

Usually source is between 95 - 99, Proxy 5 - 10, Network 20 - 35, Target 0 - 3

[Shestakov]

So bottleneck is source.
Besides off-host proxy which is best solution for the bottleneck=source, you may leverage Hardware VSS provider if you haven`t done it yet.
By the way what is your processing rate?
Thanks

[Jonathan]

Throughput is somewhere between 100MB/s and 500MB/s. But I don't take these numbers too seriously since the actual data being transferred is usually a fraction of that. A lot of VM's are rather static machines for which the daily delta is less than 1 percent of total size.

As far as I'm aware, our hosting provider does use (Dell Compellent) hardware VSS provider.

[Shestakov]

Great.
Besides VSS provider and off-host proxy you may leverage the mentioned "replica from backup" option and choose a proper job schedule.
You may also automatically check possible hyper-v and VBR configuration issues with corresponding Veeam ONE reports which are available even in free product version.
Thanks!