Hyper-V job scheduling and configuration

Hyper-V specific discussions

Hyper-V job scheduling and configuration

Veeam Logoby Jonathan » Wed Nov 30, 2016 8:24 am

The problem we’re facing, is that Veeam B&R spends an enormous amount of time waiting for the Hyper-V hosts (resource availability, snapshots, et cetera), despite the fact that the hosts have a huge amount of spare horsepower and bandwidth and the storage is 30 percent flash-based. Our hosting provider has opened tickets with Veeam on more than one occasion, but no clear culprit was ever found. This might be a result of inefficiency on the Hyper-V part, but my guess is there’s something to be won by constructing the right chain of jobs.

Currently, we have 4 Hyper-V hosts, serving a little over 100 VM’s. All VM’s are placed on shared volumes on an iSCSI storage array. Only these Hyper-V hosts have access to this array, so the only proxy option available at the moment is on-host. Total size is about 15TB, daily changed blocks about 800GB before compression.

Our backup server has an end-to-end 10GB ethernet connection to the Hyper-V hosts, as well as two replication target servers with local storage. Backup storage is local to the backup server. We run reverse incremental backups of each VM, configured as one job per shared volume (about 15 of them). Max concurrent tasks per host is set to 4. VM’s are mostly Windows Server 2012 R2 domain joined machines, but there’s also about a dozen linux servers. Each of these VM’s must be backed up as well as replicated within a time frame of between 22:00 and 6:00 hrs. I like the idea of selecting shared volumes instead of individual VM’s, because it requires no additional configuration when you create a new machine in VMM for it to be backed up.

Available options are for example one or two large backup jobs, containing all shared volumes, and a replication job with a similar setup. We could run these simultaneously, or one after the other. We could also run replication jobs off the backup files.

What would you do?
Jonathan
Enthusiast
 
Posts: 49
Liked: 1 time
Joined: Thu Oct 15, 2009 8:52 am

Re: Hyper-V job scheduling and configuration

Veeam Logoby Shestakov » Thu Dec 01, 2016 10:31 am

Hello Jonathan,

I would start from bottleneck analysis. Looks like your performance can be improved in a multiple ways, but we need to know where the bottleneck is. Could you provide jobs statistics?

Jonathan wrote:Currently, we have 4 Hyper-V hosts, serving a little over 100 VM’s. All VM’s are placed on shared volumes on an iSCSI storage array. Only these Hyper-V hosts have access to this array, so the only proxy option available at the moment is on-host. Total size is about 15TB, daily changed blocks about 800GB before compression.

In general off-host processing provides better results and shifts workload from the production host. So if you want to improve the performance consider off-host proxy.

Thanks!
Shestakov
Veeam Software
 
Posts: 5027
Liked: 419 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V job scheduling and configuration

Veeam Logoby foggy » Thu Dec 01, 2016 12:21 pm

This thread might be useful.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Hyper-V job scheduling and configuration

Veeam Logoby Jonathan » Sun Dec 04, 2016 9:30 am

Shestakov wrote:I would start from bottleneck analysis. Looks like your performance can be improved in a multiple ways, but we need to know where the bottleneck is. Could you provide jobs statistics?

In general off-host processing provides better results and shifts workload from the production host. So if you want to improve the performance consider off-host proxy.



To me, it's clear the current Hyper-V/VMM configuration IS the bottleneck. My preferred solution would be to give the Veeam B&R server direct access to the storage API, bypassing the Hyper-V hosts. Unfortunately, as this is a shared storage platform, our hosting provider is reluctant to do this. Also, I don't have any access to the Hyper-V hosts other than a limited VMM console. This leaves me half-blind. The only way I can influence backup performance is through the B&R management console.

A few days ago, I changed all of my replication jobs to use the backups as a source, rather than production storage. This had a dramatic effect on the overall replication time window. Individual throughput is roughly the same as before, but there were no traffic jams where one VM in a job would be waiting for hours to be processed.

Using the backup as a source for replication has its drawbacks. Ideally, I would like to be able to do replication multiple times a day for selected VMs. This is impossible if you don't use the actual production VM as a source. Also, the state of the replicated VM now depends on job options selected in the backup job (eg crash consistent, indexing, log flushing). But for now, I'm glad my replication jobs are finished some 9 hours earlier than before.
Jonathan
Enthusiast
 
Posts: 49
Liked: 1 time
Joined: Thu Oct 15, 2009 8:52 am

Re: Hyper-V job scheduling and configuration

Veeam Logoby Jonathan » Sun Dec 04, 2016 9:36 am

foggy wrote:This thread might be useful.


I agree, that's why I changed my VM selection to one single volume per job. In theory, this would make sure all of the VM's in this job would fall under the same snapshot, locked by one Hyper-V host. This has had virtually no effect on the 'waiting for resources' issue. It's just a little more convenient.
Jonathan
Enthusiast
 
Posts: 49
Liked: 1 time
Joined: Thu Oct 15, 2009 8:52 am

Re: Hyper-V job scheduling and configuration

Veeam Logoby Shestakov » Sun Dec 04, 2016 7:32 pm

Statement that bottleneck is configuration can mean several unrelated things.
In general when you try to improve the performance you need to know what is the slowest part of your backup job.
Could you provide you jobs` stats for us to understand which part of configuration needs to be improved first (Source/Proxy/Network/Target)?
Thanks!
Shestakov
Veeam Software
 
Posts: 5027
Liked: 419 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V job scheduling and configuration

Veeam Logoby Jonathan » Mon Dec 05, 2016 7:42 am

For a typical backup job, statistics look like this:
Source 99% > Proxy 7% > Network 25% > Target 0%

Usually source is between 95 - 99, Proxy 5 - 10, Network 20 - 35, Target 0 - 3
Jonathan
Enthusiast
 
Posts: 49
Liked: 1 time
Joined: Thu Oct 15, 2009 8:52 am

Re: Hyper-V job scheduling and configuration

Veeam Logoby Shestakov » Mon Dec 05, 2016 12:19 pm

So bottleneck is source.
Besides off-host proxy which is best solution for the bottleneck=source, you may leverage Hardware VSS provider if you haven`t done it yet.
By the way what is your processing rate?
Thanks
Shestakov
Veeam Software
 
Posts: 5027
Liked: 419 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V job scheduling and configuration

Veeam Logoby Jonathan » Wed Dec 07, 2016 7:37 am

Throughput is somewhere between 100MB/s and 500MB/s. But I don't take these numbers too seriously since the actual data being transferred is usually a fraction of that. A lot of VM's are rather static machines for which the daily delta is less than 1 percent of total size.

As far as I'm aware, our hosting provider does use (Dell Compellent) hardware VSS provider.
Jonathan
Enthusiast
 
Posts: 49
Liked: 1 time
Joined: Thu Oct 15, 2009 8:52 am

Re: Hyper-V job scheduling and configuration

Veeam Logoby Shestakov » Wed Dec 07, 2016 10:38 am

Great.
Besides VSS provider and off-host proxy you may leverage the mentioned "replica from backup" option and choose a proper job schedule.
You may also automatically check possible hyper-v and VBR configuration issues with corresponding Veeam ONE reports which are available even in free product version.
Thanks!
Shestakov
Veeam Software
 
Posts: 5027
Liked: 419 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 5 guests