Hyper-V snapshot merge or pause between jobs.

Hyper-V specific discussions

Hyper-V snapshot merge or pause between jobs.

Veeam Logoby marcinbojko » Fri Nov 27, 2015 11:01 am

Current status:
We have a batch of big jobs, chained each after previous job. Jobs are quite big (aprox 15 TB), CBT on, there is no way to tell how many data will be backuped, so no other way as chain them all together to have as low impact as possible.
Snapshot to merge (each job) is sized at 200-300GB. Merging takes a while, and there is no way to control delaying another jobs, since previous job is still merging in background. Veeam doesn't wait for completing merging as a part of whole job, it just send 'merge' to hypervisor and starts another job.
As predicted it has big impact on performance (1 background merging job, 1 job in progres, and regular use of VM).
Is there a way to:
1) force veeam to wait until background merging is completed?
2) create pause between chained jobs?
marcinbojko
Enthusiast
 
Posts: 31
Liked: 9 times
Joined: Thu Mar 06, 2014 10:40 am
Location: Lublin/Poland
Full Name: Marcin Bojko

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Shestakov » Fri Nov 27, 2015 11:24 am

Hello Marcin,
Job chaining is no considered as the best practice. Could you elaborate on the impact on performance?
I would rather run jobs in parallel, but throttle them where problems occur.
Do you use off-host proxy by the way?
Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Vitaliy S. » Fri Nov 27, 2015 11:40 am

How many concurrent tasks do you use for the proxy and repository servers?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby foggy » Fri Nov 27, 2015 1:55 pm

You can use script that checks the job status and starts the next one with some delay. However duration of this delay is somewhat you should take out of the air or at least based on your observations of previous job runs. There's no awareness of how long it could take to merge the VM backup checkpoint after volume snapshot is deleted.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby marcinbojko » Fri Nov 27, 2015 2:38 pm

@Shestakov.
JObs chaining worked perfectly.
About impact on performance: when one job is finishing its tasks it send 'merge' to hypervisor. Next job is running few second later and, until background performance (of first job) is completed we have unusually high load (merge, job, and regular vm use)
We do not use off-proxy.
@Vitaliy.S -
Only one. One job finishes and then other chained jobs are kicking in - ALWAYS 1 job at the time.

@foggy
Thanks, but I am looking for built-in options to controll this. If I am not mistaken, on vmware, job waits until merging is completed, and then starts another chained job. In hyper-v background merging is treated as 'job finished, nothing to do here, move along'
marcinbojko
Enthusiast
 
Posts: 31
Liked: 9 times
Joined: Thu Mar 06, 2014 10:40 am
Location: Lublin/Poland
Full Name: Marcin Bojko

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Shestakov » Sat Nov 28, 2015 8:54 pm

Marcin, there is no option in UI to add an interval between jobs.
As for performance, you wrote
marcinbojko wrote:until background performance (of first job) is completed we have unusually high load

The question here was about the type of high performance, is it a high latency on the datastores, memory usage on the host or anything else?
There are techniques such as datastore IO control that you can leverage.
Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby marcinbojko » Sun Nov 29, 2015 3:20 pm

@Shestakov
Thanks for the info. Performance issues are mostly related with very high disk latency and skyrocketing queues on CSVs.
And about datastore IO - I have Veeam Standard.

So, without scripting (I would like to avoid this if possible) - is there any way to control this behaviour?
marcinbojko
Enthusiast
 
Posts: 31
Liked: 9 times
Joined: Thu Mar 06, 2014 10:40 am
Location: Lublin/Poland
Full Name: Marcin Bojko

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Shestakov » Sun Nov 29, 2015 3:50 pm

I see your point. As was mentioned, there is no UI option, so script looks like the easiest way.
By the way, you can reduce impact on disk by leveraging hardware VSS provider for backup jobs. Also check if you have any redundant checkpoints you don`t need.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby marcinbojko » Mon Nov 30, 2015 6:34 am

Done and done. Hardware providers are not present for our OS (2012 R2).
How can I request those features in future releases? (waiting for a merge to complete and pause between jobs)
marcinbojko
Enthusiast
 
Posts: 31
Liked: 9 times
Joined: Thu Mar 06, 2014 10:40 am
Location: Lublin/Poland
Full Name: Marcin Bojko

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Shestakov » Mon Nov 30, 2015 10:50 am

Your feedback is taken into account, since the forums are constantly reviewed by VBR product management team.
Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Gostev » Sun Dec 06, 2015 1:40 pm

I recommend upgrading to Enterprise Edition, as Backup I/O Control is designed to addresses not only this, but also many other similar scenarios where I/O from backup processes may impact production workload availability. There is little sense for us to address this one specific case separately, especially when the solution already exists in the product as a part of another feature. More importantly, this will not be a good solution for you either (in the bigger picture), because without Backup I/O Control, you will still struggle from performance impact in many other scenarios. Right now you only see one door you think you need to break through, however there are more doors behind it.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby marcinbojko » Mon Dec 07, 2015 7:14 am

Dear Gostev,
I couldn't disagree more. Instead of paying more (so much more) money for most of the features I am never gonna use, I would like to propose a 2-door solution.
1. Manually add pause between chained jobs.
2. Wait for a background merge to complete.

If I am not mistaken (as I was told by Veeam Engineer from Poland) VMWare edition waits for a merge to complete.

I would really like to upgrade to Enterprise Plus even, but as for now we are not an academic or goverment institution - we have count every penny we have.
marcinbojko
Enthusiast
 
Posts: 31
Liked: 9 times
Joined: Thu Mar 06, 2014 10:40 am
Location: Lublin/Poland
Full Name: Marcin Bojko

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Vitaliy S. » Mon Dec 07, 2015 11:58 am 1 person likes this post

VM checkpoint merge is handled by Hyper-V itself and since we are dealing with volume snapshots we do not control this process. Once volume snapshot is removed, Hyper-V starts committing VM checkpoints in an asynchronous manner, so pausing backup job is not always possible. Hope this makes sense.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby Gostev » Mon Dec 07, 2015 12:51 pm

Marcin, OK let's agree to disagree on this one :)

What's more important for me is that you have correct expectations about your request. Backup I/O Control is the existing feature of Enterprise Edition and we will not be introducing point solutions to the specific issues it already addresses into Standard Edition, as this makes no technical or business sense.

As far as a couple of points you raise above:

1. You can already do that today using post-job script functionality (just have that script sleep for the required amount of time).

2. As Vitaly already noted - unlike with VMware backup which is based on VM-level snapshots, in case of Hyper-V we have no insight into the merge process due to operating with volume-level snapshots.

However, Hyper-V 2016 will have the new backup architecture that is also based on VM-level snapshots, so its behavior will match VMware backup, where the job waits for snapshot merge to complete.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Hyper-V snapshot merge or pause between jobs.

Veeam Logoby marcinbojko » Mon Dec 07, 2015 2:59 pm

Thanks for reply guys.
I'll try post-job script approach.
@Vitaliy S.
Yes, it does, it explains to process perfectly.
@Gostev.
Let's stay with disagree ;)
marcinbojko
Enthusiast
 
Posts: 31
Liked: 9 times
Joined: Thu Mar 06, 2014 10:40 am
Location: Lublin/Poland
Full Name: Marcin Bojko

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 13 guests