Hyper-V snapshot merge or pause between jobs.

[marcinbojko]

Current status:
We have a batch of big jobs, chained each after previous job. Jobs are quite big (aprox 15 TB), CBT on, there is no way to tell how many data will be backuped, so no other way as chain them all together to have as low impact as possible.
Snapshot to merge (each job) is sized at 200-300GB. Merging takes a while, and there is no way to control delaying another jobs, since previous job is still merging in background. Veeam doesn't wait for completing merging as a part of whole job, it just send 'merge' to hypervisor and starts another job.
As predicted it has big impact on performance (1 background merging job, 1 job in progres, and regular use of VM).
Is there a way to:
1) force veeam to wait until background merging is completed?
2) create pause between chained jobs?

[Shestakov]

Hello Marcin,
Job chaining is no considered as the best practice. Could you elaborate on the impact on performance?
I would rather run jobs in parallel, but throttle them where problems occur.
Do you use off-host proxy by the way?
Thanks!

[Vitaliy S.]

How many concurrent tasks do you use for the proxy and repository servers?

[foggy]

You can use script that checks the job status and starts the next one with some delay. However duration of this delay is somewhat you should take out of the air or at least based on your observations of previous job runs. There's no awareness of how long it could take to merge the VM backup checkpoint after volume snapshot is deleted.

[marcinbojko]

@Shestakov.
JObs chaining worked perfectly.
About impact on performance: when one job is finishing its tasks it send 'merge' to hypervisor. Next job is running few second later and, until background performance (of first job) is completed we have unusually high load (merge, job, and regular vm use)
We do not use off-proxy.
@Vitaliy.S -
Only one. One job finishes and then other chained jobs are kicking in - ALWAYS 1 job at the time.

@foggy
Thanks, but I am looking for built-in options to controll this. If I am not mistaken, on vmware, job waits until merging is completed, and then starts another chained job. In hyper-v background merging is treated as 'job finished, nothing to do here, move along'

[Shestakov]

Marcin, there is no option in UI to add an interval between jobs.
As for performance, you wrote

until background performance (of first job) is completed we have unusually high load

The question here was about the type of high performance, is it a high latency on the datastores, memory usage on the host or anything else?
There are techniques such as datastore IO control that you can leverage.
Thanks!

[marcinbojko]

@Shestakov
Thanks for the info. Performance issues are mostly related with very high disk latency and skyrocketing queues on CSVs.
And about datastore IO - I have Veeam Standard.

So, without scripting (I would like to avoid this if possible) - is there any way to control this behaviour?

[Shestakov]

I see your point. As was mentioned, there is no UI option, so script looks like the easiest way.
By the way, you can reduce impact on disk by leveraging hardware VSS provider for backup jobs. Also check if you have any redundant checkpoints you don`t need.

[marcinbojko]

Done and done. Hardware providers are not present for our OS (2012 R2).
How can I request those features in future releases? (waiting for a merge to complete and pause between jobs)

[Shestakov]

Your feedback is taken into account, since the forums are constantly reviewed by VBR product management team.
Thanks!

[Gostev]

I recommend upgrading to Enterprise Edition, as Backup I/O Control is designed to addresses not only this, but also many other similar scenarios where I/O from backup processes may impact production workload availability. There is little sense for us to address this one specific case separately, especially when the solution already exists in the product as a part of another feature. More importantly, this will not be a good solution for you either (in the bigger picture), because without Backup I/O Control, you will still struggle from performance impact in many other scenarios. Right now you only see one door you think you need to break through, however there are more doors behind it.

[marcinbojko]

Dear Gostev,
I couldn't disagree more. Instead of paying more (so much more) money for most of the features I am never gonna use, I would like to propose a 2-door solution.
1. Manually add pause between chained jobs.
2. Wait for a background merge to complete.

If I am not mistaken (as I was told by Veeam Engineer from Poland) VMWare edition waits for a merge to complete.

I would really like to upgrade to Enterprise Plus even, but as for now we are not an academic or goverment institution - we have count every penny we have.

[Vitaliy S.]

VM checkpoint merge is handled by Hyper-V itself and since we are dealing with volume snapshots we do not control this process. Once volume snapshot is removed, Hyper-V starts committing VM checkpoints in an asynchronous manner, so pausing backup job is not always possible. Hope this makes sense.

[Gostev]

Marcin, OK let's agree to disagree on this one

What's more important for me is that you have correct expectations about your request. Backup I/O Control is the existing feature of Enterprise Edition and we will not be introducing point solutions to the specific issues it already addresses into Standard Edition, as this makes no technical or business sense.

As far as a couple of points you raise above:

1. You can already do that today using post-job script functionality (just have that script sleep for the required amount of time).

2. As Vitaly already noted - unlike with VMware backup which is based on VM-level snapshots, in case of Hyper-V we have no insight into the merge process due to operating with volume-level snapshots.

However, Hyper-V 2016 will have the new backup architecture that is also based on VM-level snapshots, so its behavior will match VMware backup, where the job waits for snapshot merge to complete.

[marcinbojko]

Thanks for reply guys.
I'll try post-job script approach.
@Vitaliy S.
Yes, it does, it explains to process perfectly.
@Gostev.
Let's stay with disagree

[PenguinSSH]

Marcin, all,

We're also doing job chaining for huge file servers to split one huge 12TB VBK into several smaller VBK/disk. The reason for this is we send them to tape and don't want to restore 12TB of VBKs when we can restore just a 500GB VBK.

We've been running in disk merge process being interrupted by the other backup. To do this, I've written a script that queries the status of the merge process.

Our file servers are on Hyper-V and use a powershell script to wait for the disk merge process. The user that runs the script, Veeam service account, needs to have admin permissions on your Hyper-V hosts.

Code: Select all

$node = Invoke-Command -ComputerName vhost-prod-hq {Get-ClusterGroup -Name "SCVMM NameOfClusterGroup"}
$test = Invoke-Command -ComputerName $node.OwnerNode {Get-VM -Name YourNameofVM}


while ($test.Status -eq "Merging disks") {
    write-host $node.OwnerNode | Out-File C:\scripts\test.txt
    write-host $test.Status | Out-File C:\scripts\test1.txt
    write-host "Waiting for disk merge to complete"
    sleep 10
$test = Invoke-Command -ComputerName $node.OwnerNode {Get-VM -Name YourNameofVM}

}

That should give you the expected behavior although for a different purpose.

Edit for code:

[marcinbojko]

Thanks!