Host-based backup of Microsoft Hyper-V VMs.
Post Reply
baltechnique
Novice
Posts: 8
Liked: 1 time
Joined: Sep 30, 2013 7:11 am
Full Name: Network Consulting
Contact:

VeeamB&R and CSV

Post by baltechnique »

Hi,

why is Veeam doing a volume snapshot for each VM, even if these VMs are on a Cluster Shared Volume.

An only snapshot for each CSV would be far enough. No ? And backup would finish faster.

PS : tested on 2008R2 HyperV Cluster, but not on 2012R2
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VeeamB&R and CSV

Post by Gostev »

Hi

Reliability reasons. This approach does not currently work reliably (issues on Hyper-V VSS side). Too many intermittent VSS failures.

We actually have this logic implemented, and may end up enabling it in v8, but it will still be heavily limited to no more than a few VMs per CSV being processed at once. Folks with really fast SAN will be able to tweak the maximum amount, but for average hardware, the limit will need to be pretty low (think 3-4 VMs at once).

Thanks!
baltechnique
Novice
Posts: 8
Liked: 1 time
Joined: Sep 30, 2013 7:11 am
Full Name: Network Consulting
Contact:

Re: VeeamB&R and CSV

Post by baltechnique »

Hi,

OK for your explanation, but in facts i dont understand why :)

Since there is a SAN snapshot of the volume, each VM file is available separately (i'm not talking about vss inside the running VMs for application consistency), and since the snapshot is finished on the SAN, the redirected mode is back to normal mode for hyperV.

But i do agree that even with hardware VSS there are some snapshot failure, so in that case you failed to backup correctly each VM in this volume.
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: VeeamB&R and CSV

Post by rskinner »

Gostev wrote:Hi

Reliability reasons. This approach does not currently work reliably (issues on Hyper-V VSS side). Too many intermittent VSS failures.

We actually have this logic implemented, and may end up enabling it in v8, but it will still be heavily limited to no more than a few VMs per CSV being processed at once. Folks with really fast SAN will be able to tweak the maximum amount, but for average hardware, the limit will need to be pretty low (think 3-4 VMs at once).

Thanks!
This would be a great feature. Many of our VMs have very few data changes and the time to create the CSV VSS snapshot (2-4 mins) is one of the longest stages in the backup process for each VM. The ability to have 3-4 VMs backed-up per CSV VSS snapshot would greatly speed up our backups.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VeeamB&R and CSV

Post by Gostev »

baltechnique wrote:Since there is a SAN snapshot of the volume, each VM file is available separately (i'm not talking about vss inside the running VMs for application consistency), and since the snapshot is finished on the SAN, the redirected mode is back to normal mode for hyperV.
Yes, but unfortunately it does not work like that today :)

Hyper-V backup is an incredibly integrated monster that touches too many things. Part of the process actually goes into each guest, creates VSS snapshots inside of the running VMs, and then after SAN snapshot is created, it goes back there and performs the process known as AutoRecovery. So as soon as you have more than a few VMs in the consistency group, this process starts to fail or timeout due to much extra load on the datastore. Hyper-V team at Microsoft is well aware of this problem, in fact I've just discussed this issue again with them last month at TechEd. But the whole talk was under uber NDA, so I cannot really comment beyond known facts.
samh22a
Enthusiast
Posts: 35
Liked: 4 times
Joined: Nov 10, 2013 2:03 pm
Contact:

[MERGED] Snapshot multiple VMs at once

Post by samh22a »

Hi,

Let's assume the following scenario

- Hardware VSS Snapshot
- Hyper-V cluster
- CSV with 10 Virtual Machines

Is there any way to take a single snapshot, and backup the 10 VMs at once ? right now I can see multiple snapshots beeing created as it process all of those vm..and it doesn't seems to be very efficient

Thanks
rmeichsner
Novice
Posts: 4
Liked: never
Joined: Dec 02, 2014 2:10 pm
Full Name: Ruben Meichsner
Contact:

Re: VeeamB&R and CSV

Post by rmeichsner »

Hi Anton,

can you share any details under which circumstances this option should work in v8? The explanation in your kb is really general and doesn't contain much information regarding this option.
In our environment the setting doesn't seem to have any effect at all. (smb3 sofs)

What is the maximum number of Snapshots you recommend per smb3-Volume? If set to everything different than "1", we're facing VSS Errors from time to time...
Registrykey "HyperVRestrictConcurrentSnapshotCreation" already set to "1".

We're looking for a way to reduce the time, the backup procedure needs and get rid of the gaps in the backup process without any throughput.

Thanks,
Ruben
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VeeamB&R and CSV

Post by foggy »

Ruben, there is a setting in the job advanced settings that allows processing of multiple VMs from a single snapshot. The setting is enabled by default on the new jobs, however you should enable it manually for existing jobs.

Also, here's the quote from the What's New document regarding this feature:
Consistency groups: Jobs can now process more than one VM from the volume snapshot at a time instead of creating a separate volume snapshot for each processed VM. For reliability reasons caused by Hyper-V backup architecture, the maximum amount of VMs is limited to four per snapshot by default in case of software VSS, and eight per snapshot in case of hardware VSS, but can be increased on fast primary storage with the MaxVmCountOnHvSoftSnapshot (DWORD) and MaxVmCountOnHvHardSnapshot (DWORD) registry values under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key.
rmeichsner
Novice
Posts: 4
Liked: never
Joined: Dec 02, 2014 2:10 pm
Full Name: Ruben Meichsner
Contact:

Re: VeeamB&R and CSV

Post by rmeichsner »

Thanks for your fast answer, but unfortunately it's not the solution.
The Settings you mentioned are already set, but doesn't seem to have any effect at all.

MaxVmCountOnHvHardSnapshot and MaxVmCountOnHvSoftSnapshot are both set to 8.

That's the reason, why I'm interested in the requirements which have to be met to have this option taking any effect.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VeeamB&R and CSV

Post by foggy »

The main requirement is that only VMs residing on the same host can be processed from a single snapshot.
baltechnique
Novice
Posts: 8
Liked: 1 time
Joined: Sep 30, 2013 7:11 am
Full Name: Network Consulting
Contact:

Re: VeeamB&R and CSV

Post by baltechnique »

foggy wrote:The main requirement is that only VMs residing on the same host can be processed from a single snapshot.
Hi,

Is it needed by the Backup Job to have these VMs declared together ?

For instance if VM1 and VM3 are stored on the same CSV must i modify it like this :
VM List : VM1, VM3, VM2, VM4
or VM1, VM2, VM3, VM4 does also work ?

and you're also saying that i must migrate all VMs sharing same CSV (at least 4 by 4 or 8 by 8 if using hardware VSS) to same HyperV Host before backup ?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VeeamB&R and CSV

Post by veremin »

For instance if VM1 and VM3 are stored on the same CSV must i modify it like this :
VM List : VM1, VM3, VM2, VM4
or VM1, VM2, VM3, VM4 does also work ?
It should work either way.
and you're also saying that i must migrate all VMs sharing same CSV (at least 4 by 4 or 8 by 8 if using hardware VSS) to same HyperV Host before backup ?
If you want those VMs to be processed from a single volume snapshot, then, yes.

Thanks.
jmeisel
Influencer
Posts: 11
Liked: 2 times
Joined: Mar 05, 2015 1:06 pm
Contact:

[MERGED] Allow processing of multiple VMs with Single-Volume

Post by jmeisel »

We are running Veeam Backup & Replication 8 Patch 1 with an FC-attached off-host Proxy. Everything running on Windows Server 2012 R2.
Our Storage, where the cluster-shared-volume for the HyperV-VMs is on, is a NetApp.

In our testing-environment we are having a strange issue with the "Allow processing of multiple VMs with Single-Volume-Snapshot" Option. It seems as if this Option is useless. It doesn't matter if the Checkbox is checked or unchecked. The Backup-Job that should backup several VMs that are running on different HyperV-Hosts where all the VM data is stored on the same CSV (=ONE Volume) is always trying to create a Hardware VSS-Snapshot for each VM. And... because the disk / volume signature would be the same, the snapshot creation is failing for each VM, except for one.

Can anybody confirm this???

We have NetApp Snapdrive installed on every server. We are utilizing the Hardware VSS-Provider from NetApp. We adjusted the Volume-Shadow-Reserve on every HyperV-Host to 10GB (for the CSV).
At the moment we are wondering what's going on here...

Any ideas?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] Allow processing of multiple VMs with Single-Vo

Post by foggy »

jmeisel wrote:The Backup-Job that should backup several VMs that are running on different HyperV-Hosts...
That is the reason, for VMs to be processed from a single snapshot, they should reside on the same host.
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

[MERGED] : Optimal job design

Post by frankive »

For a small hosting company, running 2 node hyper-v failover cluster with different customeres and multiple CSV (Sans);
what is the most efficient way to backup regarding speed?
A job where each customers VM is no matter what which CSV they belong too?
Or a larger job with all the VMS where any customer is on the same CSV?

I believe that one job (with max 20 VM) backing up VMS which belongs on the same CSV) is optimal?
No VMS are having separate vm-disks on the CSVs.

and if we enable the job to use one shapshot for multiple vms, will 20 VM use that one shapshot? That seems to be very efficient?
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: Optimal job design

Post by frankive »

Seems like someone has the same question as me here: http://forums.veeam.com/microsoft-hyper ... 22167.html
So I guess my post can be deleted/merged.

I use hardware snapshot lefthand, so it seems that 8 VMS pr. job which runs on the same hyper-v host is the optimal design?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VeeamB&R and CSV

Post by veremin »

Correct.
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: VeeamB&R and CSV

Post by frankive » 3 people like this post

Pretty cool. Reduced the backup time from 11 hours to 2 1/2 hour byt using 7 jobs with 8 VMS in each (VM on same host and same CSV) and parallel processing enabled.
samh22a
Enthusiast
Posts: 35
Liked: 4 times
Joined: Nov 10, 2013 2:03 pm
Contact:

Re: VeeamB&R and CSV

Post by samh22a »

are you saying veeam can figure out where vm are located and then optimize the processing ?

let's say I backup 50 vm, located on 3 hosts with 5 different csv.. would veeam figure out the grouping and optimize the group in group of 8 vm located on the same csv/host ?

Thx
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: VeeamB&R and CSV

Post by frankive »

No. What I did was to create a job named "01 - NAME OF HYPER-V HOST - NAME OF CSV".
Then I included max 8 VM in this job and live migrated the VM to that speicif chost and made sure that the VM ALSO was located on that specified CSV.
So the hyper-v AND csv location dictaates which VM you put in the job.
I then creaate a backup copy job where I can choose longer retentions which I name: GFS (grand, father, son) - COMPANYNAME and choose the VM from the infrastructure.
Then Veeam will scna ALL jobs I have to find the backup data for that VM and I get to collect them logical in the backup copy job and it's naming.
samh22a
Enthusiast
Posts: 35
Liked: 4 times
Joined: Nov 10, 2013 2:03 pm
Contact:

Re: VeeamB&R and CSV

Post by samh22a »

my point,is that I can't event think of doing this..

let's say, 12 nodes in the cluster , 20+ CSV

I do not want to start to live migration vm before a backup.. is there any other way ?
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: VeeamB&R and CSV

Post by frankive »

Live migration of VM should not be a concern related to backup, it doesn't touch the CSV at all as far as I know, only the memory (RAM) during the migration.
If you are talking about store migrateion (csv to csv), I have never had any issued with this on my 2012 clusters. But I agree, I prefer to do storage migration AFTER a fresh backup, just in case.

I don't think there is another way to optimize Veeam backup to just use one snapshot if you are Hyper-V.

At least that is my conclusion based on the response from v. Eremin Fr 24.april 2015 3:00 p.m.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: VeeamB&R and CSV

Post by jazzoberoi »

Hi Guys,

I'm having issues with backing up multiple VM's on the same Host & CSV using the 1 snapshot option.

Host 1 & CSV 1
VM: DC1001
VM: ERP1001

Host 2 & CSV 2
VM: FS1001
VM: PS1001
VM: SEC1001

EDIT: I've opened a support case as well and the case number is 00910583

Image

What can be the issue here ?
Lix
Lurker
Posts: 1
Liked: never
Joined: Jan 23, 2013 11:26 am
Full Name: Alexander Matre
Contact:

Re: VeeamB&R and CSV

Post by Lix »

Hi

Might not apply/be useful to all, but you can run this Powershell scrip to live migrate virtual machines to the host that currently owns a csv. As always, use at your own risk.

Works great on my 2-node 2012 R2 cluster containing 40x vm and 4x csv. Must be run on one of the cluster hosts.

Code: Select all

cls

$Cluster = Get-Cluster
$AllCSV = Get-ClusterSharedVolume -Cluster $Cluster

ForEach ($CSV in $AllCSV)

{
    write-output "$($CSV.Name) is owned by $($CSV.OWnernode.Name)"
    
    #We grab the friendly name of the CSV
    $CSVVolumeInfo = $CSV | Select -Expand SharedVolumeInfo
    $CSVPath = ($CSVVolumeInfo).FriendlyVolumeName

    #We deal with the \ being and escape character for string parsing.
    $FixedCSVPath = $CSVPath -replace '\\', '\\'

    #We grab all VMs that who's owner node is different from the CSV we're working with
    #From those we grab the ones that are located on the CSV we're working with
      $VMsToMove = Get-ClusterGroup | ? {($_.GroupType –eq 'VirtualMachine') -and ( $_.OwnerNode -ne $CSV.OWnernode.Name)} | Get-VM | Where-object {($_.path -match $FixedCSVPath)}
       
     
    ForEach ($VM in $VMsToMove)

    {
        write-output "`tThe VM $($VM.Name) located on $CSVPath is not running on host $($CSV.OwnerNode.Name) who owns that CSV"
        write-output "`tbut on $($VM.Computername). It will be live migrated."
        #Live migrate that VM off to the Node that owns the CSV it resides on
        Move-ClusterVirtualMachineRole -Name $VM.Name -MigrationType Live -Node $CSV.OWnernode.Name
    }
    }
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VeeamB&R and CSV

Post by foggy »

Jazz, do you have the corresponding setting enabled? Is this a new job created in v8?
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: VeeamB&R and CSV

Post by jazzoberoi »

Hi Foggy,

Not sure what happened there but in desperation to fix the issue i restarted both the Hosts in the Hyper-V Cluster.. and what do you know, it started working again :)

It seems to be making 1 snapshot on the SAN for all the VM's located on the same CSV as intended.. however, it's only making the snapshots 1 CSV at a time.

How can i make it create 1x snapshots for both the CSV's at the same time ?
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: VeeamB&R and CSV

Post by jazzoberoi »

Hi Foggy, Any clues ?

::
It seems to be making 1 snapshot on the SAN for all the VM's located on the same CSV as intended.. however, it's only making the snapshots 1 CSV at a time.

How can i make it create 1x snapshots for both the CSV's at the same time ?
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: VeeamB&R and CSV

Post by RGijsen »

jazzoberoi wrote:Hi Guys,

I'm having issues with backing up multiple VM's on the same Host & CSV using the 1 snapshot option.

Host 1 & CSV 1
VM: DC1001
VM: ERP1001

Host 2 & CSV 2
VM: FS1001
VM: PS1001
VM: SEC1001

EDIT: I've opened a support case as well and the case number is 00910583

What can be the issue here ?

Sorry for resurrecting this, but I'm having the exact same issue with out MSA2040 units. Exact same message about 95% of the times, then falling back to software CSV (as I noticed in the past it wouldn't work on the retry either and the backup would fail). What VSS / CAPI version do you have installed on your hosts? I'm running 3.2.0 build 16 of both VSS and CAPI drivers. The MSA2040 is running GL210R004 firmware, but I've had this issue with all older firmwares it ran. Also, when it DOES work, creating the hardware snapshot takes 2-3 minutes. Is that normal? The storage is done creating the snapshot in a few seconds, it seems Veeam is waiting for 3 minutes to 'see' it?
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 24 guests