VeeamB&R and CSV

[baltechnique]

Hi,

why is Veeam doing a volume snapshot for each VM, even if these VMs are on a Cluster Shared Volume.

An only snapshot for each CSV would be far enough. No ? And backup would finish faster.

PS : tested on 2008R2 HyperV Cluster, but not on 2012R2

[Gostev]

Hi

Reliability reasons. This approach does not currently work reliably (issues on Hyper-V VSS side). Too many intermittent VSS failures.

We actually have this logic implemented, and may end up enabling it in v8, but it will still be heavily limited to no more than a few VMs per CSV being processed at once. Folks with really fast SAN will be able to tweak the maximum amount, but for average hardware, the limit will need to be pretty low (think 3-4 VMs at once).

Thanks!

[baltechnique]

Hi,

OK for your explanation, but in facts i dont understand why

Since there is a SAN snapshot of the volume, each VM file is available separately (i'm not talking about vss inside the running VMs for application consistency), and since the snapshot is finished on the SAN, the redirected mode is back to normal mode for hyperV.

But i do agree that even with hardware VSS there are some snapshot failure, so in that case you failed to backup correctly each VM in this volume.

[rskinner]

Hi

Reliability reasons. This approach does not currently work reliably (issues on Hyper-V VSS side). Too many intermittent VSS failures.

We actually have this logic implemented, and may end up enabling it in v8, but it will still be heavily limited to no more than a few VMs per CSV being processed at once. Folks with really fast SAN will be able to tweak the maximum amount, but for average hardware, the limit will need to be pretty low (think 3-4 VMs at once).

Thanks!

This would be a great feature. Many of our VMs have very few data changes and the time to create the CSV VSS snapshot (2-4 mins) is one of the longest stages in the backup process for each VM. The ability to have 3-4 VMs backed-up per CSV VSS snapshot would greatly speed up our backups.

[Gostev]

Since there is a SAN snapshot of the volume, each VM file is available separately (i'm not talking about vss inside the running VMs for application consistency), and since the snapshot is finished on the SAN, the redirected mode is back to normal mode for hyperV.

Yes, but unfortunately it does not work like that today

Hyper-V backup is an incredibly integrated monster that touches too many things. Part of the process actually goes into each guest, creates VSS snapshots inside of the running VMs, and then after SAN snapshot is created, it goes back there and performs the process known as AutoRecovery. So as soon as you have more than a few VMs in the consistency group, this process starts to fail or timeout due to much extra load on the datastore. Hyper-V team at Microsoft is well aware of this problem, in fact I've just discussed this issue again with them last month at TechEd. But the whole talk was under uber NDA, so I cannot really comment beyond known facts.

[samh22a]

Hi,

Let's assume the following scenario

- Hardware VSS Snapshot
- Hyper-V cluster
- CSV with 10 Virtual Machines

Is there any way to take a single snapshot, and backup the 10 VMs at once ? right now I can see multiple snapshots beeing created as it process all of those vm..and it doesn't seems to be very efficient

Thanks

[rmeichsner]

Hi Anton,

can you share any details under which circumstances this option should work in v8? The explanation in your kb is really general and doesn't contain much information regarding this option.
In our environment the setting doesn't seem to have any effect at all. (smb3 sofs)

What is the maximum number of Snapshots you recommend per smb3-Volume? If set to everything different than "1", we're facing VSS Errors from time to time...
Registrykey "HyperVRestrictConcurrentSnapshotCreation" already set to "1".

We're looking for a way to reduce the time, the backup procedure needs and get rid of the gaps in the backup process without any throughput.

Thanks,
Ruben

[foggy]

Ruben, there is a setting in the job advanced settings that allows processing of multiple VMs from a single snapshot. The setting is enabled by default on the new jobs, however you should enable it manually for existing jobs.

Also, here's the quote from the What's New document regarding this feature:

Consistency groups: Jobs can now process more than one VM from the volume snapshot at a time instead of creating a separate volume snapshot for each processed VM. For reliability reasons caused by Hyper-V backup architecture, the maximum amount of VMs is limited to four per snapshot by default in case of software VSS, and eight per snapshot in case of hardware VSS, but can be increased on fast primary storage with the MaxVmCountOnHvSoftSnapshot (DWORD) and MaxVmCountOnHvHardSnapshot (DWORD) registry values under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key.

[rmeichsner]

Thanks for your fast answer, but unfortunately it's not the solution.
The Settings you mentioned are already set, but doesn't seem to have any effect at all.

MaxVmCountOnHvHardSnapshot and MaxVmCountOnHvSoftSnapshot are both set to 8.

That's the reason, why I'm interested in the requirements which have to be met to have this option taking any effect.

[foggy]

The main requirement is that only VMs residing on the same host can be processed from a single snapshot.

[baltechnique]

The main requirement is that only VMs residing on the same host can be processed from a single snapshot.

Hi,

Is it needed by the Backup Job to have these VMs declared together ?

For instance if VM1 and VM3 are stored on the same CSV must i modify it like this :
VM List : VM1, VM3, VM2, VM4
or VM1, VM2, VM3, VM4 does also work ?

and you're also saying that i must migrate all VMs sharing same CSV (at least 4 by 4 or 8 by 8 if using hardware VSS) to same HyperV Host before backup ?

[veremin]

For instance if VM1 and VM3 are stored on the same CSV must i modify it like this :
VM List : VM1, VM3, VM2, VM4
or VM1, VM2, VM3, VM4 does also work ?

It should work either way.

and you're also saying that i must migrate all VMs sharing same CSV (at least 4 by 4 or 8 by 8 if using hardware VSS) to same HyperV Host before backup ?

If you want those VMs to be processed from a single volume snapshot, then, yes.

Thanks.

[jmeisel]

We are running Veeam Backup & Replication 8 Patch 1 with an FC-attached off-host Proxy. Everything running on Windows Server 2012 R2.
Our Storage, where the cluster-shared-volume for the HyperV-VMs is on, is a NetApp.

In our testing-environment we are having a strange issue with the "Allow processing of multiple VMs with Single-Volume-Snapshot" Option. It seems as if this Option is useless. It doesn't matter if the Checkbox is checked or unchecked. The Backup-Job that should backup several VMs that are running on different HyperV-Hosts where all the VM data is stored on the same CSV (=ONE Volume) is always trying to create a Hardware VSS-Snapshot for each VM. And... because the disk / volume signature would be the same, the snapshot creation is failing for each VM, except for one.

Can anybody confirm this???

We have NetApp Snapdrive installed on every server. We are utilizing the Hardware VSS-Provider from NetApp. We adjusted the Volume-Shadow-Reserve on every HyperV-Host to 10GB (for the CSV).
At the moment we are wondering what's going on here...

Any ideas?

[foggy]

The Backup-Job that should backup several VMs that are running on different HyperV-Hosts...

That is the reason, for VMs to be processed from a single snapshot, they should reside on the same host.

[frankive]

For a small hosting company, running 2 node hyper-v failover cluster with different customeres and multiple CSV (Sans);
what is the most efficient way to backup regarding speed?
A job where each customers VM is no matter what which CSV they belong too?
Or a larger job with all the VMS where any customer is on the same CSV?

I believe that one job (with max 20 VM) backing up VMS which belongs on the same CSV) is optimal?
No VMS are having separate vm-disks on the CSVs.

and if we enable the job to use one shapshot for multiple vms, will 20 VM use that one shapshot? That seems to be very efficient?

[frankive]

Seems like someone has the same question as me here: http://forums.veeam.com/microsoft-hyper ... 22167.html
So I guess my post can be deleted/merged.

I use hardware snapshot lefthand, so it seems that 8 VMS pr. job which runs on the same hyper-v host is the optimal design?

[veremin]

Correct.

[frankive]

Pretty cool. Reduced the backup time from 11 hours to 2 1/2 hour byt using 7 jobs with 8 VMS in each (VM on same host and same CSV) and parallel processing enabled.

[samh22a]

are you saying veeam can figure out where vm are located and then optimize the processing ?

let's say I backup 50 vm, located on 3 hosts with 5 different csv.. would veeam figure out the grouping and optimize the group in group of 8 vm located on the same csv/host ?

Thx

[frankive]

No. What I did was to create a job named "01 - NAME OF HYPER-V HOST - NAME OF CSV".
Then I included max 8 VM in this job and live migrated the VM to that speicif chost and made sure that the VM ALSO was located on that specified CSV.
So the hyper-v AND csv location dictaates which VM you put in the job.
I then creaate a backup copy job where I can choose longer retentions which I name: GFS (grand, father, son) - COMPANYNAME and choose the VM from the infrastructure.
Then Veeam will scna ALL jobs I have to find the backup data for that VM and I get to collect them logical in the backup copy job and it's naming.

[samh22a]

my point,is that I can't event think of doing this..

let's say, 12 nodes in the cluster , 20+ CSV

I do not want to start to live migration vm before a backup.. is there any other way ?

[frankive]

Live migration of VM should not be a concern related to backup, it doesn't touch the CSV at all as far as I know, only the memory (RAM) during the migration.
If you are talking about store migrateion (csv to csv), I have never had any issued with this on my 2012 clusters. But I agree, I prefer to do storage migration AFTER a fresh backup, just in case.

I don't think there is another way to optimize Veeam backup to just use one snapshot if you are Hyper-V.

At least that is my conclusion based on the response from v. Eremin Fr 24.april 2015 3:00 p.m.

[jazzoberoi]

Hi Guys,

I'm having issues with backing up multiple VM's on the same Host & CSV using the 1 snapshot option.

Host 1 & CSV 1
VM: DC1001
VM: ERP1001

Host 2 & CSV 2
VM: FS1001
VM: PS1001
VM: SEC1001

EDIT: I've opened a support case as well and the case number is 00910583



What can be the issue here ?

[Lix]

Hi

Might not apply/be useful to all, but you can run this Powershell scrip to live migrate virtual machines to the host that currently owns a csv. As always, use at your own risk.

Works great on my 2-node 2012 R2 cluster containing 40x vm and 4x csv. Must be run on one of the cluster hosts.

Code: Select all

cls

$Cluster = Get-Cluster
$AllCSV = Get-ClusterSharedVolume -Cluster $Cluster

ForEach ($CSV in $AllCSV)

{
    write-output "$($CSV.Name) is owned by $($CSV.OWnernode.Name)"
    
    #We grab the friendly name of the CSV
    $CSVVolumeInfo = $CSV | Select -Expand SharedVolumeInfo
    $CSVPath = ($CSVVolumeInfo).FriendlyVolumeName

    #We deal with the \ being and escape character for string parsing.
    $FixedCSVPath = $CSVPath -replace '\\', '\\'

    #We grab all VMs that who's owner node is different from the CSV we're working with
    #From those we grab the ones that are located on the CSV we're working with
      $VMsToMove = Get-ClusterGroup | ? {($_.GroupType –eq 'VirtualMachine') -and ( $_.OwnerNode -ne $CSV.OWnernode.Name)} | Get-VM | Where-object {($_.path -match $FixedCSVPath)}
       
     
    ForEach ($VM in $VMsToMove)

    {
        write-output "`tThe VM $($VM.Name) located on $CSVPath is not running on host $($CSV.OwnerNode.Name) who owns that CSV"
        write-output "`tbut on $($VM.Computername). It will be live migrated."
        #Live migrate that VM off to the Node that owns the CSV it resides on
        Move-ClusterVirtualMachineRole -Name $VM.Name -MigrationType Live -Node $CSV.OWnernode.Name
    }
    }

[foggy]

Jazz, do you have the corresponding setting enabled? Is this a new job created in v8?

[jazzoberoi]

Hi Foggy,

Not sure what happened there but in desperation to fix the issue i restarted both the Hosts in the Hyper-V Cluster.. and what do you know, it started working again

It seems to be making 1 snapshot on the SAN for all the VM's located on the same CSV as intended.. however, it's only making the snapshots 1 CSV at a time.

How can i make it create 1x snapshots for both the CSV's at the same time ?

[jazzoberoi]

Hi Foggy, Any clues ?

::
It seems to be making 1 snapshot on the SAN for all the VM's located on the same CSV as intended.. however, it's only making the snapshots 1 CSV at a time.

How can i make it create 1x snapshots for both the CSV's at the same time ?

[RGijsen]

Hi Guys,

I'm having issues with backing up multiple VM's on the same Host & CSV using the 1 snapshot option.

Host 1 & CSV 1
VM: DC1001
VM: ERP1001

Host 2 & CSV 2
VM: FS1001
VM: PS1001
VM: SEC1001

EDIT: I've opened a support case as well and the case number is 00910583

What can be the issue here ?

Sorry for resurrecting this, but I'm having the exact same issue with out MSA2040 units. Exact same message about 95% of the times, then falling back to software CSV (as I noticed in the past it wouldn't work on the retry either and the backup would fail). What VSS / CAPI version do you have installed on your hosts? I'm running 3.2.0 build 16 of both VSS and CAPI drivers. The MSA2040 is running GL210R004 firmware, but I've had this issue with all older firmwares it ran. Also, when it DOES work, creating the hardware snapshot takes 2-3 minutes. Is that normal? The storage is done creating the snapshot in a few seconds, it seems Veeam is waiting for 3 minutes to 'see' it?