Confusion on Max Snapshots setting with Hyper-V 2016

[nmdange]

I've noticed that even after my environment is now upgraded to Windows Server 2016 (with all VMs on VM Config Version 8.0), I still see VMs waiting with status "Resource not ready: Snapshot".

In Windows Server 2012 R2, this was due to the Max Snapshots setting on the CSV/SMB share, to limit the number of concurrent VSS snapshots on the volume hosting the virtual machines. On Windows Server 2016, the new RCT backup method no longer takes a volume snapshot of the host volume. The documentation says the following in regards to what this setting does on 2016+

https://helpcenter.veeam.com/docs/backu ... tml?ver=95
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

For VMs hosted on Microsoft Hyper-V Server 2016 and later, the Max snapshots value defines the maximum number of checkpoints that may be retained for a processed VM. The setting is specified at the level of a volume on which VM disks reside.

I don't really understand what this statement is supposed to mean. It sounds like it would prevent multiple checkpoints from being created simultaneously for a specific VM. When would there ever be more than 1 checkpoint for a VM? The backup checkpoint is automatically deleted so there would never be more than 1 unless multiple backup jobs are backing up the same VM at the same time.

Can the documentation be updated to clarify exactly what the behavior of this setting on 2016 is? Right now it seems like the setting imposes the same limitation on a 2016 Hyper-V host as it does on 2012 R2, even though there is no longer any architectural limitations to the number of VMs being backed up at the same time.

[wishr]

Hi Nmdange,

This setting controls the number of simultaneous (running in parallel) backups for VMs which disks reside on a single datastore volume (amount of VM snapshots/VM checkpoints depending on the Hyper-V version). In fact, this is a datastore-specific setting, not a VM-specific, and you are right assuming that any limit set in the UI can be reached when you have several backup jobs backing up VMs which disks reside on the same volume or in case of parallel VMs processing in a single job.

A few examples to be clear:
1. The setting is set to 4. You have 5 VMs residing on the same volume, you have 5 jobs running simultaneously and backing up these VMs. The limit will be exceeded.
2. The limit is set to 4. You have 5 VMs residing on the same volume, you have a single job backing up these VMs (processing usually goes in parallel). The limit will be exceeded.
3. The limit is set to 4. You have 5 Vms residing on two different volumes (one on the volume A and four on the volume B), you have a single or several jobs backing up these VMs. The limit will not be exceeded but will be reached.

Veeam B&R technology utilizes only the "new" Production checkpoints when backing up Hyper-V 2016 VMs - this feature was introduced first with Hyper-V 2016 release and is based on VSS capabilities in Windows and File System Freeze capabilities in Linux. For previous Hyper-V versions, we utilize the common "snapshots" functionality - a snapshot of a VM plus a snapshot of its memory state.

RCT technology is an approach to getting changes between different VM states. It is not related directly to the aforementioned "Max shapshots" setting, but incremental backups can be affected by it.

By the way, the root cause for getting the "Resource not ready: Snapshot" error could be different, though. Have you managed to resolve by increasing the aforementioned setting?

Please let me know if you have any additional questions.

BR,
Fedor

[PTide]

2 cents:

On Windows Server 2016, the new RCT backup method no longer takes a volume snapshot of the host volume.

As Fedor's mentioned, RCT is not directly related to backup method. But, if you use off-host backup mode, host volume snapshots are still used.

Thanks!

[nmdange]

I don't believe host volume snapshots are used with off-host backup mode when using SMB/SOFS (which is what I'm using). The off-host proxy can just access the share directly, it doesn't need to mount a snapshot like with a CSV (which requires a hardware VSS provider with transportable snapshots).

[wishr]

Hi Nmdamage,

You are correct, in the case with Hyper-V 2016, off-host backup mode, SMB3 share registered in Veeam B&R UI as a Managed Server, the data is being retrieved directly from the share. The "Snapshots" setting (checkpoints in Hyper-V 2016), in that case, controls the maximum number of held checkpoints of VM's which disks reside on the same volume that can be processed in one job.

Example:
Volume "A", Max Snapshots is set to 4, there are 5 VMs. A job for all 5 VMs is created. In case all the backup infrastructure resources are unlimited, 4 VMs will be processed in parallel, while the 5th will be waiting for a free slot.

I'll check with our internal teams for a way to improve these descriptions.

Regards,
Fedor

[nmdange]

Thanks, I think just re-wording the section on 2016 to make it clear it's not a per-VM limitation, and counts all VMs backed up concurrently towards the limit.

I've raised the value, but I have to watch the backups at night to see for sure if it's improved concurrency, since the "Resource not ready: Snapshot" message is only visible while the job is executing. I am trying to think of what reason I would have to not raise it to something really large like 99. There's no real limitations in Hyper-V in this case with the number of VMs that can be backed up at once, so increasing it significantly would make the job limited by other backup resource limitations on the proxy or repository.

[wishr]

Hi Nmdange

Correct, a significant increase of the limit may cause proxy or repository to cause bottlenecks so we recommend to check if gradual increases would help. Please let us know how it goes with the limit being increased slightly.

Thanks,
Fedor

[PTide]

I don't believe host volume snapshots are used with off-host backup mode when using SMB/SOFS (which is what I'm using). The off-host proxy can just access the share directly, it doesn't need to mount a snapshot like with a CSV (which requires a hardware VSS provider with transportable snapshots).

Although in such scenario the off-host proxy accesses it directly, the folder is still backed by CSV.

Before a backup starts, File Share Shadow Copy Provider sends a request for shadow copy to SOFS. After that, File Share Shadow Copy Agent will tell the Provider which SOFS node will perfrom Shadow Copy (the node that holds CSV coordinator role), so that the Provider will connect to that node. File Share Shadow Copy Agent on the node triggers VSS snapshot for the volume where the fileshare resides. When done, a Shadow Copy share \\<nodename>\<sharename>@{GUID} will be available for the backup proxy.

Thanks!

[nmdange]

As far as I can tell watching the SOFS nodes during a backup, I don't see this process happen anymore with Hyper-V 2016.

[PTide]

In order to see shadow copies you need to run this line on all SOFS nodes during backup:

Code: Select all

vsstrace -l 255 +f 0xffff > vsstrace.txt

Thanks!