How to scale up when using SMB - snapshot limit

[collinp]

Let's say you have thousands of Hyper-v VM's hosted on 20TB SMB Volumes on vendor storage (NetApp, EMC, etc). You have four 5TB VM's that are doing full backups residing on one of those SMB volumes. You have volume snapshots enabled and the option to parallel process backups. In this scenario, all of your other backups are queued up waiting for those 4 large VM's to finish backing up because there is a limit to 4 snapshots per volume, correct? How do you scale in this scenario? I don't want to create hundreds of small volumes and plenty of compute on the Hyper-v hosts. I have SSD storage so performance is not a concern. Is there a registry setting or some other way that I can increase to more than 4 snapshots per volume?

Case # 03033687

[foggy]

Hi Colin, Veeam B&R is able to backup multiple VMs from the same volume snapshot (and the number is configurable).

[nmdange]

I know this isn't a great answer, but upgrading Hyper-V to 2016 will eliminate this issue. The new RCT feature in 2016 eliminates the need to do snapshots of the CSV volumes.

[collinp]

I think you guys may have misunderstood my question. I am using Hyper-v 2016 and am not using CSV's, I am using SMB shares and NetApp Storage. I want to increase the limit of 4 snapshots per SMB share. I know about the other features you speak of but I am more interested in the max snapshot feature per SMB share. How do you increase it?

[nmdange]

RCT does not require volume snapshots, regardless of if it's a local CSV or a remote SMB share. I said CSV because normally even when doing SMB, it's done with a Scale-Out File Server, which has its own CSV. The only way to know if this is the case is to look at VMs that are waiting to be backed up, while the backup job is running. Does it say "Resource not ready: Snapshot"? If so, then it sounds like Veeam is enforcing the old 2012 R2 limitations which are not applicable to 2016.

You can configure Max Concurrent Snaphots per below documentation but this is only possible with a Window-based SOFS. Since you're using the NetApp directly, you can't add it to Veeam as far as I know.
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

[collinp]

It should be a feature in the console to increase this value. I can see the limit hard coded in the database.

[Mike Resseler]

Collin,

NMdange is correct. You should not need them... What configuration versions are your VMs? (Get-VM | select name, state, version)

[collinp]

The VM's are at Version 8. The Hyper-V hosts are 2016. The VM's are on SMB. No CSV's.

We are being throttled as our VM's are sitting idle with "Waiting for Snapshot". The below excerpt is just to show that the backup is doing some type of snapshot for each vm.

I am 100% confident that we are using the Windows 2016 built-in CBT driver because when we did a major upgrade of Veeam versions, we did not lose CBT, as we did in the past when Veeam managed the CBT.


Queued for processing at 9/5/2018 2:24:22 AM
Required backup infrastructure resources have been assigned
VM processing started at 9/5/2018 2:24:33 AM
VM size: 3.0 TB (1.2 TB used)
Preparing to create snapshot
Creating VM recovery checkpoint (mode: Crash consistent) 00:22
Storage initialized 00:34
Network traffic will be encrypted
Using source proxy HypervHost1 (onhost)
Saving config.wmi 00:01
VMCX configuration file (379.9 KB) 379.9 KB read at 380 KB/s00:01
VMRS configuration file (120.2 KB) 120.2 KB read at 120 KB/s00:01
Hard disk 1 (250.0 GB) 169.5 GB read at 17 MB/s [CBT]02:48:05
Hard disk 2 (750.0 GB) 465.0 GB read at 20 MB/s [CBT]06:44:41
Hard disk 3 (2.0 TB) 462.3 GB read at 20 MB/s [CBT]06:44:14
Hard disk 4 (80.0 GB) 33.6 GB read at 15 MB/s [CBT]40:35

[collinp]

One other point to make. I am not talking about volume snapshots. I don't believe those are being done because we are using Windows 2016 and rct. I think there is a limit on the number of VM's that can snapshot at once. We are getting a lot of VM's with the message: Resource not ready: Snapshot

Like a lot of people, we are not using SOFS. Is there a way to increase maximum number of concurrent snapshots with vendor storage and smb?

I definitely see the maxconcurrent snapshot setting in the database under one of the tables. When I adjusted it, I was able to increase the maximum number of snapshots. But why isn't this available in the GUI? I changed the database entry back because it is not standard.

[Mike Resseler]

Collin,

First: I see you are using crash consistent snapshot and not a production checkpoint? Any reason for that? It also looks to me that you are not using the new RCT mechanism but are falling back to the old system. The versions of VMs is good, but what is your cluster-level version?
PS: Do you see a .RCT and .MRT file per VM disk on your filesystem? That is the easiest way to see if you are using the new RCT functionality or the old CBT filter driver/
Second: Hyper-V does not use vendor snapshots like it does with VMware. The system itself is different so that number is probably not even used.

At this point in time, I would advise you to create a support call to run over your config because I think something in your settings is not correct.

[collinp]

Yes, I see both .RCT and .MRT files per VM disk. We are at a Cluster functional level of 9

We use crash consistent snapshots because this is a large hosted backup service and we don't necessarily keep track of all of the credentials required to perform application aware backups.

[collinp]

I'm not using vendor snapshots. I'm using Hyper-V snapshots.

The value of 4 in the max_snapshots_count field in the dbo.SmbFileShares table was limiting the maximum number of Hyper-V snapshots per SMB file share.

It is no longer, so consider this closed.

[Mike Resseler]

OK. Thanks for letting us know. Just one last question, was it changed through support?

[collinp]

No. Support gave me the answer that Veeam is limited on the number of snapshots allowed. See answer from support below for case# 03033687.

Having a large Hyper-V environment, a lot of times I feel like we are on our own in some of the scaling issues we experience. I have multiple examples of this. The first was with Hyper-V CSV's and CBT reset's happening every time we upgraded. Sure, other customers may not have had this issue, but other customers probably didn't have a thousand VM's.

Here is supports answer:

When using a different device, we can allow multiple VMs to be processed within a single volume snapshot by enabling the option within the job's advanced Hyper-V settings, as described here:
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Since we are not using the CSVs anymore, the options to manage the maximum amount of snapshots are limited through Veeam, and the above mentioned option may be the best way to expand the amount of VMs allowed within a single snapshot, rather than increasing the amount of snapshots we take.

[nmdange]

I'll be honest here this looks like a bug or logic error with Veeam and Hyper-V 2016.

I took a look while a job was running and I'm seeing the same behavior in my environment. It just isn't affecting my backup windows so I haven't really noticed. I see 4 VMs on a given SMB share backing up, and the other VMs all say "Resource not ready: Snapshot".

The documentation from https://helpcenter.veeam.com/docs/backu ... tml?ver=95 says

For VMs hosted on Microsoft Hyper-V Server 2016 and later, the Snapshots value defines the maximum number of checkpoints that may be retained for a processed VM. The setting is specified at the level of a shared folder in which VM disks reside.

But that is not the observed behavior. Seems to me like Veeam is still enforcing old 2012 R2 limitations.

[collinp]

Your link is referencing the scenario when using a Microsoft SMB share. Where you do have the opportunity to increase the maximum number of concurrent snapshots above 4 in the Veeam GUI (whether this works or not is a different story).

My scenario is a little different. I am using a NetApp SMB Share. In my scenario, you can't add the NetApp Storage Array to Managed Servers under Inventory in the Veeam GUI since the NetApp is a hardened storage appliance where you obviously can't add an agent or service to. So there is no place in the Veeam GUI to manipulate the SMB Share maximum concurrent Snapshot limit for the NetApp. In fact, there is no place in the Veeam GUI to even see my NetApp SMB Shares. The SMB Share paths and max concurrent snapshot value hasn't been exposed to the Veeam GUI.

It would be cool if there was a screen allowing you to see your SMB shares and adjust the maximum number of snapshots when using non-Microsoft SMB Shares. The Veeam software is obviously aware of the share paths, since there is a database table dedicated to this. Why not expose this to the GUI so we can scale the snapshot value.

[nmdange]

Right I get that, what I'm saying is the fact that Veeam is even enforcing a limitation in the first place is the problem. Even if I can change the setting since I am using a Windows-based SMB share, that should not be required because 2016 doesn't even use volume snapshots in the first place!

[collinp]

Ok. I was interpreting this a little differently. I didn't think the maximum concurrent snapshot setting was describing Volume Snapshots. I thought it meant that only 4 vm snapshots could be performed per SMB share. So it was trying to protect you so you don't degrade performance on that share too much.