Active snapshots limit reached for datastore

[chjones]

Hi,

We are running Veeam 7.0 R2 and our backup jobs have been taking a lot longer recently to complete and I've started to notice the following message in the jobs for VMs that are pending:

Resouce not ready: Active snapshosts limit reached for datastore

We have 6 hosts in a vSphere 5.1 Cluster and use a HP 3PAR as our storage array. When it came to sizing the datastore I discussed our setup with HP and was advised there was no real difference between creating multiple smaller datastores or a single large datastore. The 3PAR can create 16TB Virtual Volumes so we created a single 16TB Datastore with Thin Detect enabled and use Thick Eager Zeroed for all of our VMs.

The maximum number of powered on VMs per datastore in vSphere 5.1 is 2048. The cluster I am seeing the above message on has about 90 VMs. I currently have 4 Backup Jobs that run each night for these VMs, the jobs are split out to backup certain folders as we group our VMs based on the business unit that uses them.

I cannot find any reference to a limit on the maximum number of snapshots per datastore.

We are yet to enable Storage Snapshots for the 3PAR as we've only just got that working with a patch from Veeam Tech Suppport. Im thinking if I was to enable that the situation would be worse as Veeam snapshots every VM in the job before taking the storage snapshot which would mean a lot more snapshots would exist at the same time.

Has anyone see anything about the active snapshots limit?

Thanks,

Chris

[chrisdearden]

you need to look harder into the R2 release notes. There is a registry key introduced to aid customers who do not want more than a certain number of open snapshots on a datastore - given that each snapshot can grow with data change on the underlying Virtual machine , you want to avoid the situation where snapshot deltas will fill up the datastore , impacting all of the machines on it.

Added ability to limit maximum amount of active VM snapshots per datastore to prevent it from being overfilled with snapshot deltas. The default value of 4 active snapshots can be controlled with MaxSnapshotsPerDatastore (REG_DWORD) registry key.

in your case simply add that reg key and up the number of snapshots to something you feel comfortable with. You may need to restart the backup service after adding the key.

[chjones]

Thanks Chris. I've updated the registry key and will see how it goes.

Perhaps rather than listing such keys that can be modified in the release notes, which only tend to be read once and then never used again, these should be listed in the help files for Veeam or on your website in a central configuration register or the like? Somewhere that we can all go to that lists all of the customisations we can make.

This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.

Thanks for your response.

Chris

[Gostev]

This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.

Backup from storage snapshots works differently (VM snapshot is present for a brief moment only), so this setting does not apply there. Anyway, the reason why this setting is not exposed in the UI because it is added as a part of the patch release, where we cannot really touch the UI.

[davidb1234]

Wow I have been chasing this for a week.... I missed this. Nowhere in the documentation does it say WHERE the registry key needs to be created. Can you be more specific about the location of the key and the value? Is this in Decimal or Hex?

[foggy]

All registry values we have go into the single key, which is why we don't repeat this with every new value we introduce.

The key is:

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

The value type for this is DWORD. Sounds like you are confused about base option though, so let me explain this for you.

Basically, you should pick hex or decimal option depending on whether you are supplying the value in hex or in decimal format, for example:
- If you enter 11 and select decimal, the value stored will be 11 decimal (DWORD 0x0000000b)
- If you enter 11 and select hex, the value stored will be 17 decimal (DWORD 0x00000011)

Normally you just enter the value you want in "human" number system, and tell regedit its decimal

[chjones]

Backup from storage snapshots works differently (VM snapshot is present for a brief moment only), so this setting does not apply there. Anyway, the reason why this setting is not exposed in the UI because it is added as a part of the patch release, where we cannot really touch the UI.

Thanks Anton. The VM snapshots are only present for a short period, however a snapshot is created for all VMs in the job before the storage snapshot is taken. Therefore if you have more than 5 VMs in the job you will hit the limit.

I have increased our max snapshots to be more than the total number of concurrent VMs we can process at any time based on the sizing of our proxies and backup repository. We have Storage Snapshots working after the hotfix from Veeam Support but are not looking at enabling it for all jobs until after the new year. When that time comes we will likely need to increase the max snapshots value again to allow our jobs to complete, we'll see how that goes.

Thanks.

[tsightler]

Thanks Anton. The VM snapshots are only present for a short period, however a snapshot is created for all VMs in the job before the storage snapshot is taken. Therefore if you have more than 5 VMs in the job you will hit the limit.

I think what Anton is saying is that the per-datastore limit on snapshots does not apply when using backup from storage snapshots, only when backups are from traditional VMware snapshots.

[Gostev]

That's right. If you are saying this does apply, then this is a bug that we must fix.

[marius roma]

I get the error "Resouce not ready: Active snapshosts limit reached for datastore" in a replica job.
How can I understand if the problem occurred in the source or in the target?
I rerad in some other posts that the problem can be solved by changing a registry key: should it be changed on the Veeam B&R server or in any server hosting the proxyies?
Regards
marius

[Shestakov]

Hello Marius.
You need to create a REG_DWORD key in

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

folder of VBR server and give it a desired value.
Feel free to contact Veeam technical support for assistance. Thanks!

[marius roma]

Should I make the registry change on the Veeam B&R server or in any server hosting the proxyies?
Regards
Marius

[dellock6]

Unless otherwise advised, registry keys need to be created in the VBR server.

[Shestakov]

In this particular case you don`t need to create registry keys on proxy servers.

[rurouni]

Hello Everybody, i'm joining this thread,

We are running Backups on Both San Storage (3par with Storage Snapshots) and Nutanix hyperconverged appliances. As in the concept of Datastore management with Nutanix boxes, there is only one Huge Container(Datastore) provisionned for hosting vms.

Since we implemented backups on Nutanix boxes with Direct NFS access, we are facing this limitation during backups:

"Resource not ready: Active snapshots limit reached for datastore"

I found out that there is a Registry key allowing to bypass this limit. MaxSnapshotsPerDatastore
I would like to know if this value is a global value affecting all the datastores or if i'm able to control on which datastores i want to set or bypass a limit.
I don't want to run on an other issue on my 3PAR san datastores.

Regards.

Jean-Michel

[Vitaliy S.]

That's global. If you want to limit the load on source datastores, you can also consider using max concurrent tasks setting for each proxy server used to retrieve VMs data.

[rurouni]

Hi Vitaly,
I do not want to reduce the number of concurrent tasks on my proxy since they are under used and it will probably lengthen my backup window.
Higher in this Thread i remember Gostev said that this settings did not affect datastores processed with SAN storage snapshot feature.

Does that mean i can confidently raise the value to allow my Nutanix Containers to handle more snapshots that 4 at a time while not affecting my 3par SAN Datastores?

Regards

Jean-Michel

[Vitaliy S.]

Yes, if you want to increase this limit for Nutanix Containers, then feel free to do that. This key doesn't affect datastores used in SAN storage snapshot integration.

[odge]

I'm surprised we cant set this per datastore from the GUI yet, any plans to do so

[Gostev]

Tweaking this has been an extremely rare need (as you can see from the date of the previous post before yours) for us to justify bringing this setting into the UI... I do however plan to eventually move all registry settings into the dedicated UI section in a slightly improved format that is still however similar to the registry. This is primarily to make these settings easier to set, and to make them stored in the configuration database along with all other settings - so that they are backed up and restored with the configuration backup.

[odge]

Tweaking this has been an extremely rare need (as you can see from the date of the previous post before yours) for us to justify bringing this setting into the UI... I do however plan to eventually move all registry settings into the dedicated UI section in a slightly improved format that is still however similar to the registry. This is primarily to make these settings easier to set, and to make them stored in the configuration database along with all other settings - so that they are backed up and restored with the configuration backup.

That sounds like a good idea!

The reason we want to increase it, is we have jobs sending to a WAN repository (By WAN, I mean 2 layer 2 DCS, connected at 100mbps- but the latency seems to affect throughput). Those jobs hit that repo, and are using up 4 simultaneous jobs - slowly. The source datastore is VSAN all-flash, so it can handle the 4 jobs over WAN, plus another 4 Jobs to another SAN onsite.

[mma]

...I do however plan to eventually move all registry settings into the dedicated UI section...

https://forums.veeam.com/veeam-backup-r ... es#p253256

HALLELUJA!

[toreml]

Hi.

I get this message when I'm running backup jobs.
We have not set any limitations on the backup job under advanced.

All i've found regarding this is some registry key - MaxSnapshotsPerDatastore * that can be set on the backup and replication server - but that was for ver 7,5.

We are running ver 9,5 - and is this registry key still valid, or is there some other place I can change increase the maximum snapshots?
Today it's set to 4 for the vcenter.

Tore

[DGrinev]

Hi Tore,

The tweaking tool is the registry key you can edit.
Please review this thread for additional information. Thanks!

[ratkinsonuk]

As the number of hyper-converged installations is likely to start increasing (specifically effecting this key), can I give a +1 for moving some of these key registry settings into the GUI configuration pages.

Rob

[Kirill.Naumenko]

Hi all!

Please help me - is this counter controlled by Veeam? Or this information gets from vcenter? When Veeam Backup server makes a snapshot on one or another datastore, does it record 1 to some variable, and then increase it after next one? Does this counter go to each datastore separately?
How I can control this counter? May be from PowerCli?

[Andreas Neufert]

It is controlled by Veeam. Please see above for the Registry key to control this. This limitation is in place to avoid issues with the production load. So please increase this only with care.

[Kirill.Naumenko]

I saw information about Registry key. But I don't understand how it works. By default (when key doesn't created) value is 4. But I have some doubts about it. Therefore, all my questions in order to understand how it works.

[Andreas Neufert]

Each time we create a snapshot for a VM we check how many snapshots we have created for backup. This is per Datastore. There is no counter, we check actual situation.

[Andreas Neufert]

There is no way to check this as there is no counter. If you open the logs from backup job, you should get information at snapshot creation stage about the current value.