Host-based backup of VMware vSphere VMs.
chjones
Expert
Posts: 117
Liked: 31 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Active snapshots limit reached for datastore

Post by chjones »

Hi,

We are running Veeam 7.0 R2 and our backup jobs have been taking a lot longer recently to complete and I've started to notice the following message in the jobs for VMs that are pending:

Resouce not ready: Active snapshosts limit reached for datastore

We have 6 hosts in a vSphere 5.1 Cluster and use a HP 3PAR as our storage array. When it came to sizing the datastore I discussed our setup with HP and was advised there was no real difference between creating multiple smaller datastores or a single large datastore. The 3PAR can create 16TB Virtual Volumes so we created a single 16TB Datastore with Thin Detect enabled and use Thick Eager Zeroed for all of our VMs.

The maximum number of powered on VMs per datastore in vSphere 5.1 is 2048. The cluster I am seeing the above message on has about 90 VMs. I currently have 4 Backup Jobs that run each night for these VMs, the jobs are split out to backup certain folders as we group our VMs based on the business unit that uses them.

I cannot find any reference to a limit on the maximum number of snapshots per datastore.

We are yet to enable Storage Snapshots for the 3PAR as we've only just got that working with a patch from Veeam Tech Suppport. Im thinking if I was to enable that the situation would be worse as Veeam snapshots every VM in the job before taking the storage snapshot which would mean a lot more snapshots would exist at the same time.

Has anyone see anything about the active snapshots limit?

Thanks,

Chris
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Active snapshots limit reached for datastore

Post by chrisdearden »

you need to look harder into the R2 release notes. There is a registry key introduced to aid customers who do not want more than a certain number of open snapshots on a datastore - given that each snapshot can grow with data change on the underlying Virtual machine , you want to avoid the situation where snapshot deltas will fill up the datastore , impacting all of the machines on it.
Added ability to limit maximum amount of active VM snapshots per datastore to prevent it from being overfilled with snapshot deltas. The default value of 4 active snapshots can be controlled with MaxSnapshotsPerDatastore (REG_DWORD) registry key.
in your case simply add that reg key and up the number of snapshots to something you feel comfortable with. You may need to restart the backup service after adding the key.
chjones
Expert
Posts: 117
Liked: 31 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: Active snapshots limit reached for datastore

Post by chjones »

Thanks Chris. I've updated the registry key and will see how it goes.

Perhaps rather than listing such keys that can be modified in the release notes, which only tend to be read once and then never used again, these should be listed in the help files for Veeam or on your website in a central configuration register or the like? Somewhere that we can all go to that lists all of the customisations we can make.

This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.

Thanks for your response.

Chris
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Active snapshots limit reached for datastore

Post by Gostev »

chjones wrote:This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.
Backup from storage snapshots works differently (VM snapshot is present for a brief moment only), so this setting does not apply there. Anyway, the reason why this setting is not exposed in the UI because it is added as a part of the patch release, where we cannot really touch the UI.
davidb1234
Expert
Posts: 162
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: Active snapshots limit reached for datastore

Post by davidb1234 »

Wow I have been chasing this for a week.... I missed this. Nowhere in the documentation does it say WHERE the registry key needs to be created. Can you be more specific about the location of the key and the value? Is this in Decimal or Hex?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Active snapshots limit reached for datastore

Post by foggy »

All registry values we have go into the single key, which is why we don't repeat this with every new value we introduce.

The key is:

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
The value type for this is DWORD. Sounds like you are confused about base option though, so let me explain this for you.

Basically, you should pick hex or decimal option depending on whether you are supplying the value in hex or in decimal format, for example:
- If you enter 11 and select decimal, the value stored will be 11 decimal (DWORD 0x0000000b)
- If you enter 11 and select hex, the value stored will be 17 decimal (DWORD 0x00000011)

Normally you just enter the value you want in "human" number system, and tell regedit its decimal :wink:
chjones
Expert
Posts: 117
Liked: 31 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: Active snapshots limit reached for datastore

Post by chjones »

Gostev wrote: Backup from storage snapshots works differently (VM snapshot is present for a brief moment only), so this setting does not apply there. Anyway, the reason why this setting is not exposed in the UI because it is added as a part of the patch release, where we cannot really touch the UI.
Thanks Anton. The VM snapshots are only present for a short period, however a snapshot is created for all VMs in the job before the storage snapshot is taken. Therefore if you have more than 5 VMs in the job you will hit the limit.

I have increased our max snapshots to be more than the total number of concurrent VMs we can process at any time based on the sizing of our proxies and backup repository. We have Storage Snapshots working after the hotfix from Veeam Support but are not looking at enabling it for all jobs until after the new year. When that time comes we will likely need to increase the max snapshots value again to allow our jobs to complete, we'll see how that goes.

Thanks.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Active snapshots limit reached for datastore

Post by tsightler »

chjones wrote:Thanks Anton. The VM snapshots are only present for a short period, however a snapshot is created for all VMs in the job before the storage snapshot is taken. Therefore if you have more than 5 VMs in the job you will hit the limit.
I think what Anton is saying is that the per-datastore limit on snapshots does not apply when using backup from storage snapshots, only when backups are from traditional VMware snapshots.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Active snapshots limit reached for datastore

Post by Gostev »

That's right. If you are saying this does apply, then this is a bug that we must fix.
marius roma
Veteran
Posts: 459
Liked: 5 times
Joined: Feb 01, 2012 12:04 pm
Full Name: Mario
Contact:

[MERGED] Active snapshosts limit reached for datastore

Post by marius roma »

I get the error "Resouce not ready: Active snapshosts limit reached for datastore" in a replica job.
How can I understand if the problem occurred in the source or in the target?
I rerad in some other posts that the problem can be solved by changing a registry key: should it be changed on the Veeam B&R server or in any server hosting the proxyies?
Regards
marius
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Active snapshots limit reached for datastore

Post by Shestakov »

Hello Marius.
You need to create a REG_DWORD key in

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
folder of VBR server and give it a desired value.
Feel free to contact Veeam technical support for assistance. Thanks!
marius roma
Veteran
Posts: 459
Liked: 5 times
Joined: Feb 01, 2012 12:04 pm
Full Name: Mario
Contact:

Re: Active snapshots limit reached for datastore

Post by marius roma »

Should I make the registry change on the Veeam B&R server or in any server hosting the proxyies?
Regards
Marius
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Active snapshots limit reached for datastore

Post by dellock6 »

Unless otherwise advised, registry keys need to be created in the VBR server.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Active snapshots limit reached for datastore

Post by Shestakov »

In this particular case you don`t need to create registry keys on proxy servers.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Active snapshots limit reached for datastore

Post by rurouni »

Hello Everybody, i'm joining this thread,

We are running Backups on Both San Storage (3par with Storage Snapshots) and Nutanix hyperconverged appliances. As in the concept of Datastore management with Nutanix boxes, there is only one Huge Container(Datastore) provisionned for hosting vms.

Since we implemented backups on Nutanix boxes with Direct NFS access, we are facing this limitation during backups:

"Resource not ready: Active snapshots limit reached for datastore"

I found out that there is a Registry key allowing to bypass this limit. MaxSnapshotsPerDatastore
I would like to know if this value is a global value affecting all the datastores or if i'm able to control on which datastores i want to set or bypass a limit.
I don't want to run on an other issue on my 3PAR san datastores.

Regards.

Jean-Michel
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Active snapshots limit reached for datastore

Post by Vitaliy S. »

That's global. If you want to limit the load on source datastores, you can also consider using max concurrent tasks setting for each proxy server used to retrieve VMs data.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Active snapshots limit reached for datastore

Post by rurouni »

Hi Vitaly,
I do not want to reduce the number of concurrent tasks on my proxy since they are under used and it will probably lengthen my backup window.
Higher in this Thread i remember Gostev said that this settings did not affect datastores processed with SAN storage snapshot feature.

Does that mean i can confidently raise the value to allow my Nutanix Containers to handle more snapshots that 4 at a time while not affecting my 3par SAN Datastores?

Regards

Jean-Michel
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Active snapshots limit reached for datastore

Post by Vitaliy S. »

Yes, if you want to increase this limit for Nutanix Containers, then feel free to do that. This key doesn't affect datastores used in SAN storage snapshot integration.
odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

Re: Active snapshots limit reached for datastore

Post by odge »

I'm surprised we cant set this per datastore from the GUI yet, any plans to do so
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Active snapshots limit reached for datastore

Post by Gostev » 2 people like this post

Tweaking this has been an extremely rare need (as you can see from the date of the previous post before yours) for us to justify bringing this setting into the UI... I do however plan to eventually move all registry settings into the dedicated UI section in a slightly improved format that is still however similar to the registry. This is primarily to make these settings easier to set, and to make them stored in the configuration database along with all other settings - so that they are backed up and restored with the configuration backup.
odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

Re: Active snapshots limit reached for datastore

Post by odge »

Gostev wrote:Tweaking this has been an extremely rare need (as you can see from the date of the previous post before yours) for us to justify bringing this setting into the UI... I do however plan to eventually move all registry settings into the dedicated UI section in a slightly improved format that is still however similar to the registry. This is primarily to make these settings easier to set, and to make them stored in the configuration database along with all other settings - so that they are backed up and restored with the configuration backup.
That sounds like a good idea!

The reason we want to increase it, is we have jobs sending to a WAN repository (By WAN, I mean 2 layer 2 DCS, connected at 100mbps- but the latency seems to affect throughput). Those jobs hit that repo, and are using up 4 simultaneous jobs - slowly. The source datastore is VSAN all-flash, so it can handle the 4 jobs over WAN, plus another 4 Jobs to another SAN onsite.
mma
Service Provider
Posts: 111
Liked: 21 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Active snapshots limit reached for datastore

Post by mma »

Gostev wrote:...I do however plan to eventually move all registry settings into the dedicated UI section...
https://forums.veeam.com/veeam-backup-r ... es#p253256

HALLELUJA! :mrgreen:
toreml
Expert
Posts: 103
Liked: 3 times
Joined: Sep 13, 2017 11:12 am
Full Name: Tore Mejlænder-Larsen
Contact:

[MERGED] Active snapshots limit reached for datastore

Post by toreml »

Hi.

I get this message when I'm running backup jobs.
We have not set any limitations on the backup job under advanced.

All i've found regarding this is some registry key - MaxSnapshotsPerDatastore * that can be set on the backup and replication server - but that was for ver 7,5.

We are running ver 9,5 - and is this registry key still valid, or is there some other place I can change increase the maximum snapshots?
Today it's set to 4 for the vcenter.

Tore
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Active snapshots limit reached for datastore

Post by DGrinev »

Hi Tore,

The tweaking tool is the registry key you can edit.
Please review this thread for additional information. Thanks!
ratkinsonuk
Expert
Posts: 101
Liked: 14 times
Joined: Dec 10, 2018 10:59 am
Full Name: Robert Atkinson
Contact:

Re: Active snapshots limit reached for datastore

Post by ratkinsonuk » 1 person likes this post

As the number of hyper-converged installations is likely to start increasing (specifically effecting this key), can I give a +1 for moving some of these key registry settings into the GUI configuration pages.

Rob
Kirill.Naumenko
Service Provider
Posts: 7
Liked: 1 time
Joined: Jul 30, 2018 8:47 am
Full Name: Kirill Naumenko
Contact:

Re: Active snapshots limit reached for datastore

Post by Kirill.Naumenko »

Hi all!

Please help me - is this counter controlled by Veeam? Or this information gets from vcenter? When Veeam Backup server makes a snapshot on one or another datastore, does it record 1 to some variable, and then increase it after next one? Does this counter go to each datastore separately?
How I can control this counter? May be from PowerCli?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Active snapshots limit reached for datastore

Post by Andreas Neufert »

It is controlled by Veeam. Please see above for the Registry key to control this. This limitation is in place to avoid issues with the production load. So please increase this only with care.
Kirill.Naumenko
Service Provider
Posts: 7
Liked: 1 time
Joined: Jul 30, 2018 8:47 am
Full Name: Kirill Naumenko
Contact:

Re: Active snapshots limit reached for datastore

Post by Kirill.Naumenko »

I saw information about Registry key. But I don't understand how it works. By default (when key doesn't created) value is 4. But I have some doubts about it. Therefore, all my questions in order to understand how it works.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Active snapshots limit reached for datastore

Post by Andreas Neufert »

Each time we create a snapshot for a VM we check how many snapshots we have created for backup. This is per Datastore. There is no counter, we check actual situation.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Active snapshots limit reached for datastore

Post by Andreas Neufert »

There is no way to check this as there is no counter. If you open the logs from backup job, you should get information at snapshot creation stage about the current value.
Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 65 guests