Active snapshots limit reached for datastore

VMware specific discussions

Active snapshots limit reached for datastore

Veeam Logoby chjones » Thu Dec 05, 2013 9:48 pm

Hi,

We are running Veeam 7.0 R2 and our backup jobs have been taking a lot longer recently to complete and I've started to notice the following message in the jobs for VMs that are pending:

Resouce not ready: Active snapshosts limit reached for datastore

We have 6 hosts in a vSphere 5.1 Cluster and use a HP 3PAR as our storage array. When it came to sizing the datastore I discussed our setup with HP and was advised there was no real difference between creating multiple smaller datastores or a single large datastore. The 3PAR can create 16TB Virtual Volumes so we created a single 16TB Datastore with Thin Detect enabled and use Thick Eager Zeroed for all of our VMs.

The maximum number of powered on VMs per datastore in vSphere 5.1 is 2048. The cluster I am seeing the above message on has about 90 VMs. I currently have 4 Backup Jobs that run each night for these VMs, the jobs are split out to backup certain folders as we group our VMs based on the business unit that uses them.

I cannot find any reference to a limit on the maximum number of snapshots per datastore.

We are yet to enable Storage Snapshots for the 3PAR as we've only just got that working with a patch from Veeam Tech Suppport. Im thinking if I was to enable that the situation would be worse as Veeam snapshots every VM in the job before taking the storage snapshot which would mean a lot more snapshots would exist at the same time.

Has anyone see anything about the active snapshots limit?

Thanks,

Chris
chjones
Enthusiast
 
Posts: 83
Liked: 25 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Active snapshots limit reached for datastore

Veeam Logoby chrisdearden » Fri Dec 06, 2013 8:01 am

you need to look harder into the R2 release notes. There is a registry key introduced to aid customers who do not want more than a certain number of open snapshots on a datastore - given that each snapshot can grow with data change on the underlying Virtual machine , you want to avoid the situation where snapshot deltas will fill up the datastore , impacting all of the machines on it.

Added ability to limit maximum amount of active VM snapshots per datastore to prevent it from being overfilled with snapshot deltas. The default value of 4 active snapshots can be controlled with MaxSnapshotsPerDatastore (REG_DWORD) registry key.


in your case simply add that reg key and up the number of snapshots to something you feel comfortable with. You may need to restart the backup service after adding the key.
chrisdearden
Expert
 
Posts: 1529
Liked: 225 times
Joined: Wed Jul 21, 2010 9:47 am
Full Name: Chris Dearden

Re: Active snapshots limit reached for datastore

Veeam Logoby chjones » Sun Dec 08, 2013 10:01 pm

Thanks Chris. I've updated the registry key and will see how it goes.

Perhaps rather than listing such keys that can be modified in the release notes, which only tend to be read once and then never used again, these should be listed in the help files for Veeam or on your website in a central configuration register or the like? Somewhere that we can all go to that lists all of the customisations we can make.

This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.

Thanks for your response.

Chris
chjones
Enthusiast
 
Posts: 83
Liked: 25 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Active snapshots limit reached for datastore

Veeam Logoby Gostev » Tue Dec 10, 2013 5:58 pm

chjones wrote:This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.

Backup from storage snapshots works differently (VM snapshot is present for a brief moment only), so this setting does not apply there. Anyway, the reason why this setting is not exposed in the UI because it is added as a part of the patch release, where we cannot really touch the UI.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Active snapshots limit reached for datastore

Veeam Logoby davidb1234 » Tue Dec 10, 2013 7:16 pm

Wow I have been chasing this for a week.... I missed this. Nowhere in the documentation does it say WHERE the registry key needs to be created. Can you be more specific about the location of the key and the value? Is this in Decimal or Hex?
davidb1234
Expert
 
Posts: 144
Liked: 15 times
Joined: Tue Nov 15, 2011 8:47 pm
Full Name: David Borden

Re: Active snapshots limit reached for datastore

Veeam Logoby foggy » Tue Dec 10, 2013 7:50 pm

All registry values we have go into the single key, which is why we don't repeat this with every new value we introduce.

The key is:
Code: Select all
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

The value type for this is DWORD. Sounds like you are confused about base option though, so let me explain this for you.

Basically, you should pick hex or decimal option depending on whether you are supplying the value in hex or in decimal format, for example:
- If you enter 11 and select decimal, the value stored will be 11 decimal (DWORD 0x0000000b)
- If you enter 11 and select hex, the value stored will be 17 decimal (DWORD 0x00000011)

Normally you just enter the value you want in "human" number system, and tell regedit its decimal :wink:
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Active snapshots limit reached for datastore

Veeam Logoby chjones » Mon Dec 16, 2013 12:35 am

Gostev wrote:
chjones wrote:This setting defaulting to 4 and not being exposed in the GUI seems a bit strange to me, especially with the push for Storage Snapshots. If you have 5 VMs on a 3PAR Datastore and you enable Storage Snapshot Integration you cannot backup all of the VMs in a single job as you'll hit 4 snaphots and then the job will wait before creating the fifth snapshot.

Backup from storage snapshots works differently (VM snapshot is present for a brief moment only), so this setting does not apply there. Anyway, the reason why this setting is not exposed in the UI because it is added as a part of the patch release, where we cannot really touch the UI.


Thanks Anton. The VM snapshots are only present for a short period, however a snapshot is created for all VMs in the job before the storage snapshot is taken. Therefore if you have more than 5 VMs in the job you will hit the limit.

I have increased our max snapshots to be more than the total number of concurrent VMs we can process at any time based on the sizing of our proxies and backup repository. We have Storage Snapshots working after the hotfix from Veeam Support but are not looking at enabling it for all jobs until after the new year. When that time comes we will likely need to increase the max snapshots value again to allow our jobs to complete, we'll see how that goes.

Thanks.
chjones
Enthusiast
 
Posts: 83
Liked: 25 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Active snapshots limit reached for datastore

Veeam Logoby tsightler » Mon Dec 16, 2013 3:43 am

chjones wrote:Thanks Anton. The VM snapshots are only present for a short period, however a snapshot is created for all VMs in the job before the storage snapshot is taken. Therefore if you have more than 5 VMs in the job you will hit the limit.


I think what Anton is saying is that the per-datastore limit on snapshots does not apply when using backup from storage snapshots, only when backups are from traditional VMware snapshots.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Active snapshots limit reached for datastore

Veeam Logoby Gostev » Mon Dec 16, 2013 1:55 pm

That's right. If you are saying this does apply, then this is a bug that we must fix.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

[MERGED] Active snapshosts limit reached for datastore

Veeam Logoby marius roma » Fri Jun 17, 2016 6:33 am

I get the error "Resouce not ready: Active snapshosts limit reached for datastore" in a replica job.
How can I understand if the problem occurred in the source or in the target?
I rerad in some other posts that the problem can be solved by changing a registry key: should it be changed on the Veeam B&R server or in any server hosting the proxyies?
Regards
marius
marius roma
Expert
 
Posts: 317
Liked: 3 times
Joined: Wed Feb 01, 2012 12:04 pm
Full Name: Mario

Re: Active snapshots limit reached for datastore

Veeam Logoby Shestakov » Fri Jun 17, 2016 11:35 am

Hello Marius.
You need to create a REG_DWORD key in
Code: Select all
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

folder of VBR server and give it a desired value.
Feel free to contact Veeam technical support for assistance. Thanks!
Shestakov
Veeam Software
 
Posts: 4861
Liked: 395 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Active snapshots limit reached for datastore

Veeam Logoby marius roma » Fri Jun 17, 2016 3:31 pm

Should I make the registry change on the Veeam B&R server or in any server hosting the proxyies?
Regards
Marius
marius roma
Expert
 
Posts: 317
Liked: 3 times
Joined: Wed Feb 01, 2012 12:04 pm
Full Name: Mario

Re: Active snapshots limit reached for datastore

Veeam Logoby dellock6 » Fri Jun 17, 2016 3:33 pm

Unless otherwise advised, registry keys need to be created in the VBR server.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Active snapshots limit reached for datastore

Veeam Logoby Shestakov » Fri Jun 17, 2016 3:50 pm

In this particular case you don`t need to create registry keys on proxy servers.
Shestakov
Veeam Software
 
Posts: 4861
Liked: 395 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Active snapshots limit reached for datastore

Veeam Logoby rurouni » Thu Jul 14, 2016 6:40 pm

Hello Everybody, i'm joining this thread,

We are running Backups on Both San Storage (3par with Storage Snapshots) and Nutanix hyperconverged appliances. As in the concept of Datastore management with Nutanix boxes, there is only one Huge Container(Datastore) provisionned for hosting vms.

Since we implemented backups on Nutanix boxes with Direct NFS access, we are facing this limitation during backups:

"Resource not ready: Active snapshots limit reached for datastore"

I found out that there is a Registry key allowing to bypass this limit. MaxSnapshotsPerDatastore
I would like to know if this value is a global value affecting all the datastores or if i'm able to control on which datastores i want to set or bypass a limit.
I don't want to run on an other issue on my 3PAR san datastores.

Regards.

Jean-Michel
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 9 guests