Danger of snapshotting production VMs

[christiankelly]

I wanted to pose a question to the Veeam community as well as Veeam about some issues I’ve seen with the usage of snapshots on production VMs. I have gotten into some very “sticky” situations when VMware snapshots don’t get removed correctly and can quickly end up with a long chain of snapshots which can’t be removed. So far with VMware support I have been able to get the VMs back to normal but in one case I needed to clone the VM to do so.

I know that VMware requires snapshots for backups and replication via the backup API, so these problems are not unique to Veeam and would show up in VMware’s own backup product. (Although I heard last year at VMworld that for replication VMware is was moving to something other than snapshots, but are not opening up the API. #VMwareFail)

Has anyone else run into issues with snapshots on production VMs? So far I have run into two main issues:
- VMware fails on the removal of a snapshot because of some locked file or unexplained VMware bug and it takes a VMware support call and a lot of work to get things back in order.
- Very busy production VMs can’t stay in a snapshot without experiencing degraded performance which prevents using Veeam replication if it can’t fit into a small enough window where users wouldn’t be effected.

I know the second issue isn’t resolvable without an architectural change or the ability to create a replica from a backup. (This would allow for a “quick” local backup and then a slow replication of the backup which could take as long as needed without degrading performance)

As far as a “fix” to the first issue. I would love to see an option in Veeam which you could force the job to fail if the production VM had a snapshot in place. This would stop Veeam from exacerbating the issue if the snapshot couldn’t be removed. I have found if I can stop the replication or backup job when it only has 5 snapshots vs. 20 it’s easier to get rid of them. I would see it as a checkbox under “advanced” where it would say “Skip VM if snapshot exists”.

I’m not sure if others would see value in an option like this or if I’m the only one that has had to deal with production VM’s that have 20 or 30 snapshots chained together on Monday morning because a client had replication running every 4 hours and had a failure late on Friday.

Thanks,

[howesh]

I have run into similar issues, but I have done the following. If you cannot remove the snapshots, it has always been for me that the drives are mounted in my veeam server. I just edit the veeam server and disconnect the drives. Then the snapshots remove without an issue. Second turn on snapshot warnings. If a snapshot grows over a gigabyte in size there is a warning. That way in the morning if there is a warning I can see if a snapshot did not get removed.

[christiankelly]

Yes some of the easier cases have been that the VM was mounted on the proxy VM. But there have been many cases where this was not the fix. Most of my clients have the snapshot warning, but it doesn't seem to be enough as if you have replication or backups at a high interval you can get into trouble very fast.

[veremin]

Although I heard last year at VMworld that for replication VMware is was moving to something other than snapshots.

As far as I’m concerned, VMware Replication still relies on the snapshots, though on the special type of them - during each replication cycle hidden snapshot is being created in order to keep the replicated state intact. This is a special type of snapshot that is not visible in the Snapshot Manager.

As far as a “fix” to the first issue. I would love to see an option in Veeam which you could force the job to fail if the production VM had a snapshot in place.

As a potential workaround you might want to write a custom PS script that would:

1) Check whether VMs within a given job have snapshots.
2) In case there are snapshots present, the script would output a corresponding message regarding it and exclude these VMs from being processed.
3) Start a backup job.

If you need any assistance with scripting it, don’t hesitate to let me know.

Additionally, I’d strongly recommend deploying Veeam One in your environment, since it provides you with a number of alarms and monitoring data through which you can analyze state of your virtual and backup infrastructure. The presence of obsolete snapshots can be successfully tracked by it, as well.

Anyway, thanks for the feedback; highly-appreciated.

Hope this helps.
Thanks.

[tuscani]

I have been burned.. see:

http://forums.veeam.com/viewtopic.php?f=24&t=15037

Keep in mine though this was ESX 3.5

Have never had this issue with 4.1.. *knocks on wood*

Only issues we have had on 4.1 is stunned VMs when creating snapshots manually.. never had a stunned VM with VEEAM based snapshots though.. kind of weird.

[yastt]

Unfortunately, I have exact same problem, even in the latest version of Veeam 7 Enterprise Plus. Despite reports of the proper removal of snapshots that I see in Veeam logs, snapshots are not deleted.
I did manage a simple script that I run every night from Task Scheduler.

#command to run from scheduler:
powershell.exe " & 'path\delete_snapshots.ps1' "

#delete_snapshots.ps1 code:
add-pssnapin VMware.VimAutomation.Core
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -WarningAction SilentlyContinue -Confirm:$False
connect-viserver -server name_of_your_vcenter
Get-Snapshot -vm * -name "VEEAM BACKUP*" | Remove-Snapshot -Confirm:$false
Get-VM | Where-Object { $ _.Extensiondata.Runtime.ConsolidationNeeded } | ForEach-Object { $_.ExtensionData.ConsolidateVMDisks () }

[foggy]

Thanks for sharing, much appreciated.

Also, I would like to note that there is a known issue with v7, where snapshots are not committed at the end of the job if vCenter server is added to Veeam console both as a Windows server and as the vCenter server under the same name. There is a workaround available through support for this particular case.

[pendragoncrw]

We have had this problem as well going back to Veeam 5 and ESXi 4.0.

What we have done to compensate are two things:
1. Setup vCenter alarms for snapshots larger than 2gb. This typically indicates either a very busy server in a backup window or a snapshot that did not get removed properly.
2. Setup a VeeamOne report that monitors for snapshots older than 2 days and review the report each morning.

Our standard procedure with orphan snapshots has been to power-down the VM if the snapshots are very big (12gb+), take a manual snapshot, then choose delete all. If the snapshots are smaller, we do it with the vm powered on, but during the least busy time of day.

[Gostev]

Unfortunately, I have exact same problem, even in the latest version of Veeam 7 Enterprise Plus. Despite reports of the proper removal of snapshots that I see in Veeam logs, snapshots are not deleted.

If this happens all the time, make sure it is not this known issue > http://www.veeam.com/KB1810