Snapshot removal issues of a large VM

[ITGerman]

We have the same Problem on Veeam Backup & Replication v7 7.0.0.871. Some VM´s stuck for a very long time after snapshot removal. (Windows Server 2008 R2 Std)
A Reboot of the affected Virtual Machines fixes the Problem temporarily.

[cby]

This is not a Veeam issue, it's down to the way snapshot merges occur.

See my previous post (http://forums.veeam.com/vmware-vsphere- ... tml#p99749) and for a better understanding check out http://vmdamentals.com/?p=332

We found the best solution without resorting to tricky backup scheduling is to use a network keepalive/heartbeat where possible. In some cases the slightest latency in network comms is critical.

[ITGerman]

Same Problem today. After The Backups succeeded the VM and the OS stuck.
After we migrate the VM to another Host the VM works fine. VMware Support and Veeam Support finds no solution ;-(

[claudiofolu]

Same here, both cases opened still no resolution

[Vitaliy S.]

What are you case IDs? What about datastore performance (latency, IOPs) when snapshot commit operation starts? Have you tried to relocate the affected VM to less loaded datastore just to check if it helps or not?

[nicold]

We've been working through some VMware related issues with the OS of VMs locking up and requiring a vMotion or suspend/resume to get things unstuck again. VMware have directed us to the following article referencing an issue with VM performance - Virtual machines running on VMware ESXi 5.5 with vShield endpoint activated get into a hung state during snapshot operations. There is a solution being released as part of the ESXi 55EP05 (Express Patch 5) which should be released sometime in the first week of December.

[nicold]

We've been working through some VMware related issues with the OS of VMs locking up and requiring a vMotion or suspend/resume to get things unstuck again. VMware have directed us to the following article referencing an issue with VM performance - Virtual machines running on VMware ESXi 5.5 with vShield endpoint activated get into a hung state during snapshot operations. There is a solution being released as part of the ESXi 55EP05 (Express Patch 5) which should be released sometime in the first week of December.

This update is now available and can be seen in vSphere Update Manager. http://kb.vmware.com/kb/2096282.

[ITGerman]

Hello together,

thats unbelievable. VMware ESXi 5.5, Patch ESXi550-201412401-BG fixes the Problem.

We are so happy

Thanks a lot nicold !!

Wish all of you a happy new year.

Best regards.

[martin.zidu]

We had this issue to for one of our large VM.

I have found out, after many days of investigation, that problem was, that there were old "Temporary Veeam Snapshot" in vSphere.

It was obvious, because "temporary veeam snapshots" are created during backup process and after finish are removed. This one was about week old.

I have deleted it in vSphere and then backups normally working again.

[fdizone]

Just had this same exact issue. Our files server VM has become quite large, about 2.2 TB. All has been well until a couple weeks ago when at the end of the work day, we were put into a panic when the file server "went down" and was unresponsive. But only for a couple minutes, then it came back. I wasn't sure what the cause was, but I knew the VM itself hadn't gone down, just the access to it. Discovered that the Veeam backup was causing it as it removed the snapshot.

I saw this post and check, and sure enough I had 2 Temporary Veeam Snapshots. I deleted those (one of which caused unresponsiveness when deleted), and now the backup runs without issue.

[jamesarems]

Dear tech guys,

We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help

[Shestakov]

Hello James,
Veeam does not manage snapshot creation/removal tasks but rather issues the API call to vSphere to do that. There seems to be an issue on VMware side.
Please review the topic for the additional information.
Thanks.

[jb1095]

Dear tech guys,

We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help

Try moving the disks on your large VM to a SAS aggregate as they are probably on SATA. We experienced the same issue. Moving the VMs to our SAS drives eliminated this issue for us.

[cfreeman4798]

Hello,

We have Veeam 8.0 update 2 and for some reason when running a backup or replication, the connection to our Exchange 2013 VM is lost. We cannot figure out how to keep this from happening. It is frustrating because our end-users mail gets stuck in the their outbox and they are forced to restart their outlook programs while the backup is taking place. We obviously try to run it while the least amount of people on here but this is the only VM that drops connections. Any ideas?

thanks,

Chase

[foggy]

Chase, you can find some hints on how to eliminate this effect in this thread. Thanks.

[Benji.Sawyer]

When a backup job is running we are getting notifications that our VM's are occasionally dropping a small number of packets (never more than 1,000). We are running Veaam 8.0.0.817 and ESXi 5.5.0.0 2068190. The drops happen throughout the time the backup or replication job is running and one VM may experience pack drop 2 or 3 times (it normally takes 30 minutes to run the backup and replication jobs).

I've seen some posting where this was kind of expected behavior in older version of ESXi, but nothing about v 5.5. Is this still something that is expected? Is it actually a problem? I don't think we are using VSS in our job set up (no exchange or SQL), but would that fix this (or should we be using it anyways as a best practice)?

Thanks for taking the time to read this and thanks for your help

[foggy]

Do you probably see this during the snapshot commit operation?

[Benji.Sawyer]

Foogy,

I'm going to assume you were directing that comment at me, since my question has been merged into this beast of a post (although I'm not sure it really should be). How would I go about checking to see if this occurred during a commit process? My gut instinct would be to say no since we'll get the message of dropped packets for the same VM several times (minutes apart) during a single backup, but if you could point me in the direction of how I would check to see if it does coincide with the commit I'll take a look.

Thanks

[veremin]

Check the job session statistics and see whether the problem happens during step that is called "Removing VM snapshot". Thanks.

[Benji.Sawyer]

There is no timestamp in there for the individual steps, but it looks like the answer is sometimes. It might also be happening during the Remove step but it is definitely happening at other times even when the specific VM isn't actively being backed up.

[veremin]

What do you mean by "when the specific VM isn't actively being backed up"? VMs sometimes drop packets, even when no backup job are running against them? Anyway, it might be worth trying to reproduce that by manually creating a snapshot manually, keeping it long enough (the time similar to the one backup job takes), deleting it and seeing whether the issue occurs. Thanks.

[Benji.Sawyer]

Sorry for the confusion.

What I mean is that the entire job runs for ~30 minutes. During this the individual VMs only run a for a few minutes each (depending on the size). So VM1 may be the first to back up and it will drop packets while it is actually backing up and it might drop some more later after it is finished but the entire job is still running (say VM 4 is actually backing up). Does that make sense?

[veremin]

Got it. Though, once the backup is finished for particular VM, Veeam doesn't touch any longer, so, hardly packet drops might be related to VB&R in this case (if, of course, your time observations are correct).

Anyway, I'd still recommend performing manual test that I've described just to see whether the issue can or cannot be reproduced without Veeam presence.

Thanks.

[Benji.Sawyer]

v.Eremin,
Thanks for the suggestion. When I did a manual snapshot it also experienced packet loss, so I'm guessing this is really a VMWare issues, instead of Veeam, right?

[foggy]

Yes, since you can rule out Veeam B&R from the picture completely.

[unsichtbarre]

I am looking for strategies to reduce snapshot stun time. On a 15 minute replication, it is perceived that the snapshot stun is an issue and I would like to consider any and all strategies for reducing it.

Any Ideas? Thanks,
-J

[foggy]

John, some hints on reducing snapshot commit time are discussed in this huge thread. Also, consider replicating from backups that does not involve snapshots of the production VMs at all.

[unsichtbarre]

Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.

P.S. I thought that posting on such an old/long thread was discouraged.

[foggy]

Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.

Right, I missed the frequency thing. What kind of source storage do you have? Any chance of using storage snapshots for replication?

P.S. I thought that posting on such an old/long thread was discouraged.

We always try to keep similar discussions in one place for more effective search.

[michael.johnson]

We are noticing the same sort of issues with vm snapshots. The interesting for us is this just started happening when we updated vSphere with the latest updates (v 5.5.0, 2718055). It was about a month ago and ever since that time we have see this issue.

What version of vSphere 5.5 are you on? 2718055?

Thanks,
-Mike