Discussions specific to the VMware vSphere hypervisor
Post Reply
ITGerman
Novice
Posts: 3
Liked: 1 time
Joined: Aug 27, 2014 4:47 am
Contact:

Re: Snapshot removal issues of a large VM

Post by ITGerman » Aug 27, 2014 4:53 am

We have the same Problem on Veeam Backup & Replication v7 7.0.0.871. Some VM´s stuck for a very long time after snapshot removal. (Windows Server 2008 R2 Std)
A Reboot of the affected Virtual Machines fixes the Problem temporarily.

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by cby » Sep 01, 2014 8:54 am

This is not a Veeam issue, it's down to the way snapshot merges occur.

See my previous post (http://forums.veeam.com/vmware-vsphere- ... tml#p99749) and for a better understanding check out http://vmdamentals.com/?p=332

We found the best solution without resorting to tricky backup scheduling is to use a network keepalive/heartbeat where possible. In some cases the slightest latency in network comms is critical.

ITGerman
Novice
Posts: 3
Liked: 1 time
Joined: Aug 27, 2014 4:47 am
Contact:

Re: Snapshot removal issues of a large VM

Post by ITGerman » Nov 07, 2014 9:15 am

Same Problem today. After The Backups succeeded the VM and the OS stuck.
After we migrate the VM to another Host the VM works fine. VMware Support and Veeam Support finds no solution ;-(

claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Snapshot removal issues of a large VM

Post by claudiofolu » Dec 01, 2014 7:02 pm

Same here, both cases opened still no resolution

Vitaliy S.
Product Manager
Posts: 23183
Liked: 1602 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Dec 01, 2014 8:05 pm

What are you case IDs? What about datastore performance (latency, IOPs) when snapshot commit operation starts? Have you tried to relocate the affected VM to less loaded datastore just to check if it helps or not?

nicold
Service Provider
Posts: 13
Liked: 5 times
Joined: Sep 08, 2011 12:47 am
Full Name: Donny Nicol
Contact:

Re: Snapshot removal issues of a large VM

Post by nicold » Dec 02, 2014 1:44 am 2 people like this post

We've been working through some VMware related issues with the OS of VMs locking up and requiring a vMotion or suspend/resume to get things unstuck again. VMware have directed us to the following article referencing an issue with VM performance - Virtual machines running on VMware ESXi 5.5 with vShield endpoint activated get into a hung state during snapshot operations. There is a solution being released as part of the ESXi 55EP05 (Express Patch 5) which should be released sometime in the first week of December.
Wherever you go there you are.

nicold
Service Provider
Posts: 13
Liked: 5 times
Joined: Sep 08, 2011 12:47 am
Full Name: Donny Nicol
Contact:

Re: Snapshot removal issues of a large VM

Post by nicold » Dec 04, 2014 3:23 am 2 people like this post

nicold wrote:We've been working through some VMware related issues with the OS of VMs locking up and requiring a vMotion or suspend/resume to get things unstuck again. VMware have directed us to the following article referencing an issue with VM performance - Virtual machines running on VMware ESXi 5.5 with vShield endpoint activated get into a hung state during snapshot operations. There is a solution being released as part of the ESXi 55EP05 (Express Patch 5) which should be released sometime in the first week of December.
This update is now available and can be seen in vSphere Update Manager. http://kb.vmware.com/kb/2096282.
Wherever you go there you are.

ITGerman
Novice
Posts: 3
Liked: 1 time
Joined: Aug 27, 2014 4:47 am
Contact:

Re: Snapshot removal issues of a large VM

Post by ITGerman » Dec 29, 2014 1:12 pm 1 person likes this post

Hello together,

thats unbelievable. VMware ESXi 5.5, Patch ESXi550-201412401-BG fixes the Problem.

We are so happy ;-)

Thanks a lot nicold !!

Wish all of you a happy new year.

Best regards.

martin.zidu
Lurker
Posts: 2
Liked: 1 time
Joined: Nov 13, 2014 10:55 am
Full Name: Martin Zidu
Contact:

Re: Snapshot removal issues of a large VM

Post by martin.zidu » Jan 05, 2015 6:36 am 1 person likes this post

We had this issue to for one of our large VM.

I have found out, after many days of investigation, that problem was, that there were old "Temporary Veeam Snapshot" in vSphere.

It was obvious, because "temporary veeam snapshots" are created during backup process and after finish are removed. This one was about week old.

I have deleted it in vSphere and then backups normally working again.

fdizone
Enthusiast
Posts: 27
Liked: 1 time
Joined: Jul 19, 2012 8:52 pm
Full Name: Matthew Shull
Contact:

Re: Snapshot removal issues of a large VM

Post by fdizone » Jan 06, 2015 8:21 pm 1 person likes this post

Just had this same exact issue. Our files server VM has become quite large, about 2.2 TB. All has been well until a couple weeks ago when at the end of the work day, we were put into a panic when the file server "went down" and was unresponsive. But only for a couple minutes, then it came back. I wasn't sure what the cause was, but I knew the VM itself hadn't gone down, just the access to it. Discovered that the Veeam backup was causing it as it removed the snapshot.

I saw this post and check, and sure enough I had 2 Temporary Veeam Snapshots. I deleted those (one of which caused unresponsiveness when deleted), and now the backup runs without issue.

jamesarems
Lurker
Posts: 1
Liked: never
Joined: Apr 06, 2015 2:08 pm
Full Name: James PS
Contact:

[MERGED] Problem with vmware snapshot and veeam

Post by jamesarems » Apr 06, 2015 2:21 pm

Dear tech guys,

We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help

Shestakov
Veeam Software
Posts: 7039
Liked: 725 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Snapshot removal issues of a large VM

Post by Shestakov » Apr 06, 2015 2:39 pm

Hello James,
Veeam does not manage snapshot creation/removal tasks but rather issues the API call to vSphere to do that. There seems to be an issue on VMware side.
Please review the topic for the additional information.
Thanks.

jb1095
Enthusiast
Posts: 35
Liked: 10 times
Joined: Mar 03, 2015 9:32 pm
Full Name: Jon Brite
Contact:

Re: [MERGED] Problem with vmware snapshot and veeam

Post by jb1095 » Apr 06, 2015 5:16 pm 2 people like this post

jamesarems wrote:Dear tech guys,

We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help
Try moving the disks on your large VM to a SAS aggregate as they are probably on SATA. We experienced the same issue. Moving the VMs to our SAS drives eliminated this issue for us.

cfreeman4798
Novice
Posts: 5
Liked: never
Joined: Apr 15, 2015 7:22 pm
Full Name: Chase Freeman
Contact:

[MERGED] Exchange 2013 Disconnected during Backup

Post by cfreeman4798 » May 14, 2015 3:15 pm

Hello,

We have Veeam 8.0 update 2 and for some reason when running a backup or replication, the connection to our Exchange 2013 VM is lost. We cannot figure out how to keep this from happening. It is frustrating because our end-users mail gets stuck in the their outbox and they are forced to restart their outlook programs while the backup is taking place. We obviously try to run it while the least amount of people on here but this is the only VM that drops connections. Any ideas?

thanks,

Chase

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » May 14, 2015 3:48 pm

Chase, you can find some hints on how to eliminate this effect in this thread. Thanks.

Benji.Sawyer
Novice
Posts: 5
Liked: never
Joined: May 15, 2015 1:13 pm
Contact:

[MERGED] Packets Dropping During Replication and Backup - ES

Post by Benji.Sawyer » May 15, 2015 1:25 pm

When a backup job is running we are getting notifications that our VM's are occasionally dropping a small number of packets (never more than 1,000). We are running Veaam 8.0.0.817 and ESXi 5.5.0.0 2068190. The drops happen throughout the time the backup or replication job is running and one VM may experience pack drop 2 or 3 times (it normally takes 30 minutes to run the backup and replication jobs).

I've seen some posting where this was kind of expected behavior in older version of ESXi, but nothing about v 5.5. Is this still something that is expected? Is it actually a problem? I don't think we are using VSS in our job set up (no exchange or SQL), but would that fix this (or should we be using it anyways as a best practice)?

Thanks for taking the time to read this and thanks for your help

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » May 15, 2015 2:21 pm

Do you probably see this during the snapshot commit operation?

Benji.Sawyer
Novice
Posts: 5
Liked: never
Joined: May 15, 2015 1:13 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by Benji.Sawyer » May 18, 2015 1:07 am

Foogy,

I'm going to assume you were directing that comment at me, since my question has been merged into this beast of a post (although I'm not sure it really should be). How would I go about checking to see if this occurred during a commit process? My gut instinct would be to say no since we'll get the message of dropped packets for the same VM several times (minutes apart) during a single backup, but if you could point me in the direction of how I would check to see if it does coincide with the commit I'll take a look.

Thanks

veremin
Product Manager
Posts: 17047
Liked: 1470 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » May 18, 2015 8:36 am

Check the job session statistics and see whether the problem happens during step that is called "Removing VM snapshot". Thanks.

Benji.Sawyer
Novice
Posts: 5
Liked: never
Joined: May 15, 2015 1:13 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by Benji.Sawyer » May 19, 2015 12:20 am

There is no timestamp in there for the individual steps, but it looks like the answer is sometimes. It might also be happening during the Remove step but it is definitely happening at other times even when the specific VM isn't actively being backed up.

veremin
Product Manager
Posts: 17047
Liked: 1470 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » May 19, 2015 9:25 am

What do you mean by "when the specific VM isn't actively being backed up"? VMs sometimes drop packets, even when no backup job are running against them? Anyway, it might be worth trying to reproduce that by manually creating a snapshot manually, keeping it long enough (the time similar to the one backup job takes), deleting it and seeing whether the issue occurs. Thanks.

Benji.Sawyer
Novice
Posts: 5
Liked: never
Joined: May 15, 2015 1:13 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by Benji.Sawyer » May 19, 2015 3:08 pm

Sorry for the confusion.

What I mean is that the entire job runs for ~30 minutes. During this the individual VMs only run a for a few minutes each (depending on the size). So VM1 may be the first to back up and it will drop packets while it is actually backing up and it might drop some more later after it is finished but the entire job is still running (say VM 4 is actually backing up). Does that make sense?

veremin
Product Manager
Posts: 17047
Liked: 1470 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » May 20, 2015 8:14 am

Got it. Though, once the backup is finished for particular VM, Veeam doesn't touch any longer, so, hardly packet drops might be related to VB&R in this case (if, of course, your time observations are correct).

Anyway, I'd still recommend performing manual test that I've described just to see whether the issue can or cannot be reproduced without Veeam presence.

Thanks.

Benji.Sawyer
Novice
Posts: 5
Liked: never
Joined: May 15, 2015 1:13 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by Benji.Sawyer » May 22, 2015 3:20 pm

v.Eremin,
Thanks for the suggestion. When I did a manual snapshot it also experienced packet loss, so I'm guessing this is really a VMWare issues, instead of Veeam, right?

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » May 22, 2015 3:43 pm

Yes, since you can rule out Veeam B&R from the picture completely.

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

[MERGED] Need to reduce stun time

Post by unsichtbarre » Jun 15, 2015 3:11 pm

I am looking for strategies to reduce snapshot stun time. On a 15 minute replication, it is perceived that the snapshot stun is an issue and I would like to consider any and all strategies for reducing it.

Any Ideas? Thanks,
-J
-The Invisible Admin-
http://www.johnborhek.com

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » Jun 15, 2015 4:00 pm

John, some hints on reducing snapshot commit time are discussed in this huge thread. Also, consider replicating from backups that does not involve snapshots of the production VMs at all.

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Snapshot removal issues of a large VM

Post by unsichtbarre » Jun 15, 2015 4:39 pm

Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.

P.S. I thought that posting on such an old/long thread was discouraged.
-The Invisible Admin-
http://www.johnborhek.com

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » Jun 16, 2015 9:10 am

unsichtbarre wrote:Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.
Right, I missed the frequency thing. What kind of source storage do you have? Any chance of using storage snapshots for replication?
unsichtbarre wrote:P.S. I thought that posting on such an old/long thread was discouraged.
We always try to keep similar discussions in one place for more effective search.

michael.johnson
Lurker
Posts: 1
Liked: never
Joined: Jun 17, 2015 5:14 pm
Full Name: Michael Johnson
Contact:

Re: Snapshot removal issues of a large VM

Post by michael.johnson » Jun 17, 2015 5:23 pm

We are noticing the same sort of issues with vm snapshots. The interesting for us is this just started happening when we updated vSphere with the latest updates (v 5.5.0, 2718055). It was about a month ago and ever since that time we have see this issue.

What version of vSphere 5.5 are you on? 2718055?

Thanks,
-Mike

Post Reply

Who is online

Users browsing this forum: No registered users and 16 guests