-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: Aug 27, 2014 4:47 am
- Contact:
Re: Snapshot removal issues of a large VM
We have the same Problem on Veeam Backup & Replication v7 7.0.0.871. Some VM´s stuck for a very long time after snapshot removal. (Windows Server 2008 R2 Std)
A Reboot of the affected Virtual Machines fixes the Problem temporarily.
A Reboot of the affected Virtual Machines fixes the Problem temporarily.
-
- Enthusiast
- Posts: 97
- Liked: 6 times
- Joined: Feb 24, 2009 5:02 pm
- Contact:
Re: Snapshot removal issues of a large VM
This is not a Veeam issue, it's down to the way snapshot merges occur.
See my previous post (http://forums.veeam.com/vmware-vsphere- ... tml#p99749) and for a better understanding check out http://vmdamentals.com/?p=332
We found the best solution without resorting to tricky backup scheduling is to use a network keepalive/heartbeat where possible. In some cases the slightest latency in network comms is critical.
See my previous post (http://forums.veeam.com/vmware-vsphere- ... tml#p99749) and for a better understanding check out http://vmdamentals.com/?p=332
We found the best solution without resorting to tricky backup scheduling is to use a network keepalive/heartbeat where possible. In some cases the slightest latency in network comms is critical.
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: Aug 27, 2014 4:47 am
- Contact:
Re: Snapshot removal issues of a large VM
Same Problem today. After The Backups succeeded the VM and the OS stuck.
After we migrate the VM to another Host the VM works fine. VMware Support and Veeam Support finds no solution ;-(
After we migrate the VM to another Host the VM works fine. VMware Support and Veeam Support finds no solution ;-(
-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Re: Snapshot removal issues of a large VM
Same here, both cases opened still no resolution
-
- VP, Product Management
- Posts: 27302
- Liked: 2775 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
What are you case IDs? What about datastore performance (latency, IOPs) when snapshot commit operation starts? Have you tried to relocate the affected VM to less loaded datastore just to check if it helps or not?
-
- Service Provider
- Posts: 13
- Liked: 5 times
- Joined: Sep 08, 2011 12:47 am
- Full Name: Donny Nicol
- Contact:
Re: Snapshot removal issues of a large VM
We've been working through some VMware related issues with the OS of VMs locking up and requiring a vMotion or suspend/resume to get things unstuck again. VMware have directed us to the following article referencing an issue with VM performance - Virtual machines running on VMware ESXi 5.5 with vShield endpoint activated get into a hung state during snapshot operations. There is a solution being released as part of the ESXi 55EP05 (Express Patch 5) which should be released sometime in the first week of December.
Wherever you go there you are.
-
- Service Provider
- Posts: 13
- Liked: 5 times
- Joined: Sep 08, 2011 12:47 am
- Full Name: Donny Nicol
- Contact:
Re: Snapshot removal issues of a large VM
This update is now available and can be seen in vSphere Update Manager. http://kb.vmware.com/kb/2096282.nicold wrote:We've been working through some VMware related issues with the OS of VMs locking up and requiring a vMotion or suspend/resume to get things unstuck again. VMware have directed us to the following article referencing an issue with VM performance - Virtual machines running on VMware ESXi 5.5 with vShield endpoint activated get into a hung state during snapshot operations. There is a solution being released as part of the ESXi 55EP05 (Express Patch 5) which should be released sometime in the first week of December.
Wherever you go there you are.
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: Aug 27, 2014 4:47 am
- Contact:
Re: Snapshot removal issues of a large VM
Hello together,
thats unbelievable. VMware ESXi 5.5, Patch ESXi550-201412401-BG fixes the Problem.
We are so happy
Thanks a lot nicold !!
Wish all of you a happy new year.
Best regards.
thats unbelievable. VMware ESXi 5.5, Patch ESXi550-201412401-BG fixes the Problem.
We are so happy
Thanks a lot nicold !!
Wish all of you a happy new year.
Best regards.
-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Nov 13, 2014 10:55 am
- Full Name: Martin Zidu
- Contact:
Re: Snapshot removal issues of a large VM
We had this issue to for one of our large VM.
I have found out, after many days of investigation, that problem was, that there were old "Temporary Veeam Snapshot" in vSphere.
It was obvious, because "temporary veeam snapshots" are created during backup process and after finish are removed. This one was about week old.
I have deleted it in vSphere and then backups normally working again.
I have found out, after many days of investigation, that problem was, that there were old "Temporary Veeam Snapshot" in vSphere.
It was obvious, because "temporary veeam snapshots" are created during backup process and after finish are removed. This one was about week old.
I have deleted it in vSphere and then backups normally working again.
-
- Enthusiast
- Posts: 27
- Liked: 1 time
- Joined: Jul 19, 2012 8:52 pm
- Full Name: Matthew Shull
- Contact:
Re: Snapshot removal issues of a large VM
Just had this same exact issue. Our files server VM has become quite large, about 2.2 TB. All has been well until a couple weeks ago when at the end of the work day, we were put into a panic when the file server "went down" and was unresponsive. But only for a couple minutes, then it came back. I wasn't sure what the cause was, but I knew the VM itself hadn't gone down, just the access to it. Discovered that the Veeam backup was causing it as it removed the snapshot.
I saw this post and check, and sure enough I had 2 Temporary Veeam Snapshots. I deleted those (one of which caused unresponsiveness when deleted), and now the backup runs without issue.
I saw this post and check, and sure enough I had 2 Temporary Veeam Snapshots. I deleted those (one of which caused unresponsiveness when deleted), and now the backup runs without issue.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Apr 06, 2015 2:08 pm
- Full Name: James PS
- Contact:
[MERGED] Problem with vmware snapshot and veeam
Dear tech guys,
We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help
We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Snapshot removal issues of a large VM
Hello James,
Veeam does not manage snapshot creation/removal tasks but rather issues the API call to vSphere to do that. There seems to be an issue on VMware side.
Please review the topic for the additional information.
Thanks.
Veeam does not manage snapshot creation/removal tasks but rather issues the API call to vSphere to do that. There seems to be an issue on VMware side.
Please review the topic for the additional information.
Thanks.
-
- Enthusiast
- Posts: 35
- Liked: 10 times
- Joined: Mar 03, 2015 9:32 pm
- Full Name: Jon Brite
- Contact:
Re: [MERGED] Problem with vmware snapshot and veeam
Try moving the disks on your large VM to a SAS aggregate as they are probably on SATA. We experienced the same issue. Moving the VMs to our SAS drives eliminated this issue for us.jamesarems wrote:Dear tech guys,
We have a test lab , a veeam backup and replication scenario with vmware vcenter . We are using vmware esxi 5.5 update 2 and vcenter 5.5 and the latest veeam 8 replication software. Actually when we taking backup and replication of virtual machines more than 1TB size , vmware is not responding and giving task message like "Deleting snapshot 99%" . After two days it will get deleted but the same time veeam backup is not responding because "snapshot removal in process" . Is this problem is with veeam or vmware ? Please help
-
- Novice
- Posts: 5
- Liked: never
- Joined: Apr 15, 2015 7:22 pm
- Full Name: Chase Freeman
- Contact:
[MERGED] Exchange 2013 Disconnected during Backup
Hello,
We have Veeam 8.0 update 2 and for some reason when running a backup or replication, the connection to our Exchange 2013 VM is lost. We cannot figure out how to keep this from happening. It is frustrating because our end-users mail gets stuck in the their outbox and they are forced to restart their outlook programs while the backup is taking place. We obviously try to run it while the least amount of people on here but this is the only VM that drops connections. Any ideas?
thanks,
Chase
We have Veeam 8.0 update 2 and for some reason when running a backup or replication, the connection to our Exchange 2013 VM is lost. We cannot figure out how to keep this from happening. It is frustrating because our end-users mail gets stuck in the their outbox and they are forced to restart their outlook programs while the backup is taking place. We obviously try to run it while the least amount of people on here but this is the only VM that drops connections. Any ideas?
thanks,
Chase
-
- Veeam Software
- Posts: 21124
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Snapshot removal issues of a large VM
Chase, you can find some hints on how to eliminate this effect in this thread. Thanks.
-
- Novice
- Posts: 5
- Liked: never
- Joined: May 15, 2015 1:13 pm
- Contact:
[MERGED] Packets Dropping During Replication and Backup - ES
When a backup job is running we are getting notifications that our VM's are occasionally dropping a small number of packets (never more than 1,000). We are running Veaam 8.0.0.817 and ESXi 5.5.0.0 2068190. The drops happen throughout the time the backup or replication job is running and one VM may experience pack drop 2 or 3 times (it normally takes 30 minutes to run the backup and replication jobs).
I've seen some posting where this was kind of expected behavior in older version of ESXi, but nothing about v 5.5. Is this still something that is expected? Is it actually a problem? I don't think we are using VSS in our job set up (no exchange or SQL), but would that fix this (or should we be using it anyways as a best practice)?
Thanks for taking the time to read this and thanks for your help
I've seen some posting where this was kind of expected behavior in older version of ESXi, but nothing about v 5.5. Is this still something that is expected? Is it actually a problem? I don't think we are using VSS in our job set up (no exchange or SQL), but would that fix this (or should we be using it anyways as a best practice)?
Thanks for taking the time to read this and thanks for your help
-
- Veeam Software
- Posts: 21124
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Snapshot removal issues of a large VM
Do you probably see this during the snapshot commit operation?
-
- Novice
- Posts: 5
- Liked: never
- Joined: May 15, 2015 1:13 pm
- Contact:
Re: Snapshot removal issues of a large VM
Foogy,
I'm going to assume you were directing that comment at me, since my question has been merged into this beast of a post (although I'm not sure it really should be). How would I go about checking to see if this occurred during a commit process? My gut instinct would be to say no since we'll get the message of dropped packets for the same VM several times (minutes apart) during a single backup, but if you could point me in the direction of how I would check to see if it does coincide with the commit I'll take a look.
Thanks
I'm going to assume you were directing that comment at me, since my question has been merged into this beast of a post (although I'm not sure it really should be). How would I go about checking to see if this occurred during a commit process? My gut instinct would be to say no since we'll get the message of dropped packets for the same VM several times (minutes apart) during a single backup, but if you could point me in the direction of how I would check to see if it does coincide with the commit I'll take a look.
Thanks
-
- Product Manager
- Posts: 20343
- Liked: 2281 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
Check the job session statistics and see whether the problem happens during step that is called "Removing VM snapshot". Thanks.
-
- Novice
- Posts: 5
- Liked: never
- Joined: May 15, 2015 1:13 pm
- Contact:
Re: Snapshot removal issues of a large VM
There is no timestamp in there for the individual steps, but it looks like the answer is sometimes. It might also be happening during the Remove step but it is definitely happening at other times even when the specific VM isn't actively being backed up.
-
- Product Manager
- Posts: 20343
- Liked: 2281 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
What do you mean by "when the specific VM isn't actively being backed up"? VMs sometimes drop packets, even when no backup job are running against them? Anyway, it might be worth trying to reproduce that by manually creating a snapshot manually, keeping it long enough (the time similar to the one backup job takes), deleting it and seeing whether the issue occurs. Thanks.
-
- Novice
- Posts: 5
- Liked: never
- Joined: May 15, 2015 1:13 pm
- Contact:
Re: Snapshot removal issues of a large VM
Sorry for the confusion.
What I mean is that the entire job runs for ~30 minutes. During this the individual VMs only run a for a few minutes each (depending on the size). So VM1 may be the first to back up and it will drop packets while it is actually backing up and it might drop some more later after it is finished but the entire job is still running (say VM 4 is actually backing up). Does that make sense?
What I mean is that the entire job runs for ~30 minutes. During this the individual VMs only run a for a few minutes each (depending on the size). So VM1 may be the first to back up and it will drop packets while it is actually backing up and it might drop some more later after it is finished but the entire job is still running (say VM 4 is actually backing up). Does that make sense?
-
- Product Manager
- Posts: 20343
- Liked: 2281 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
Got it. Though, once the backup is finished for particular VM, Veeam doesn't touch any longer, so, hardly packet drops might be related to VB&R in this case (if, of course, your time observations are correct).
Anyway, I'd still recommend performing manual test that I've described just to see whether the issue can or cannot be reproduced without Veeam presence.
Thanks.
Anyway, I'd still recommend performing manual test that I've described just to see whether the issue can or cannot be reproduced without Veeam presence.
Thanks.
-
- Novice
- Posts: 5
- Liked: never
- Joined: May 15, 2015 1:13 pm
- Contact:
Re: Snapshot removal issues of a large VM
v.Eremin,
Thanks for the suggestion. When I did a manual snapshot it also experienced packet loss, so I'm guessing this is really a VMWare issues, instead of Veeam, right?
Thanks for the suggestion. When I did a manual snapshot it also experienced packet loss, so I'm guessing this is really a VMWare issues, instead of Veeam, right?
-
- Veeam Software
- Posts: 21124
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Snapshot removal issues of a large VM
Yes, since you can rule out Veeam B&R from the picture completely.
-
- Service Provider
- Posts: 234
- Liked: 40 times
- Joined: Mar 08, 2010 4:05 pm
- Full Name: John Borhek
- Contact:
[MERGED] Need to reduce stun time
I am looking for strategies to reduce snapshot stun time. On a 15 minute replication, it is perceived that the snapshot stun is an issue and I would like to consider any and all strategies for reducing it.
Any Ideas? Thanks,
-J
Any Ideas? Thanks,
-J
John Borhek, Solutions Architect
https://vmsources.com
https://vmsources.com
-
- Veeam Software
- Posts: 21124
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Snapshot removal issues of a large VM
John, some hints on reducing snapshot commit time are discussed in this huge thread. Also, consider replicating from backups that does not involve snapshots of the production VMs at all.
-
- Service Provider
- Posts: 234
- Liked: 40 times
- Joined: Mar 08, 2010 4:05 pm
- Full Name: John Borhek
- Contact:
Re: Snapshot removal issues of a large VM
Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.
P.S. I thought that posting on such an old/long thread was discouraged.
P.S. I thought that posting on such an old/long thread was discouraged.
John Borhek, Solutions Architect
https://vmsources.com
https://vmsources.com
-
- Veeam Software
- Posts: 21124
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Snapshot removal issues of a large VM
Right, I missed the frequency thing. What kind of source storage do you have? Any chance of using storage snapshots for replication?unsichtbarre wrote:Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.
We always try to keep similar discussions in one place for more effective search.unsichtbarre wrote:P.S. I thought that posting on such an old/long thread was discouraged.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Jun 17, 2015 5:14 pm
- Full Name: Michael Johnson
- Contact:
Re: Snapshot removal issues of a large VM
We are noticing the same sort of issues with vm snapshots. The interesting for us is this just started happening when we updated vSphere with the latest updates (v 5.5.0, 2718055). It was about a month ago and ever since that time we have see this issue.
What version of vSphere 5.5 are you on? 2718055?
Thanks,
-Mike
What version of vSphere 5.5 are you on? 2718055?
Thanks,
-Mike
Who is online
Users browsing this forum: Amazon [Bot], Google [Bot], Semrush [Bot] and 16 guests