Host-based backup of VMware vSphere VMs.
Post Reply
jbennett
Enthusiast
Posts: 33
Liked: 12 times
Joined: Jun 23, 2015 3:47 pm
Full Name: Justin Bennett
Location: Los Angeles, CA
Contact:

Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett » 1 person likes this post

We've been backing up these VMs for months successfully with VMware Tools Quiescence disabled. They're all running on a single ESXi 6.0 host, all VMs running Windows Server 2012 R2 and VMware Tools build 9536.

The other day, I enabled "VMware Tools Quiescence", the backup job kicked off a failure to create snapshot "an error occurred while saving the snapshot: failed to quiesce the virtual machine". Other 2012 and 2012 R2 VMs with the same VMware Tools build had no issue backing up from the same host.

Found four related articles - none resolved the issue.

http://kb.vmware.com/selfservice/micros ... Id=1002310
http://kb.vmware.com/selfservice/micros ... Id=2034002
http://kb.vmware.com/selfservice/micros ... Id=2069952
http://kb.vmware.com/selfservice/micros ... Id=2006849

I've confirmed that the OS is receiving Event Log Errors 140, 157, 12289. I've tried shutting down the VMs, reinstalling VMware Tools, and rebooting them. No luck.

Image
Image
Image
Image

What DID fix the issue. I migrated the VMs to a newer ESXi host and upgraded VMware Tools to build 9541. This "appears" to have resolved / fixed issue. I'm am seeing Event Log Errors for 147, 150, 12289, but the backup is now completing when creating the snapshot. I'm going to migrate the VMs back during an off hour and test backup again - (going from virtual standard switch (VSS) to virtual distributed switch (VDS) is ok, but its failing to migrate from VDS to VSS).

Image
Image

Best of luck!
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
alanbolte
Veteran
Posts: 635
Liked: 174 times
Joined: Jun 18, 2012 8:58 pm
Full Name: Alan Bolte
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by alanbolte »

Why use VMware Tools Quiesce instead of Application Aware Image Processing on a Windows server?
jbennett
Enthusiast
Posts: 33
Liked: 12 times
Joined: Jun 23, 2015 3:47 pm
Full Name: Justin Bennett
Location: Los Angeles, CA
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett »

We are also using Application Aware Image Processing. We've been told VMware Tools Quiescence may help limit/minimize the snapshot VM stun - is this incorrect?
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
Posts: 33
Liked: 12 times
Joined: Jun 23, 2015 3:47 pm
Full Name: Justin Bennett
Location: Los Angeles, CA
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett »

I take that back - upon further inspection, these VMs do not have Application Aware Processing, because their network connection live beyond access of our Veeam B&R server. They're only accessible from our station network, not our server network.
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by mkretzer » 1 person likes this post

Hello,

we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

It seems like 50 % of our tools upgraded VMs can no longer be quienced.

We opened a VMware ticket.

MK
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by PTide »

these VMs do not have Application Aware Processing, because their network connection live beyond access of our Veeam B&R server.
Have you considered using VIX API?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by Vitaliy S. »

jbennett wrote:We've been told VMware Tools Quiescence may help limit/minimize the snapshot VM stun - is this incorrect?
Yes, there is no difference in how VM snapshot is committed, so this shouldn't be the case. If you want to minimize the impact of snapshot commit operation, then integration with storage snapshots will definitely help here. Also we have an existing topic discussing this issue, please check it out, should be useful.
mkretzer wrote:we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

It seems like 50 % of our tools upgraded VMs can no longer be quienced.
Does this also happen when you create VM snapshots manually via vSphere Client?
jbennett
Enthusiast
Posts: 33
Liked: 12 times
Joined: Jun 23, 2015 3:47 pm
Full Name: Justin Bennett
Location: Los Angeles, CA
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett »

PTide wrote:Have you considered using VIX API?
I like the VIX API option, but really, we need to reconfigure our VMs so Veeam can communicate with them. These systems that were erring are only RDP Hosts and I wasn't really too concerned about application aware processing, but its nice to at least have the VMware Tools Quiesce function working - thus sharing all the info.
Vitaliy S. wrote:Yes, there is no difference in how VM snapshot is committed, so this shouldn't be the case. If you want to minimize the impact of snapshot commit operation, then integration with storage snapshots will definitely help here. Also we have an existing topic discussing this issue, please check it out, should be useful.
We've been considering storage integration and been testing it out. Unfortunately, our current SANs are either not compatible or need major upgrading - besides the Ent Plus upgrade.

Thank you for this information about VIX API and storage snapshot integration.

Image
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
Posts: 33
Liked: 12 times
Joined: Jun 23, 2015 3:47 pm
Full Name: Justin Bennett
Location: Los Angeles, CA
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett »

FYI

I've just confirmed that the problem persisted after migrating the VMs back to the original ESXi host. The build number of the ESXi host showing the issue is 6.0.0, 2494585. So, my initial thought of upgrading VMware Tools from build 9536 going to build 9541 being a "fix" was wrong and meant nothing towards my issue.

The ESXi host that seemed to resolve the issue once the VMs were migrated to it is build number 6.0.0, 3247720.

I'm planning on patching the troubled ESXi host soon and I'll report if upgrading it to the latest build makes any change.
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by mkretzer » 1 person likes this post

Vitaliy S. wrote:Does this also happen when you create VM snapshots manually via vSphere Client?
Sure, 100 % reproducable.

MK
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by Vitaliy S. »

Ok, please let us know what VMware support team responds to your ticket.
jbennett
Enthusiast
Posts: 33
Liked: 12 times
Joined: Jun 23, 2015 3:47 pm
Full Name: Justin Bennett
Location: Los Angeles, CA
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett » 2 people like this post

After updating the ESXi host running the VMs from 6.0.0, 2494585 to 6.0.0, 3247720 this weekend, the issue has appeared to been resolved - backups with VMware Tools Quiescence enabled and no snapshot issues. The solution maybe in the ESXi host's build or maybe in the ESXi host's reboot, not sure - just giving my feedback.

Good luck!
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
btmaus
Expert
Posts: 138
Liked: 10 times
Joined: Jul 17, 2015 9:02 am
Full Name: Glenn L
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by btmaus »

mkretzer wrote:we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.
Same here, also experiencing the issue since going to 5.5U3a. VMs are in a DMZ network that VBR cannot reach, so have to use VMware Quiesce.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by Vitaliy S. »

btmaus wrote:VMs are in a DMZ network that VBR cannot reach, so have to use VMware Quiesce.
You can use AAIP option for DMZ VMs too, network connection from VBR server to protected VMs is not required. Pre-freeze process can be done via VIX API, moreover it is done automatically if direct network connection cannot be used.
Post Reply

Who is online

Users browsing this forum: Lewpy and 84 guests