Backup Fails on VMs with VMware Tools Quiescence Enabled

VMware specific discussions

Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby jbennett » Wed Dec 16, 2015 10:31 pm 1 person likes this post

We've been backing up these VMs for months successfully with VMware Tools Quiescence disabled. They're all running on a single ESXi 6.0 host, all VMs running Windows Server 2012 R2 and VMware Tools build 9536.

The other day, I enabled "VMware Tools Quiescence", the backup job kicked off a failure to create snapshot "an error occurred while saving the snapshot: failed to quiesce the virtual machine". Other 2012 and 2012 R2 VMs with the same VMware Tools build had no issue backing up from the same host.

Found four related articles - none resolved the issue.

http://kb.vmware.com/selfservice/micros ... Id=1002310
http://kb.vmware.com/selfservice/micros ... Id=2034002
http://kb.vmware.com/selfservice/micros ... Id=2069952
http://kb.vmware.com/selfservice/micros ... Id=2006849

I've confirmed that the OS is receiving Event Log Errors 140, 157, 12289. I've tried shutting down the VMs, reinstalling VMware Tools, and rebooting them. No luck.

Image
Image
Image
Image

What DID fix the issue. I migrated the VMs to a newer ESXi host and upgraded VMware Tools to build 9541. This "appears" to have resolved / fixed issue. I'm am seeing Event Log Errors for 147, 150, 12289, but the backup is now completing when creating the snapshot. I'm going to migrate the VMs back during an off hour and test backup again - (going from virtual standard switch (VSS) to virtual distributed switch (VDS) is ok, but its failing to migrate from VDS to VSS).

Image
Image

Best of luck!
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
 
Posts: 33
Liked: 12 times
Joined: Tue Jun 23, 2015 3:47 pm
Location: Hemet, CA
Full Name: Justin Bennett

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby alanbolte » Wed Dec 16, 2015 10:52 pm

Why use VMware Tools Quiesce instead of Application Aware Image Processing on a Windows server?
alanbolte
Expert
 
Posts: 635
Liked: 170 times
Joined: Mon Jun 18, 2012 8:58 pm
Full Name: Alan Bolte

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby jbennett » Wed Dec 16, 2015 11:10 pm

We are also using Application Aware Image Processing. We've been told VMware Tools Quiescence may help limit/minimize the snapshot VM stun - is this incorrect?
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
 
Posts: 33
Liked: 12 times
Joined: Tue Jun 23, 2015 3:47 pm
Location: Hemet, CA
Full Name: Justin Bennett

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby jbennett » Wed Dec 16, 2015 11:34 pm

I take that back - upon further inspection, these VMs do not have Application Aware Processing, because their network connection live beyond access of our Veeam B&R server. They're only accessible from our station network, not our server network.
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
 
Posts: 33
Liked: 12 times
Joined: Tue Jun 23, 2015 3:47 pm
Location: Hemet, CA
Full Name: Justin Bennett

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby mkretzer » Thu Dec 17, 2015 7:21 am 1 person likes this post

Hello,

we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

It seems like 50 % of our tools upgraded VMs can no longer be quienced.

We opened a VMware ticket.

MK
mkretzer
Expert
 
Posts: 309
Liked: 69 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby PTide » Thu Dec 17, 2015 9:41 am

these VMs do not have Application Aware Processing, because their network connection live beyond access of our Veeam B&R server.
Have you considered using VIX API?
PTide
Veeam Software
 
Posts: 3021
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby Vitaliy S. » Thu Dec 17, 2015 10:36 am

jbennett wrote:We've been told VMware Tools Quiescence may help limit/minimize the snapshot VM stun - is this incorrect?

Yes, there is no difference in how VM snapshot is committed, so this shouldn't be the case. If you want to minimize the impact of snapshot commit operation, then integration with storage snapshots will definitely help here. Also we have an existing topic discussing this issue, please check it out, should be useful.

mkretzer wrote:we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

It seems like 50 % of our tools upgraded VMs can no longer be quienced.

Does this also happen when you create VM snapshots manually via vSphere Client?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby jbennett » Thu Dec 17, 2015 3:27 pm

PTide wrote:Have you considered using VIX API?


I like the VIX API option, but really, we need to reconfigure our VMs so Veeam can communicate with them. These systems that were erring are only RDP Hosts and I wasn't really too concerned about application aware processing, but its nice to at least have the VMware Tools Quiesce function working - thus sharing all the info.

Vitaliy S. wrote:Yes, there is no difference in how VM snapshot is committed, so this shouldn't be the case. If you want to minimize the impact of snapshot commit operation, then integration with storage snapshots will definitely help here. Also we have an existing topic discussing this issue, please check it out, should be useful.


We've been considering storage integration and been testing it out. Unfortunately, our current SANs are either not compatible or need major upgrading - besides the Ent Plus upgrade.

Thank you for this information about VIX API and storage snapshot integration.

Image
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
 
Posts: 33
Liked: 12 times
Joined: Tue Jun 23, 2015 3:47 pm
Location: Hemet, CA
Full Name: Justin Bennett

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby jbennett » Thu Dec 17, 2015 3:50 pm

FYI

I've just confirmed that the problem persisted after migrating the VMs back to the original ESXi host. The build number of the ESXi host showing the issue is 6.0.0, 2494585. So, my initial thought of upgrading VMware Tools from build 9536 going to build 9541 being a "fix" was wrong and meant nothing towards my issue.

The ESXi host that seemed to resolve the issue once the VMs were migrated to it is build number 6.0.0, 3247720.

I'm planning on patching the troubled ESXi host soon and I'll report if upgrading it to the latest build makes any change.
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
 
Posts: 33
Liked: 12 times
Joined: Tue Jun 23, 2015 3:47 pm
Location: Hemet, CA
Full Name: Justin Bennett

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby mkretzer » Sun Dec 20, 2015 9:03 pm 1 person likes this post

Vitaliy S. wrote:Does this also happen when you create VM snapshots manually via vSphere Client?


Sure, 100 % reproducable.

MK
mkretzer
Expert
 
Posts: 309
Liked: 69 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby Vitaliy S. » Mon Dec 21, 2015 12:21 pm

Ok, please let us know what VMware support team responds to your ticket.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby jbennett » Mon Dec 21, 2015 4:26 pm 2 people like this post

After updating the ESXi host running the VMs from 6.0.0, 2494585 to 6.0.0, 3247720 this weekend, the issue has appeared to been resolved - backups with VMware Tools Quiescence enabled and no snapshot issues. The solution maybe in the ESXi host's build or maybe in the ESXi host's reboot, not sure - just giving my feedback.

Good luck!
-Justin
[@cajeeper]|[http://www.allthingstechie.net]
jbennett
Enthusiast
 
Posts: 33
Liked: 12 times
Joined: Tue Jun 23, 2015 3:47 pm
Location: Hemet, CA
Full Name: Justin Bennett

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby btmaus » Tue Dec 22, 2015 9:26 am

mkretzer wrote:we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.


Same here, also experiencing the issue since going to 5.5U3a. VMs are in a DMZ network that VBR cannot reach, so have to use VMware Quiesce.
btmaus
Expert
 
Posts: 128
Liked: 9 times
Joined: Fri Jul 17, 2015 9:02 am
Full Name: Glenn L

Re: Backup Fails on VMs with VMware Tools Quiescence Enabled

Veeam Logoby Vitaliy S. » Tue Dec 22, 2015 10:17 am

btmaus wrote:VMs are in a DMZ network that VBR cannot reach, so have to use VMware Quiesce.

You can use AAIP option for DMZ VMs too, network connection from VBR server to protected VMs is not required. Pre-freeze process can be done via VIX API, moreover it is done automatically if direct network connection cannot be used.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 12 guests