Backup Fails on VMs with VMware Tools Quiescence Enabled

[jbennett]

We've been backing up these VMs for months successfully with VMware Tools Quiescence disabled. They're all running on a single ESXi 6.0 host, all VMs running Windows Server 2012 R2 and VMware Tools build 9536.

The other day, I enabled "VMware Tools Quiescence", the backup job kicked off a failure to create snapshot "an error occurred while saving the snapshot: failed to quiesce the virtual machine". Other 2012 and 2012 R2 VMs with the same VMware Tools build had no issue backing up from the same host.

Found four related articles - none resolved the issue.

http://kb.vmware.com/selfservice/micros ... Id=1002310
http://kb.vmware.com/selfservice/micros ... Id=2034002
http://kb.vmware.com/selfservice/micros ... Id=2069952
http://kb.vmware.com/selfservice/micros ... Id=2006849

I've confirmed that the OS is receiving Event Log Errors 140, 157, 12289. I've tried shutting down the VMs, reinstalling VMware Tools, and rebooting them. No luck.






What DID fix the issue. I migrated the VMs to a newer ESXi host and upgraded VMware Tools to build 9541. This "appears" to have resolved / fixed issue. I'm am seeing Event Log Errors for 147, 150, 12289, but the backup is now completing when creating the snapshot. I'm going to migrate the VMs back during an off hour and test backup again - (going from virtual standard switch (VSS) to virtual distributed switch (VDS) is ok, but its failing to migrate from VDS to VSS).




Best of luck!

[alanbolte]

Why use VMware Tools Quiesce instead of Application Aware Image Processing on a Windows server?

[jbennett]

We are also using Application Aware Image Processing. We've been told VMware Tools Quiescence may help limit/minimize the snapshot VM stun - is this incorrect?

[jbennett]

I take that back - upon further inspection, these VMs do not have Application Aware Processing, because their network connection live beyond access of our Veeam B&R server. They're only accessible from our station network, not our server network.

[mkretzer]

Hello,

we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

It seems like 50 % of our tools upgraded VMs can no longer be quienced.

We opened a VMware ticket.

MK

[PTide]

these VMs do not have Application Aware Processing, because their network connection live beyond access of our Veeam B&R server.

Have you considered using VIX API?

[Vitaliy S.]

We've been told VMware Tools Quiescence may help limit/minimize the snapshot VM stun - is this incorrect?

Yes, there is no difference in how VM snapshot is committed, so this shouldn't be the case. If you want to minimize the impact of snapshot commit operation, then integration with storage snapshots will definitely help here. Also we have an existing topic discussing this issue, please check it out, should be useful.

we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

It seems like 50 % of our tools upgraded VMs can no longer be quienced.

Does this also happen when you create VM snapshots manually via vSphere Client?

[jbennett]

Have you considered using VIX API?

I like the VIX API option, but really, we need to reconfigure our VMs so Veeam can communicate with them. These systems that were erring are only RDP Hosts and I wasn't really too concerned about application aware processing, but its nice to at least have the VMware Tools Quiesce function working - thus sharing all the info.

Yes, there is no difference in how VM snapshot is committed, so this shouldn't be the case. If you want to minimize the impact of snapshot commit operation, then integration with storage snapshots will definitely help here. Also we have an existing topic discussing this issue, please check it out, should be useful.

We've been considering storage integration and been testing it out. Unfortunately, our current SANs are either not compatible or need major upgrading - besides the Ent Plus upgrade.

Thank you for this information about VIX API and storage snapshot integration.

[jbennett]

FYI

I've just confirmed that the problem persisted after migrating the VMs back to the original ESXi host. The build number of the ESXi host showing the issue is 6.0.0, 2494585. So, my initial thought of upgrading VMware Tools from build 9536 going to build 9541 being a "fix" was wrong and meant nothing towards my issue.

The ESXi host that seemed to resolve the issue once the VMs were migrated to it is build number 6.0.0, 3247720.

I'm planning on patching the troubled ESXi host soon and I'll report if upgrading it to the latest build makes any change.

[mkretzer]

Does this also happen when you create VM snapshots manually via vSphere Client?

Sure, 100 % reproducable.

MK

[Vitaliy S.]

Ok, please let us know what VMware support team responds to your ticket.

[jbennett]

After updating the ESXi host running the VMs from 6.0.0, 2494585 to 6.0.0, 3247720 this weekend, the issue has appeared to been resolved - backups with VMware Tools Quiescence enabled and no snapshot issues. The solution maybe in the ESXi host's build or maybe in the ESXi host's reboot, not sure - just giving my feedback.

Good luck!

[btmaus]

we have this issue since VMware tools 10 which is included in the latest ESX 5.5U3.

Same here, also experiencing the issue since going to 5.5U3a. VMs are in a DMZ network that VBR cannot reach, so have to use VMware Quiesce.

[Vitaliy S.]

VMs are in a DMZ network that VBR cannot reach, so have to use VMware Quiesce.

You can use AAIP option for DMZ VMs too, network connection from VBR server to protected VMs is not required. Pre-freeze process can be done via VIX API, moreover it is done automatically if direct network connection cannot be used.