VMware Tools Quiescence vs App Aware?

[nmace]

I just read Gostev's email newsletter about issues with VMware Tools Quiescence vs App Aware issues.

I have this checkbox enabled for all of my jobs, especially my domain controller job.

Enable application-aware processing

So if I am understanding the issue in the newsletter correctly, I'm doing things the correct way and the VMware Tools Quiescence issue isn't an issue for us. Is this correct?

Thanks!

[Andreas Neufert]

Correct. For Windows VMs when you enabled application-aware processing you force the Veeam own guest processing.

The other option can be found under Job-Storage-Advanced button-VMware. For Windows when both are enabled the Veeam one will be used.

[Didi7]

Ok, this is interesting? Is there any public info about this besides Gostev's newletter? If this is of great concern, your competitors would have much trouble, because only Veeam has AAIP to circumvent problems with VMware tools quiescence!

Could be a trigger for some to switch to Veeam, wouldn't it?

Regards,
Didi7

[mkretzer]

We were the company with the corruption.

First of all, i think this is a corner case as we had this enabled for 4000 VMs for years now and only now had a measurable, reproducible corruption one one Domain Controller. Also, i don't even know if its a REAL corruption - NTFS was nearly always marked as unclean (causing chkdsk at boot) but after doing chkdsk there were only minimal changed to the filesystem. Still, this caused quite some panic as somewhere, something happens with the filesystem which should not happen...

Also, the bugs causing this seem to be fixed in W2019.

So many customers might never run into this situation.

[Gostev]

Actually, when talking about the second (domain controller) issue, I was not talking about your company Michael. I did not even realize you also had DC affected with the corruption...

[Didi7]

Maybe the bug is fixed in Windows Server 2019 but not everybody uses the latest server os, so where do these bugs occur and is it only Veeam or even Gostev that knows about it?

Is there any public info available?

Does the bug only occur with a specific VMware tools version or are all versions affected and are all vSphere versions affected?

What about Veeam's competitors? Should they fear, that all there backups using VMware tools quiescence are none application consistent or is this complete nonsense?

[mkretzer]

Here you go: https://kb.vmware.com/s/article/2006849

Interesting about this is the "Related Information" - i always believed that these messages are safe to ignore. Thats why i found it strange that Gostev pointed to these messages in eventlog!

[Gostev]

Well, at least in your case they were not "safe to ignore" since you did have some actual data loss/corruption, which was addressed by switching off VMware Tools quiescence

@Didi7 VMware will need to troubleshoot this. All we know at the moment is that disabling VMware Tools quiescence and switching to AAIP resolved the corresponding issues in both support cases, thus my recommendation.

[Didi7]

@mkretzer:
Thanks for the the KB.

@Gostev
Good to know.

[mkretzer]

Gostev, i am still not 100 % sure that the corruption and the messages are connected.

We had hundrets of machines with these messages and never before had issues...

The argumentation from VMware is basically "the NTFS is readonly so the messages are normal" - why should the volume be corrupted when it is read only?

[Gostev]

Because it's not actually read-only at least not all of the time!

VSS is quite a complexity monster really, it's just not obvious until you start digging deeper and learning about its internals like auto-recovery phase, which is a big pain in the butt to deal with. I only know about this because many years ago we had major issues with application-aware Exchange processing due to it, and in the end our devs had to add some incredible amount of code into both backup and restore logic just to get application-aware Exchange processing alone working reliably with auto-recovery!

Here's a short summary about the auto-recovery phase:

If the shadow copy is successfully created, the Volume Shadow Copy Service returns the location information for the shadow copy to the requester. In some cases, the shadow copy can be temporarily made available as a read-write volume so that VSS and one or more applications can alter the contents of the shadow copy before the shadow copy is finished. This phase is called Auto-recovery, and it is used to undo any file-system or application transactions on the shadow copy volume that were not completed before the shadow copy was created.

While I cannot be sure, I think it is very likely this phase is the culprit also here. Remember VMware Tools is not a backup application, so it is safe to assume VMware doesn't really test backup/restore scenarios with different applications thoroughly.