Quiesced snapshots failing after latest Windows 2016 Update

[Amarokada]

This is not a Veeam issue per se, but could likely be something Veeam users might also experience so I'm posting here for any recent experiences from Veeam users.

I have built 2 brand new vSphere 6.7 U2 clusters, and have a vanilla Windows 2016 VM (with paravirtual scsi, and vmxnet3) on them. When triggering a snapshot with quiescence it works 100% of the time, and vssadmin list writers shows all writers with no error, and state: stable.

As soon as I install the latest April cumulative updates (KB4493470 or KB4493473) which are from 9th April and 25th April consecutively, I can no longer take quiesced snapshots. They fail with:

The guest OS has reported an error during quiescing. The error code was: 5 The error message was: 'VssSyncStart' operation failed: IDispatch error #8723 (0x80042413)

When looking at the VM straight after, 5-6 VSS writers state [5] "Waiting for Completion". Then if you wait for some minutes more only the WMI Writer and System Writer stay in this waiting state, and never clear until you restart the cryptographic and WMI services. But the next snapshot will just do the same error again.

This control VM is just the OS with no applications installed, except for VMware tools (I've tried 10.3.5 and 10.3.10).

Is anyone else having issues in a similar environment after upgrading to the April Windows 2016 patches?

[Gostev]

Hello, we were unable to reproduce the issue internally with a quick test. Could you please open a support case to let us take a look at your environment closer? Thanks.

[Gostev]

Just to be sure, are you using VMware Tools quiescence or Veeam's application-aware processing?

[Amarokada]

This is simply with the Vmware tools quiescence, not application aware mode, which is why I know this is not a Veeam issue per se but it is the first time I'm seeing this in a repeatable fashion. Each time I install the April Windows patches it fails, and if I roll back to the snapshot I have before the updates it's fine again.

I have raised ticket 03562330 just now and attached logs.

Thanks for your help.

[Gostev]

Ah, sorry I misunderstood your originally.

For everyone else: we were able to reproduce the issue at the customer's environment without Veeam involved, by using VMware Tools quiescense when creating VM snapshots without VM memory (no issue otherwise).

[Amarokada]

I have raised a ticket with Microsoft now, as this seems to be something related to our CIS hardening GPO and April+ Windows updates. When either is present on its own -quiesce is fine, but with both applied the VSS writers get locked up. I'm hoping they can delve deeping into VSS as see what has changed.

[Amarokada]

I'll reply back here when/if Microsoft are able to track down the Windows component that has been updated that causes this. From event logs alone there is nothing extra showing I can see between the failed and working -quiesce snapshots.

One of the reasons I found out about this bug is from cloning an online Windows VM (which secretly triggers the -quiesce option to snapshot the VM and clone the snapshot). I never knew it did this under the hood as there is no message in vCenter about it unless it fails.

My thanks to Veeam for taking an interest in this issue, despite this not actually being a Veeam bug in any way.