Losing Windows Update History

[mkaec]

I've been having a problem for a while with Server 2012 R2 (and older) VMs in that they will regularly lose the Window Update history. The summary page will state that updates were never installed. Then if I click on " View Update History" it will be blank. If I install new updates, the summary page will show the date and time of installation and the history will list just the new updates. But, eventually, the system will revert to the previously described state.

I realized today that this is only happening with VMs I am backing up with Veeam B&R. I have some VMs that are either not backed up, or that are backed up by another application, and they are not exhibiting this problem.

One other data point is that this problem appears to have surfaced after upgrading the hosts to Windows Server 2016.

I'm using B&R 9.5.0.1536.

Is anyone else with Server 2016 hosts seeing, or not seeing, this behavior with the 2012 R2 (and older) VMs you are backing up with Veeam B&R?

[Mike Resseler]

Mark,

This is the first time I have ever heard of this. When you say it reverted back to the previous state, does that mean that the updates are not installed anymore? Or are they still installed but you can't see them in the history anymore?

I did a quick google and it looks like others have suffered from this also, but I couldn't quite find a solution for it. They did state that the updates were installed, just that they weren't visible anymore in update history

[mkaec]

I meant the status go back to the previous state. The updates stay installed, but the history goes away.

[Mike Resseler]

From what I can read, it happens if you use the windows update troubleshooter or when something happened to the C:\Windows\SoftwareDistribution folder. I'm pretty sure that backing up a VM with Veeam doesn't clean out that folder. Any change there are other tools that clean-out the history (for space savings for example)

[mkaec]

Yes, I've done my share of deleting out "C:\Windows\SoftwareDistribution" on Windows 7 systems to troubleshoot Windows update problems. (But, I've only done it on a handful of VMs when I was first troubleshooting this problem.) Now that I've discovered B&R may be involved, I'll do more troubleshooting around it. If I can reliably reproduce it, I'll open a support case.

[Mike Resseler]

Yes. That would be a good idea. If it is Veeam B&R we need to figure out why, but as I said, I have never seen that behavior from B&R (but then again, you never know with software )

[mkaec]

So, I set up a test VM and was able to reliably replicate the problem. If I kept the VM out of the Veeam backup jobs, then the Windows update history would be fine. If I included it in a backup job, then the history would be lost. But, it wouldn't happen right away. It could be a few hours after the backup. I had been hoping for something more dramatic to show support.

Based on what I've been seeing, I think it is related to the "Enable Hyper-V guest quiescence" option (and possibly having AAP off at the same time). I tend to keep AAP off so that I don't have to give Veeam admin access to all of the systems. I had both options off on the backup job that processed my DCs due to the Server 2016 compatibility issue with down-level DCs and their Windows Update history was never lost.

However, it seems that installing KB4074837 on the VMs has fixed the issue. Since installing it, all of the VMs that got it have retained the Windows Update history going forward. That's good news except KB4074837 is not available for the R1 releases of Windows Server 2008 and Windows Server 2012, and I've still got some of those in my environment.

[Mike Resseler]

Mark,

Thanks for the additional info. Did you pass this to Veeam support also?
For your other servers. Are those still under lifecycle support? Can you update them with the latest SP? Or is that not an option in your environment?

[mkaec]

I didn't get around to opening the support case because I didn't get the repro steps to where I wanted them. It's not an option to upgrade the 2008 and 2012 servers to the R2 variant (either for technical or monetary reasons). Out of the 4, only Server 2012 R2 is still in mainstream support. The rest are in extended support. But one could argue this is a Server 2016 support issue as they are listed as supported by Hyper-V 2016 and this problem doesn't occur if Server 2012 R2 is the host.

[mkaec]

I have confirmed it. The 2012 and 2008 VMs I'm backing up with Veeam B&R lose their Windows Update history some time after the backup. Any 2012 and 2008 VMs I don't back up retain their Windows Update history. If I back up with Windows Server Backup, the VMs retain their Windows Update history.

[Nick-SAC]

Hello Marc,

FWIW, I’m not seeing the this issue when using VB&R to back up 2008-R2 Guest VM’s running on a 2008-R2 Hyper-V Host, so it wouldn’t appear to be an inherent problem (at least not in 2008-R2).

I have seen the Update History disappear (routinely) after doing a System (Physical or VM) Restore from Backup (whether using VB&R or Windows Backup).

As you probably know; even in those cases where the History has disappeared, they still show up in: Control Panel > Programs and Features > View Installed Updates.

Good Luck,
Nick

[mkaec]

To reproduce you need a Server 2008 (R1) or Server 2012 (R1) VM running on a Hyper-V 2016 host. R2 VMs are fine with latest integration services and this didn't happen on our 2012 R2 hosts.

The update history disappearing is just a very visible symptom. The real problem is that updates are failing to install with errors during the scheduled patch window and we have to manually install them. Usually when we manually install them, they fail to install on the first run. Then we run the install a second time right away and they install fine. So, whatever is clearing the history is also putting the update system into an error state.

[mkaec]

I finally got all the details together for a support case. It is 03061926.

[rkmc]

Is there any update on this issue?

Seeing the same symptom. 2012 (original) VM on a 2016 host. Windows update history is missing.

[mkaec]

...Seeing the same symptom. 2012 (original) VM on a 2016 host. Windows update history is missing.

Thanks for chiming in. I was feeling a little lonely.

My support case started out a little rough. Right away the rep started out by saying it looked like a Microsoft issue and suggested that I open a case with Microsoft. There has been a little back and forth, and I have been able to prevent the case from being closed thus far.

I was asked to run the "Full Mode" steps in https://www.veeam.com/kb1980. I was actually hopeful that would replicate the problem as I could use something like that to open a case with Microsoft. But, as it is now, they would just tell me it is a Veeam problem.

The latest status is that I was provided three scripts to run to try to replicate the problem. I've been working on those with my test VM. Testing this is difficult because the problem does not manifest immediately after a backup.

[mkaec]

I've done a lot of testing and have been able to get down the repro steps.

1.Install Windows updates.
2.Reboot VM.
3.Wait 10 minutes.
4.Without logging in to the VM, run a Veeam B&R backup.

I found that if I log in to the VM after the post-install reboot, but before the Veeam backup runs, then the VM doesn't lose the Windows Update history.

[mkaec]

Veeam Support has come up with a simpler way to reproduce.

1. Install Windows updates.
2. Reboot VM.
3. Wait 10 minutes.
4. Create a production checkpoint in Hyper-V Manager or using PowerShell.

I have confirmed this will do it on my system. If I create a standard checkpoint, the Windows Update history is not erased. But, it seems there is no way to ask Veeam B&R to choose a standard checkpoint over a production checkpoint for the VMs susceptible to the problem.

I'm sure Veeam Support is quite happy to have found a way to reproduce the problem without the involvement of Veeam software. But, this is still an issue that probably affects every Veeam customer that backs up Server 2008 or Server 2012 VMs hosted on Server 2016.

[Mike Resseler]

Marc,
Thanks for this.

The difference between a standard and production checkpoint are quite large, personally, I would not be happy with standard checkpoints as they are just crash-consistent checkpoints. No app-awareness.

Now for the issue itself. Are you aware of Veeam support going to talk to MSFT? Are you going to create a ticket?

[mkaec]

I opened a case with Microsoft.

I asked the Veeam support rep about the possibility of Veeam reporting the issue to Microsoft directly because I thought Veeam would have access to better support channels than me. My ticket will start with a sub-contractor that, based on my past experiences, seems like they are more interested in closing the ticket than solving the problem.

But, he told me that Veeam's special Microsoft access is only available for API issues. Otherwise, if Veeam tried to open a ticket for a non-API issue, they'd go through a similar experience as me.

[Mike Resseler]

Thanks for letting me know!
I do hope you have success with the ticket though... Keep us informed

[Technogod]

I restored an Agent backup using B&R and my Windows Update history is missing.I'm using the latest version of all three products. Have there been any other cases of this reported?

[Mike Resseler]

Not yet that I am aware off. We also have no further information on the research with Microsoft I'm afraid.

[Nick-SAC]

Technogod

As I noted further up this thread, I have seen the Update History disappear routinely after doing a System (Physical or VM) Restore from Backup whether using VB&R or Windows Backup.

Nick

[mkaec]

I posted earlier that installing KB4074837 fixed the problem in 2012 R2 and 2008 R2. But, that KB does not look correct. I think I meant to write KB4072650. (Or Microsoft changed things on me.)

[mkaec]

With further testing, I've been able to come up with even easier repro steps:

1. Leave the VM idle for awhile with no user logged on.
2. Create a production checkpoint.
3. Notice that "C:\Windows\SoftwareDistribution\DataStore\DataStore.edb" on the VM has been deleted.

[Mike Resseler]

So it is basically the VSS writer that probably destroys that edb file... Any luck already with the support call with MSFT?

[mkaec]

I figured out that stopping the Windows Update service allows creating the checkpoint to delete the DataStore.edb file while logged in. This allowed me to run Process Monitor and catch VSSVC.exe deleting it.

[mkaec]

The case has been escalated once within Convergys. The new rep seems on board with the idea that this is a bug only Microsoft can address. He asked for my patience because apparently there are a lot of requirements to escalate further.

I was told there is a strict policy that a single case can only involve a single VM. So, it looks like the best case scenario is that a fix gets released for Server 2012, but not Server 2008. I'm afraid if I try to open a second case for my Server 2008 VM, it would get closed because Server 2008 is in extended support. (Server 2012 is in mainstream support through October.) My argument would be that I'm seeking support for Server 2016, but the VM OS being in extended support is such low hanging fruit.

[Mike Resseler]

Thanks for keeping us posted Marc. It looks like it is still a long 'support' path to go for you

[GeraldT]

Just stumbled on this post after looking for something else.

We've had the same problem and have had a call in with MS 'Premier' Support now for over 18 months.

This is purely an MS issue.

To reproduce this

1) Create, or use an existing 2012 R2 HV VM on a 2016 HV Host. (ours is Core Build 14393)
2) Log in to the VM
3) Install and run ProcMon filtered for datastore.edb
4) Navigate to C:\Windows\SoftwareDistribution\DataStore
5) Create a 'Production' checkpoint

Watch the datastore.edb file 'disappear', and vssvc.exe in ProcMon the cause of the delete.
If I remember correctly, this deletes the datastore.edb file about 90% of the time a production checkpoint is created.

Microsoft have known about vssvc.exe being the cause for well over a year, and they were able to reproduce the issue on their systems back then.
This is a 'code defect'.