Losing Windows Update History

Discussions specific to Microsoft Hyper-V hypervisor

Losing Windows Update History

Veeam Logoby mkaec » Tue Feb 13, 2018 5:28 pm

I've been having a problem for a while with Server 2012 R2 (and older) VMs in that they will regularly lose the Window Update history. The summary page will state that updates were never installed. Then if I click on " View Update History" it will be blank. If I install new updates, the summary page will show the date and time of installation and the history will list just the new updates. But, eventually, the system will revert to the previously described state.

I realized today that this is only happening with VMs I am backing up with Veeam B&R. I have some VMs that are either not backed up, or that are backed up by another application, and they are not exhibiting this problem.

One other data point is that this problem appears to have surfaced after upgrading the hosts to Windows Server 2016.

I'm using B&R 9.5.0.1536.

Is anyone else with Server 2016 hosts seeing, or not seeing, this behavior with the 2012 R2 (and older) VMs you are backing up with Veeam B&R?
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby Mike Resseler » Wed Feb 14, 2018 7:12 am

Mark,

This is the first time I have ever heard of this. When you say it reverted back to the previous state, does that mean that the updates are not installed anymore? Or are they still installed but you can't see them in the history anymore?

I did a quick google and it looks like others have suffered from this also, but I couldn't quite find a solution for it. They did state that the updates were installed, just that they weren't visible anymore in update history
Mike Resseler
Veeam Software
 
Posts: 4272
Liked: 479 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Losing Windows Update History

Veeam Logoby mkaec » Wed Feb 14, 2018 4:41 pm

I meant the status go back to the previous state. The updates stay installed, but the history goes away.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby Mike Resseler » Thu Feb 15, 2018 6:10 am

From what I can read, it happens if you use the windows update troubleshooter or when something happened to the C:\Windows\SoftwareDistribution folder. I'm pretty sure that backing up a VM with Veeam doesn't clean out that folder. Any change there are other tools that clean-out the history (for space savings for example)
Mike Resseler
Veeam Software
 
Posts: 4272
Liked: 479 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Losing Windows Update History

Veeam Logoby mkaec » Thu Feb 15, 2018 3:01 pm

Yes, I've done my share of deleting out "C:\Windows\SoftwareDistribution" on Windows 7 systems to troubleshoot Windows update problems. (But, I've only done it on a handful of VMs when I was first troubleshooting this problem.) Now that I've discovered B&R may be involved, I'll do more troubleshooting around it. If I can reliably reproduce it, I'll open a support case.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby Mike Resseler » Thu Feb 15, 2018 3:32 pm

Yes. That would be a good idea. If it is Veeam B&R we need to figure out why, but as I said, I have never seen that behavior from B&R (but then again, you never know with software ;-))
Mike Resseler
Veeam Software
 
Posts: 4272
Liked: 479 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Losing Windows Update History

Veeam Logoby mkaec » Mon Mar 12, 2018 2:53 pm

So, I set up a test VM and was able to reliably replicate the problem. If I kept the VM out of the Veeam backup jobs, then the Windows update history would be fine. If I included it in a backup job, then the history would be lost. But, it wouldn't happen right away. It could be a few hours after the backup. I had been hoping for something more dramatic to show support.

Based on what I've been seeing, I think it is related to the "Enable Hyper-V guest quiescence" option (and possibly having AAP off at the same time). I tend to keep AAP off so that I don't have to give Veeam admin access to all of the systems. I had both options off on the backup job that processed my DCs due to the Server 2016 compatibility issue with down-level DCs and their Windows Update history was never lost.

However, it seems that installing KB4074837 on the VMs has fixed the issue. Since installing it, all of the VMs that got it have retained the Windows Update history going forward. That's good news except KB4074837 is not available for the R1 releases of Windows Server 2008 and Windows Server 2012, and I've still got some of those in my environment.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby Mike Resseler » Mon Mar 12, 2018 5:36 pm

Mark,

Thanks for the additional info. Did you pass this to Veeam support also?
For your other servers. Are those still under lifecycle support? Can you update them with the latest SP? Or is that not an option in your environment?
Mike Resseler
Veeam Software
 
Posts: 4272
Liked: 479 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Losing Windows Update History

Veeam Logoby mkaec » Mon Mar 12, 2018 6:37 pm

I didn't get around to opening the support case because I didn't get the repro steps to where I wanted them. It's not an option to upgrade the 2008 and 2012 servers to the R2 variant (either for technical or monetary reasons). Out of the 4, only Server 2012 R2 is still in mainstream support. The rest are in extended support. But one could argue this is a Server 2016 support issue as they are listed as supported by Hyper-V 2016 and this problem doesn't occur if Server 2012 R2 is the host.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby mkaec » Fri Jun 15, 2018 2:11 pm

I have confirmed it. The 2012 and 2008 VMs I'm backing up with Veeam B&R lose their Windows Update history some time after the backup. Any 2012 and 2008 VMs I don't back up retain their Windows Update history. If I back up with Windows Server Backup, the VMs retain their Windows Update history.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby Nick-SAC » Mon Jun 18, 2018 1:28 pm

Hello Marc,

FWIW, I’m not seeing the this issue when using VB&R to back up 2008-R2 Guest VM’s running on a 2008-R2 Hyper-V Host, so it wouldn’t appear to be an inherent problem (at least not in 2008-R2).

I have seen the Update History disappear (routinely) after doing a System (Physical or VM) Restore from Backup (whether using VB&R or Windows Backup).

As you probably know; even in those cases where the History has disappeared, they still show up in: Control Panel > Programs and Features > View Installed Updates.

Good Luck,
Nick
Nick-SAC
Influencer
 
Posts: 21
Liked: 1 time
Joined: Fri Oct 27, 2017 5:42 pm
Full Name: Nick

Re: Losing Windows Update History

Veeam Logoby mkaec » Tue Jun 19, 2018 6:05 pm

To reproduce you need a Server 2008 (R1) or Server 2012 (R1) VM running on a Hyper-V 2016 host. R2 VMs are fine with latest integration services and this didn't happen on our 2012 R2 hosts.

The update history disappearing is just a very visible symptom. The real problem is that updates are failing to install with errors during the scheduled patch window and we have to manually install them. Usually when we manually install them, they fail to install on the first run. Then we run the install a second time right away and they install fine. So, whatever is clearing the history is also putting the update system into an error state.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby mkaec » Tue Jun 19, 2018 10:55 pm

I finally got all the details together for a support case. It is 03061926.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Losing Windows Update History

Veeam Logoby rkmc » Wed Jul 11, 2018 10:53 pm

Is there any update on this issue?

Seeing the same symptom. 2012 (original) VM on a 2016 host. Windows update history is missing.
rkmc
Lurker
 
Posts: 1
Liked: never
Joined: Wed Jul 11, 2018 10:51 pm
Full Name: Rick Kuchynka

Re: Losing Windows Update History

Veeam Logoby mkaec » Thu Jul 12, 2018 4:31 pm

rkmc wrote:...Seeing the same symptom. 2012 (original) VM on a 2016 host. Windows update history is missing.

Thanks for chiming in. I was feeling a little lonely.

My support case started out a little rough. Right away the rep started out by saying it looked like a Microsoft issue and suggested that I open a case with Microsoft. There has been a little back and forth, and I have been able to prevent the case from being closed thus far.

I was asked to run the "Full Mode" steps in https://www.veeam.com/kb1980. I was actually hopeful that would replicate the problem as I could use something like that to open a case with Microsoft. But, as it is now, they would just tell me it is a Veeam problem.

The latest status is that I was provided three scripts to run to try to replicate the problem. I've been working on those with my test VM. Testing this is difficult because the problem does not manifest immediately after a backup.
mkaec
Expert
 
Posts: 244
Liked: 54 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Google [Bot], ryanC and 10 guests