Discussions specific to the Microsoft Hyper-V hypervisor
mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Nov 13, 2018 7:23 pm

GeraldT,

Did Microsoft share with you a KB number or some other internal ID that documents the code defect? The support group that has my case is claiming they can't reproduce. If I could provide them with a number to prove the issue is real, that would be a big help.

I'm surprised that you noted the VM was running 2012 R2. Is that still happening to you today? After KB4072650 was installed in my environment, the problem stopped occurring with 2012 R2 and 2008 R2 VMs. But, KB4072650 doesn't update the integration services for the R1 2012 and 2008 operating systems. I've been trying to get Microsoft to release the updated integration services for the R1 operating systems.

GeraldT
Lurker
Posts: 2
Liked: 1 time
Joined: Nov 13, 2018 11:20 am
Full Name: Gerald T
Contact:

Re: Losing Windows Update History

Post by GeraldT » Nov 14, 2018 8:49 am

mkaec

I don't have any internal IDs or documents for a code defect.

Our version of IS is 6.3.9600.18692 in the main.
I've noted that KB as previously did not know about that, but installed that on a test server yesterday and find that the IS version is now 6.3.9600.18907 - this I find in the registry.

Currently I see 14 servers (out of 700+) that currently have no update history, (13 x 2012 R2, and 1 2008 R2) and I see that the datastore.edb creation date is within the last few days.


Looking back on my 100+ emails with MS 'Premier' Support, I see an email from Oct 2018.
'We are working with PG on this issue to file a formal Code Defect which leaves us in a position to formally complete all documentation to file a code defect.'
'I would like to inform that the support case will not be charged to your contract since we are working on an identified code defect and hence any time accumulated on this case will be free of charge.'
The next email from them was Feb this year out of the blue.
'We are continually chasing this with our product group for a fix for Server 2016 v1607'
Then nothing until July when a new engineer has taken this up...
"There was a hypothesis..."
" ...another course of actions is to consider a systematic repro environment to debug."

So they seemed to be going backward here as time goes on.

I wonder if they have 'formally' classed this as a code defect, even though our support case costs (60+ hours charged by August 2017) was FOC.

I'm in training for the next 3 days, so will get back later in testing the production snapshots, but it would appear they fixed it in that recent KB you stated.
We have only 1 2012 non R2 and 2 or 3 non R2 2008s btw.

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Dec 21, 2018 10:09 pm

I recently started backing up my test/dev environment with Veeam Free and I noticed this problem is occurring on the Windows 7 and windows 8.1 VMs. That's because the version of the integration services that fixes the bug was released for only Windows Server 2008 R2 and Windows Server 2012 R2 specifically, but not Windows 7 or Windows 8.1. Fun...

wishr
Veeam Software
Posts: 398
Liked: 44 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr » Dec 26, 2018 12:01 pm

Hi everyone,

Could you please check under "Programs and Features", "Installed Updates" - are the previously installed updates showing here after restore? If they are in place, then you just lost the record of which updates were installed using a given user account.

Also, I've performed some digging and found out that this may be a by design behavior of Microsoft VSS. There are some comments on the second page stating the same.

Anyway, we definitely recommend everyone experiencing the same symptoms to open a support case with Microsoft. This will help to bring more attention to this issue.

Thanks

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Dec 26, 2018 3:30 pm

Yes, the updates are still in Programs and Features. The main problem isn't that the history is missing, but that this condition prevents updates from automatically installing correctly in the future. Each month, the updates fail to install with an error and then I need to manually install them on each server.

"design behavior" is a bit of semantics. The behavior probably was coded on purpose. But, it causes a problem. So, I consider it a defect. And it was removed from later releases of the integration services. The problem is that the new version has only been released for two versions of Windows and there are still several others that experience the problem.

wishr
Veeam Software
Posts: 398
Liked: 44 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr » Dec 26, 2018 3:56 pm

Hi Marc,

Thank you for the comments. Definitely makes sense and I think this is a very unfortunate situation, especially keeping in mind the EoL dates of the affected products.

From Veeam perspective, we do keep an eye on that situation and will be happy to jump in and assist when we are able to do so.

Regards,
Fedor

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Dec 26, 2018 4:02 pm

I think Veeam really brings the issue out. It causes production checkpoints to be created every time a backup is run (which is usually daily). Administrators rarely manually create production checkpoints.

wishr
Veeam Software
Posts: 398
Liked: 44 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr » Dec 26, 2018 4:18 pm

Well, production checkpoints are only used by B&R when backing up Hyper-V 2016 VMs since it was introduced only with that release, and Microsoft recommends using production checkpoints for production workloads. Moreover, it's a default setting. Standard checkpoints can be only useful if you are backing up applications that are not VSS-aware. Btw, could you please let us know what are the hidden reasons for prioritizing standard checkpoints over production ones in Hyper-V 2016 in your case?

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Jan 15, 2019 9:18 pm

Standard checkpoints don't causes issues with Windows Update. That would be the main reason to use them in Hyper-V 2016, as a work-around for the present issue.

wishr
Veeam Software
Posts: 398
Liked: 44 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr » Jan 16, 2019 12:59 pm

Hi Marc,

Since Windows Update is lost when VSS is used (as per this thread) you may try disabling application processing (just choose the 3rd option mentioned on the screenshot by the link). VSS will not be used in that case.

Please be careful with this option since the backup will not be app-consistent by default with such configuration, however, you may still use Scripts with this configuration to achieve consistency. It should help you to do the same you are able to do using Standard checkpoints.

BR,
Fedor

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Jan 16, 2019 2:21 pm

AAP is disabled, but I have Guest Quiescence turned on and think that is causing the production checkpoints to be used.

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Jan 16, 2019 2:25 pm

I realized that the deletion of the Windows Update history is not actually the cause of the problem I'm trying to solve. When the production checkpoint is created, VSSVC.exe is also clearing out the Download folder. Even if Windows Update has the DataStore.edb file locked, those files still get deleted.

It's odd that VSSVC.exe is doing the deleting when I think the bug is actually in vmicvss. I don't know enough about VSS to know what vmicvss could be doing that makes VSSVC.exe want to empty out the DataStore and Download folders.

wishr
Veeam Software
Posts: 398
Liked: 44 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr » Jan 16, 2019 3:53 pm

B&R utilizes only "production checkpoints" Hyper-V feature when backing up Hyper-V 2016 VMs, however, you can "emulate" Standard checkpoints creation (VSS-less) using the suggested method (see above). With this configuration, VSS should not touch both DataStore.edb (not sure about the Download folder, though) because VSS won't be used at all. This is the same how Standard checkpoints work.

Guest Quiescence also relies on VSS and is disabled by default. If you'd like to fully "emulate" standard checkpoints you should disable it, but please keep in mind the backwash. If the associated risks are not acceptable then standard checkpoints do not fit your use-case (and in most situations, they do not fit production use-cases).

mkaec
Expert
Posts: 292
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » Jan 21, 2019 8:12 pm

GeraldT wrote:
Nov 13, 2018 12:17 pm
...
We've had the same problem and have had a call in with MS 'Premier' Support now for over 18 months.
Geraldt,

I'm curious if you are dealing with actual Microsoft support, or contracted support. All the contractor email addresses start with "v-". For example:

v-johnsmith@microsoft.com

Post Reply

Who is online

Users browsing this forum: ZTSDave and 8 guests