Host-based backup of Microsoft Hyper-V VMs.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

GeraldT,

Did Microsoft share with you a KB number or some other internal ID that documents the code defect? The support group that has my case is claiming they can't reproduce. If I could provide them with a number to prove the issue is real, that would be a big help.

I'm surprised that you noted the VM was running 2012 R2. Is that still happening to you today? After KB4072650 was installed in my environment, the problem stopped occurring with 2012 R2 and 2008 R2 VMs. But, KB4072650 doesn't update the integration services for the R1 2012 and 2008 operating systems. I've been trying to get Microsoft to release the updated integration services for the R1 operating systems.
GeraldT
Lurker
Posts: 2
Liked: 1 time
Joined: Nov 13, 2018 11:20 am
Full Name: Gerald T
Contact:

Re: Losing Windows Update History

Post by GeraldT »

mkaec

I don't have any internal IDs or documents for a code defect.

Our version of IS is 6.3.9600.18692 in the main.
I've noted that KB as previously did not know about that, but installed that on a test server yesterday and find that the IS version is now 6.3.9600.18907 - this I find in the registry.

Currently I see 14 servers (out of 700+) that currently have no update history, (13 x 2012 R2, and 1 2008 R2) and I see that the datastore.edb creation date is within the last few days.


Looking back on my 100+ emails with MS 'Premier' Support, I see an email from Oct 2018.
'We are working with PG on this issue to file a formal Code Defect which leaves us in a position to formally complete all documentation to file a code defect.'
'I would like to inform that the support case will not be charged to your contract since we are working on an identified code defect and hence any time accumulated on this case will be free of charge.'
The next email from them was Feb this year out of the blue.
'We are continually chasing this with our product group for a fix for Server 2016 v1607'
Then nothing until July when a new engineer has taken this up...
"There was a hypothesis..."
" ...another course of actions is to consider a systematic repro environment to debug."

So they seemed to be going backward here as time goes on.

I wonder if they have 'formally' classed this as a code defect, even though our support case costs (60+ hours charged by August 2017) was FOC.

I'm in training for the next 3 days, so will get back later in testing the production snapshots, but it would appear they fixed it in that recent KB you stated.
We have only 1 2012 non R2 and 2 or 3 non R2 2008s btw.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

I recently started backing up my test/dev environment with Veeam Free and I noticed this problem is occurring on the Windows 7 and windows 8.1 VMs. That's because the version of the integration services that fixes the bug was released for only Windows Server 2008 R2 and Windows Server 2012 R2 specifically, but not Windows 7 or Windows 8.1. Fun...
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr »

Hi everyone,

Could you please check under "Programs and Features", "Installed Updates" - are the previously installed updates showing here after restore? If they are in place, then you just lost the record of which updates were installed using a given user account.

Also, I've performed some digging and found out that this may be a by design behavior of Microsoft VSS. There are some comments on the second page stating the same.

Anyway, we definitely recommend everyone experiencing the same symptoms to open a support case with Microsoft. This will help to bring more attention to this issue.

Thanks
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

Yes, the updates are still in Programs and Features. The main problem isn't that the history is missing, but that this condition prevents updates from automatically installing correctly in the future. Each month, the updates fail to install with an error and then I need to manually install them on each server.

"design behavior" is a bit of semantics. The behavior probably was coded on purpose. But, it causes a problem. So, I consider it a defect. And it was removed from later releases of the integration services. The problem is that the new version has only been released for two versions of Windows and there are still several others that experience the problem.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr »

Hi Marc,

Thank you for the comments. Definitely makes sense and I think this is a very unfortunate situation, especially keeping in mind the EoL dates of the affected products.

From Veeam perspective, we do keep an eye on that situation and will be happy to jump in and assist when we are able to do so.

Regards,
Fedor
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

I think Veeam really brings the issue out. It causes production checkpoints to be created every time a backup is run (which is usually daily). Administrators rarely manually create production checkpoints.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr »

Well, production checkpoints are only used by B&R when backing up Hyper-V 2016 VMs since it was introduced only with that release, and Microsoft recommends using production checkpoints for production workloads. Moreover, it's a default setting. Standard checkpoints can be only useful if you are backing up applications that are not VSS-aware. Btw, could you please let us know what are the hidden reasons for prioritizing standard checkpoints over production ones in Hyper-V 2016 in your case?
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

Standard checkpoints don't causes issues with Windows Update. That would be the main reason to use them in Hyper-V 2016, as a work-around for the present issue.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr »

Hi Marc,

Since Windows Update is lost when VSS is used (as per this thread) you may try disabling application processing (just choose the 3rd option mentioned on the screenshot by the link). VSS will not be used in that case.

Please be careful with this option since the backup will not be app-consistent by default with such configuration, however, you may still use Scripts with this configuration to achieve consistency. It should help you to do the same you are able to do using Standard checkpoints.

BR,
Fedor
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

AAP is disabled, but I have Guest Quiescence turned on and think that is causing the production checkpoints to be used.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

I realized that the deletion of the Windows Update history is not actually the cause of the problem I'm trying to solve. When the production checkpoint is created, VSSVC.exe is also clearing out the Download folder. Even if Windows Update has the DataStore.edb file locked, those files still get deleted.

It's odd that VSSVC.exe is doing the deleting when I think the bug is actually in vmicvss. I don't know enough about VSS to know what vmicvss could be doing that makes VSSVC.exe want to empty out the DataStore and Download folders.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Losing Windows Update History

Post by wishr »

B&R utilizes only "production checkpoints" Hyper-V feature when backing up Hyper-V 2016 VMs, however, you can "emulate" Standard checkpoints creation (VSS-less) using the suggested method (see above). With this configuration, VSS should not touch both DataStore.edb (not sure about the Download folder, though) because VSS won't be used at all. This is the same how Standard checkpoints work.

Guest Quiescence also relies on VSS and is disabled by default. If you'd like to fully "emulate" standard checkpoints you should disable it, but please keep in mind the backwash. If the associated risks are not acceptable then standard checkpoints do not fit your use-case (and in most situations, they do not fit production use-cases).
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec »

GeraldT wrote: Nov 13, 2018 12:17 pm ...
We've had the same problem and have had a call in with MS 'Premier' Support now for over 18 months.
Geraldt,

I'm curious if you are dealing with actual Microsoft support, or contracted support. All the contractor email addresses start with "v-". For example:

v-johnsmith@microsoft.com
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » 2 people like this post

I wanted to provide an update on the situation.

Right at the start, the response from Microsoft's support was that they were unable to reproduce the problem. It seemed like they were going to close the case based on this. Then the Veeam support rep saved the day. At my request, he emailed into the case stating that he had independently reproduced the issue in Veeam's environment. One of the systems I reproduced the bug on didn't have any Veeam software at all. So, even Veeam software couldn't be claimed to be a common variable. Microsoft's support never acknowledged being able to reproduce the bug, but they did eventually escalate the case to someone that acknowledged its existence. In fact, the conversation started to note that the bug was in vmbus.sys.

Then the messaging changed to telling me that support for Windows Server 2012 was over. I pointed out that the public documentation makes no difference in the support dates between R2 and the original release. They are identical. And I pointed out that my case submitted while Server 2012 was still in mainstream support, grandfathering it in to mainstream support. And I pointed out that the Hyper-V documentation notes that the R1 releases are supported guests. This is really a Hyper-V 2016 issue and should fall under Server 2016 mainstream support.

They maintained the notion that support for Server 2012 was over and encouraged me to upgrade the systems to R2. Everyone I had been dealing with had been a contractor. Sure that it was just the contractor trying to avoid a hit on their stats from escalating a case out, I started asking for the case to be transferred to someone who did not have the "v-" at the start of the email address. Then they showed me a quote from an email from a Microsoft technical adviser that did not have the "v-". The email stated "We have stopped developing and updating vmbus.sys(integration Services) for Windows 2008 /2012. So any possible fix would be to upgrade to 2012 R2." It seems that Microsoft's unofficial policy has been to abandon the R1 releases contrary to the published support policy.

Given what the technical advisor had been telling them, the contractor kept pushing me to upgrade to R2. I kept explaining that I only had licenses for R1 and that it would not be legal for me to just upgrade the systems. If I was not aware of the license situation I could have gotten into a lot of trouble. It would have been easy for me to use the R2 VL key to upgrade the licenses. Then the next time a SAM audit rolled around, I'd find myself arguing that the use was legal because Microsoft support authorized it. I asked for the licenses to be officially upgraded so that I could follow through with what they wanted me to do, but they would not do it. Yeah, that would have been a fun audit.

I kept on that support for the R1 releases was publicly no different than the R2 releases, that my case being submitted while the OS was in mainstream support offering it grandfathered status, and that this is really a Hyper-V 2016 issue. Eventually, they agreed to submit a bug report to their LCS database. They gave me an LCS number that I could not use to look anything up myself. When I asked how to check the status, I was told not to worry about it, that they would contact me if there was any news. "Don't call us. We'll call you." I figure this was just another way of getting rid of me. I've had the experience before of vendors submitting bugs and enhancement requests into a database with no intention of acting on them.

The whole thing was super frustrating. My original request to Microsoft was essentially "I'm experiencing a bug. You have already fixed it in a product that uses the same identical binaries. Please provide an installer that will install those binaries on the product I'm using." And they wouldn't do it. They violated their published support policy and informally stopped supporting the product ahead of schedule.

Then something unexpected happened. I was recently on a Server 2012 R2 host and I noticed it stating that an upgrade was required for the integration services for the VM. I installed the update and that brought the integration services on the VM up to version 6.3.9600.19456. I took the vmguest.iso over to a Server 2016 host that has a Server 2012 host on it, mounted it, and installed the update in that VM. That upgraded the integration services there. And the bug is fixed. I guess Microsoft had a reason to release an update for Server 2012 R2 hosts and just compiled the latest source code. The new integration services are not coming down through Windows Update like they are supposed to when the host is Server 2016. So, everyone out there experiencing this problem will still experience it. But at least the fix is available somehow.

I reached out to the support rep several times asking if the LCS case shows the bug fix has been released. He's told me a couple of times that he would look into it, but hasn't gotten back to me in a month.

So, if you're using Veeam to back up Window Server 2008, Windows Server 2012, Windows 7, or Windows 8 VMs, I highly recommend you retrieve a copy of vmguest.iso from a fully patched Windows Server 2012 R2 hyper-v host. That'll fix this Windows Update bug for those VMs. Even if you're not using Veeam to backup those VMs, the fix will allow you to use production checkpoints without damaging the VMs.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Losing Windows Update History

Post by mkaec » 1 person likes this post

If you don't want to read the prior message, here is the TLDR version:

If you're using Veeam to back up Window Server 2008, Windows Server 2012, Windows 7, or Windows 8 VMs, I highly recommend you retrieve a copy of vmguest.iso from a fully patched Windows Server 2012 R2 hyper-v host. That'll fix this Windows Update bug for those VMs. Even if you're not using Veeam to backup those VMs, the fix will allow you to use production checkpoints without damaging the VMs.
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests