KB4013429 breaks Hyper-V 2016 backup, BSOD loop on uninstall

[AndraZ]

I was just updating HyperV Windows 2016 hosts with Microsoft monthly (March) security patches and after restart Veeam backups stopped working from updated hosts. There is still one host not updated and backups from that one are working fine.
Error: Task failed. Failed to expand object ALPZVPN1. Error: [WMI] Empty result. Failed to get VM owner by ID.
Anyone else stumbled across this situation?

I've submitted a case ID #02103355
It is the same for new jobs and freshly added VMs as for old existing jobs. If VM is on updated hosts, backup won't work, and if it is moved to the not updated host, backup works.

I've tried:
Hosts and cluster were rescaned in Veeam infrastructure tab. Agents were reapplied, no errors found. All services are running. All VMs are visible in Veeam console, VM size recalculation in a job configuration is working without a problem.
Hosts and Veeam are all pretty freshly installed (in 2017). There is a possibility that just a restart caused malfunction and is not related to March's upgrades. I'm unable to find anything specific to this error online, so I don't know what else should I try. I don't want to restart the last host. It could be that even without updates it would stop backuping.

Infrastructure:
We have Veeam B&R 9.5U1 freshly installed a month ago on Windows Server 2016 on a physical host . It was just upgraded with Microsoft Updates and restarted.
We also have 3 HyperV production hosts on Windows Server 2016. Two of them were just upgraded with regular March security patches (KB3150513, KB4013429, KB4013418) and restarted.

Any pointers on how to proceed would be appreciated
Thanks and best regards

[RGijsen]

Hi,
I've had a ticket this week (02100004) which was about Veeam failing with WMI errors on our 2016 Datacenter Hyper-V cluster after installing KB4013429. Uninstalling that patch resulted in a BSOD bootloop, so the host had to be reinstalled. I've been able to reproduce that issue several times. On some hosts uninstalling works fine, and after uninstalling WMI / Veeam works fine again. This ticket was closed as it's clearly caused by KB4013429, so it's a MS thing initially. However, today I stumbled on this post where someone has the exact same issues after KB4013429. This also happens to be on Datacenter, not sure if that would be the common denominator here. Also, on my site it only happened on clustered nodes, we also have standalone hosts, and they are still fine after installing KB4013429.

So I just want to warn you, do proper testing before rolling out KB4013429.

[AndraZ]

Updates from support:

KB4013429 is causing this issue. I would suggest removing it, but there are reports that after removal update host might get BSOD bootloop:
https://social.technet.microsoft.com/Fo ... f=required

When asked about number of users affected and possible timeline on MS part, no specific answer was possible. Case is in research/testing status though and I hope for more information to come.

My plan is to wait a while more on a fix/workaround and use the remaining host for backup of most critical VMs, others can wait a while without Veeam backup (have also alternative backups). I don't plan uninstalling KB4013429 just yet.

[RGijsen]

Yes, same issue here. I've had ticket 02100004. As the issue clearly is the MS KB, the ticket was closed quite early. Today I've seen on Technet forums that there are more people with the same issues, so I opened up another case, just to allow Veeam to inform other users, as this might have extreme big impact. Also uninstalling KB4013429 on about half of the machines results in a BSOD loop.

see microsoft-hyper-v-f25/be-aware-of-kb401 ... 41788.html and https://social.technet.microsoft.com/Fo ... f=required as well.

This is potientially a very, very dangerous update.

[AndraZ]

We have a cluster of Windows Server 2016 Standard, so it is not just Datacenter issue.

[RGijsen]

Veeam now has a KB for this:

https://www.veeam.com/kb2262

Bottom line; you should install this update on ALL hosts in your cluster. Usually we deliberately don't do that, in order to be able to move VM's to another host if issues arise.

[AndraZ]

Anyone with an updated 2016 cluster, that has KB4013429 and Veeam works OK? Ideally one that had the same issue during updates...

I don't want to loose my last backup-working host blindly trusting this KB

[Gostev]

We have one in our QC lab for sure

[Alek14]

We have applied MS re-issued Update yesterday: https://support.microsoft.com/en-us/help/4015438

The backups have run successfully overnight. 2016 Hyper-V Datacenter Clusters

[Mike Resseler]

thanks for letting us know @alek14!

Please keep us informed if there still would be issues

Mike

[Gostev]

Hmm, it does not look like "re-issued" to me, but rather this is some new and different update? Or is it cumulative?

[RGijsen]

I took the plunge and updated all my hosts with KB4015438 and Veeam still works - kinda. One VM still gives errors: 22-3-2017 13:32:13 :: Error: [WMI] Empty result.
Failed to get information about VM config path 'f3c7ba62-e30b-48fa-acff-0fcb968229d1'.

So most of it is ok now - not all. I'll hold my breath for tonights backups and let you know.

[Alek14]

What I meant was, probably the replacement for KB4013429. Cause when we built new 2016 this update was not in the list of available on MS Update but 4015438. Some call it "emergency fix" . Either way, two successful nights of backups

[nmdange]

Looks like another cumulative update came out today: https://support.microsoft.com/en-us/help/4016635

The previous update (KB4015438) wasn't on WSUS but this one is.

[AndraZ]

I've successfully upgraded the whole cluster and Veeam backup is now working fine.
I had troubles with KB4015438 on one host (the one that was not upgraded last weekend). It was failing with 0x800705b4 after 30-60 minutes of "preparing the updates". Restart didn't help and I retried 4 times (once with an "upgrade and shutdown").
I solved it by manually downloading KB4015438 for server 2016 (1.055GB in size), which installed successfully.
Thank you for feedback on the original problem and best regards

[Mike Resseler]

Andraz,

Thanks for the additional input. One note for all of you: The reason why the manual download is so large is because it contains all updates from GA till now (so be prepared, it will only get bigger ). If you do it through windows update, it actually looks what you already have and then only downloads the bits needed for your server, so size will be less. In the manual mode that unfortunately doesn't work

So for those who want to do the manual method, keep that in mind so you don't download it manually on each server

[RGijsen]

Not entirely true; MS is working on delta-updates for 'standalone' as well. For example this one has them already available: http://www.catalog.update.microsoft.com ... =KB4013429. Still around 700MB which for some is a pain in the @ss but still better than 1GB+. Personnaly I not in favour of this constant cumulative update system. Every month something breaks. Then it's AppV. Then it's WMI. Then it's MS Dynamics. It's nearly always their own software that breaks by the way in our environment. In t he past we could uninstall the specific patch and keep secure with the rest. Now it's all or nothing. I can also see the benefit of it, but the drawbacks weigh more for me.

[mkaec]

...Personnaly I not in favour of this constant cumulative update system. Every month something breaks...

I think the cumulative update approach is far superior to what they were doing with Windows 7/Windows Server 2008 R2. They would diagnose and fix the bug and then release it as a hotfix only. That means many systems out there are running with numerous known bugs.

Here is an example: The task scheduler has a bug in which the "Run task as soon as possible after a scheduled start is missed" option doesn't work. This would be a good fix to release into Windows Update.

https://support.microsoft.com/en-us/hel ... er-2008-r2

This situation was created because the support model used to be that they released hotfixes for the immediate need and then later released a service pack containing all the fixes. Microsoft stopped releasing services packs, but didn't correct the support model to actually deploy the bug fixes to regular systems until Windows 8 was released.

[Mike Resseler]

@RGijsen,

I didn't saw that delta update yet. That is interesting so thanks for that. I'm going to look at it. 700 MB is still a lot so I am not sure what is exactly in there.

@Mark

I agree with you, and it is just a matter of getting used to it (And yes @Rgijsen, MSFT does need to step up its game in releasing better updates but the thought is good). I still see too many 2012R2 hosts out there where the admin claims that they are fully up-to-date but backup fails. Simply checking if the hotfixes are installed mostly solves the issue, but those hotfixes never made it to Windows Update so it is a manual process...

[RGijsen]

Sure, I agree the idea behind cumulative updates are good, but then they shouldn't break our environment every single month. That's my main complaint. Every single month something breaks. With KB3216755 it was SQL. With KB4013429 it was Veeam and MS Dynamics CRM. Not fixed with the next KB4015438. Dynamics (an MS product) was only fixed in KB4016635. That's another 2GB of updates further. Now I don't mind bandwidth or anything, but these cumulative updates take ages to install, and after that TiWorker eats one cpu for anything between 0 and two hours or so. And apart from one or two of the cumulative fixed we've had something break every single month, and usually it's MS own programs that stop working, like AppV.

Now while I like the idea of 'when they now fix it, at least the fix ends up on all machines'. True, but we are a very small company. No MS Premier account. That means when I log a case with MS, we first have to answer a buttload of useless questions over and over again and upload several gigabytes of logfiles. Maybe the issue then gets accepted, sometimes they even state the businesscase is too small to fix it for us. That means even if the issue is accepted, it takes months before a fix is released. We have several MS hotfixes on our account, but most of them took 9 - 15 months to create.
So yeah, for us it was definitely better to be able to uninstall a specific patch, and keep the rest updated.

Now ontopic again

[bjackson@agi.com]

Not to beat a dead horse, but I wanted to share that I've patched all of my 2016 cluster nodes with KB4013429. No problems performing backups initially.