KB4013429 breaks Hyper-V 2016 backup, BSOD loop on uninstall

Hyper-V specific discussions

KB4013429 breaks Hyper-V 2016 backup, BSOD loop on uninstall

Veeam Logoby AndraZ » Sat Mar 18, 2017 12:43 am

I was just updating HyperV Windows 2016 hosts with Microsoft monthly (March) security patches and after restart Veeam backups stopped working from updated hosts. There is still one host not updated and backups from that one are working fine.
Error: Task failed. Failed to expand object ALPZVPN1. Error: [WMI] Empty result. Failed to get VM owner by ID.
Anyone else stumbled across this situation?

I've submitted a case ID #02103355
It is the same for new jobs and freshly added VMs as for old existing jobs. If VM is on updated hosts, backup won't work, and if it is moved to the not updated host, backup works.

I've tried:
Hosts and cluster were rescaned in Veeam infrastructure tab. Agents were reapplied, no errors found. All services are running. All VMs are visible in Veeam console, VM size recalculation in a job configuration is working without a problem.
Hosts and Veeam are all pretty freshly installed (in 2017). There is a possibility that just a restart caused malfunction and is not related to March's upgrades. I'm unable to find anything specific to this error online, so I don't know what else should I try. I don't want to restart the last host. It could be that even without updates it would stop backuping.

Infrastructure:
We have Veeam B&R 9.5U1 freshly installed a month ago on Windows Server 2016 on a physical host . It was just upgraded with Microsoft Updates and restarted.
We also have 3 HyperV production hosts on Windows Server 2016. Two of them were just upgraded with regular March security patches (KB3150513, KB4013429, KB4013418) and restarted.

Any pointers on how to proceed would be appreciated
Thanks and best regards
AndraZ
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Jan 31, 2017 12:40 am
Full Name: Andraz Kopac

Be aware of KB4013429 with clusters

Veeam Logoby RGijsen » Sun Mar 19, 2017 8:19 am

Hi,
I've had a ticket this week (02100004) which was about Veeam failing with WMI errors on our 2016 Datacenter Hyper-V cluster after installing KB4013429. Uninstalling that patch resulted in a BSOD bootloop, so the host had to be reinstalled. I've been able to reproduce that issue several times. On some hosts uninstalling works fine, and after uninstalling WMI / Veeam works fine again. This ticket was closed as it's clearly caused by KB4013429, so it's a MS thing initially. However, today I stumbled on this post where someone has the exact same issues after KB4013429. This also happens to be on Datacenter, not sure if that would be the common denominator here. Also, on my site it only happened on clustered nodes, we also have standalone hosts, and they are still fine after installing KB4013429.

So I just want to warn you, do proper testing before rolling out KB4013429.
RGijsen
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: KB4013429 breaks Hyper-V 2016 backup

Veeam Logoby AndraZ » Sun Mar 19, 2017 5:26 pm

Updates from support:
KB4013429 is causing this issue. I would suggest removing it, but there are reports that after removal update host might get BSOD bootloop:
https://social.technet.microsoft.com/Fo ... f=required

When asked about number of users affected and possible timeline on MS part, no specific answer was possible. Case is in research/testing status though and I hope for more information to come.

My plan is to wait a while more on a fix/workaround and use the remaining host for backup of most critical VMs, others can wait a while without Veeam backup (have also alternative backups). I don't plan uninstalling KB4013429 just yet.
AndraZ
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Jan 31, 2017 12:40 am
Full Name: Andraz Kopac

Re: KB4013429 breaks Hyper-V 2016 backup

Veeam Logoby RGijsen » Sun Mar 19, 2017 7:11 pm

Yes, same issue here. I've had ticket 02100004. As the issue clearly is the MS KB, the ticket was closed quite early. Today I've seen on Technet forums that there are more people with the same issues, so I opened up another case, just to allow Veeam to inform other users, as this might have extreme big impact. Also uninstalling KB4013429 on about half of the machines results in a BSOD loop.

see microsoft-hyper-v-f25/be-aware-of-kb4013429-with-clusters-t41788.html and https://social.technet.microsoft.com/Fo ... f=required as well.

This is potientially a very, very dangerous update.
RGijsen
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby AndraZ » Sun Mar 19, 2017 8:42 pm

We have a cluster of Windows Server 2016 Standard, so it is not just Datacenter issue.
AndraZ
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Jan 31, 2017 12:40 am
Full Name: Andraz Kopac

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby RGijsen » Tue Mar 21, 2017 1:01 pm

Veeam now has a KB for this:

https://www.veeam.com/kb2262

Bottom line; you should install this update on ALL hosts in your cluster. Usually we deliberately don't do that, in order to be able to move VM's to another host if issues arise.
RGijsen
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby AndraZ » Tue Mar 21, 2017 1:18 pm

Anyone with an updated 2016 cluster, that has KB4013429 and Veeam works OK? Ideally one that had the same issue during updates...

I don't want to loose my last backup-working host blindly trusting this KB :)
AndraZ
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Jan 31, 2017 12:40 am
Full Name: Andraz Kopac

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby Gostev » Tue Mar 21, 2017 7:26 pm

We have one in our QC lab for sure ;)
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby Alek14 » Wed Mar 22, 2017 2:08 am

We have applied MS re-issued Update yesterday: https://support.microsoft.com/en-us/help/4015438

The backups have run successfully overnight. 2016 Hyper-V Datacenter Clusters
Alek14
Novice
 
Posts: 4
Liked: never
Joined: Wed Jun 04, 2014 10:52 pm

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby Mike Resseler » Wed Mar 22, 2017 6:33 am

thanks for letting us know @alek14!

Please keep us informed if there still would be issues

Mike
Mike Resseler
Veeam Software
 
Posts: 3173
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby Gostev » Wed Mar 22, 2017 11:25 am

Hmm, it does not look like "re-issued" to me, but rather this is some new and different update? Or is it cumulative?
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby RGijsen » Wed Mar 22, 2017 12:33 pm

I took the plunge and updated all my hosts with KB4015438 and Veeam still works - kinda. One VM still gives errors: 22-3-2017 13:32:13 :: Error: [WMI] Empty result.
Failed to get information about VM config path 'f3c7ba62-e30b-48fa-acff-0fcb968229d1'.

So most of it is ok now - not all. I'll hold my breath for tonights backups and let you know.
RGijsen
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby Alek14 » Wed Mar 22, 2017 9:09 pm

What I meant was, probably the replacement for KB4013429. Cause when we built new 2016 this update was not in the list of available on MS Update but 4015438. Some call it "emergency fix" :-). Either way, two successful nights of backups
Alek14
Novice
 
Posts: 4
Liked: never
Joined: Wed Jun 04, 2014 10:52 pm

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby nmdange » Thu Mar 23, 2017 2:18 am

Looks like another cumulative update came out today: https://support.microsoft.com/en-us/help/4016635

The previous update (KB4015438) wasn't on WSUS but this one is.
nmdange
Expert
 
Posts: 194
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins

Veeam Logoby AndraZ » Sat Mar 25, 2017 4:31 pm 1 person likes this post

I've successfully upgraded the whole cluster and Veeam backup is now working fine.
I had troubles with KB4015438 on one host (the one that was not upgraded last weekend). It was failing with 0x800705b4 after 30-60 minutes of "preparing the updates". Restart didn't help and I retried 4 times (once with an "upgrade and shutdown").
I solved it by manually downloading KB4015438 for server 2016 (1.055GB in size), which installed successfully.
Thank you for feedback on the original problem and best regards
AndraZ
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Jan 31, 2017 12:40 am
Full Name: Andraz Kopac

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Yahoo [Bot] and 7 guests