-
- Influencer
- Posts: 12
- Liked: 1 time
- Joined: Jan 31, 2017 12:40 am
- Full Name: Andraz Kopac
- Contact:
KB4013429 breaks Hyper-V 2016 backup, BSOD loop on uninstall
I was just updating HyperV Windows 2016 hosts with Microsoft monthly (March) security patches and after restart Veeam backups stopped working from updated hosts. There is still one host not updated and backups from that one are working fine.
Error: Task failed. Failed to expand object ALPZVPN1. Error: [WMI] Empty result. Failed to get VM owner by ID.
Anyone else stumbled across this situation?
I've submitted a case ID #02103355
It is the same for new jobs and freshly added VMs as for old existing jobs. If VM is on updated hosts, backup won't work, and if it is moved to the not updated host, backup works.
I've tried:
Hosts and cluster were rescaned in Veeam infrastructure tab. Agents were reapplied, no errors found. All services are running. All VMs are visible in Veeam console, VM size recalculation in a job configuration is working without a problem.
Hosts and Veeam are all pretty freshly installed (in 2017). There is a possibility that just a restart caused malfunction and is not related to March's upgrades. I'm unable to find anything specific to this error online, so I don't know what else should I try. I don't want to restart the last host. It could be that even without updates it would stop backuping.
Infrastructure:
We have Veeam B&R 9.5U1 freshly installed a month ago on Windows Server 2016 on a physical host . It was just upgraded with Microsoft Updates and restarted.
We also have 3 HyperV production hosts on Windows Server 2016. Two of them were just upgraded with regular March security patches (KB3150513, KB4013429, KB4013418) and restarted.
Any pointers on how to proceed would be appreciated
Thanks and best regards
Error: Task failed. Failed to expand object ALPZVPN1. Error: [WMI] Empty result. Failed to get VM owner by ID.
Anyone else stumbled across this situation?
I've submitted a case ID #02103355
It is the same for new jobs and freshly added VMs as for old existing jobs. If VM is on updated hosts, backup won't work, and if it is moved to the not updated host, backup works.
I've tried:
Hosts and cluster were rescaned in Veeam infrastructure tab. Agents were reapplied, no errors found. All services are running. All VMs are visible in Veeam console, VM size recalculation in a job configuration is working without a problem.
Hosts and Veeam are all pretty freshly installed (in 2017). There is a possibility that just a restart caused malfunction and is not related to March's upgrades. I'm unable to find anything specific to this error online, so I don't know what else should I try. I don't want to restart the last host. It could be that even without updates it would stop backuping.
Infrastructure:
We have Veeam B&R 9.5U1 freshly installed a month ago on Windows Server 2016 on a physical host . It was just upgraded with Microsoft Updates and restarted.
We also have 3 HyperV production hosts on Windows Server 2016. Two of them were just upgraded with regular March security patches (KB3150513, KB4013429, KB4013418) and restarted.
Any pointers on how to proceed would be appreciated
Thanks and best regards
-
- Expert
- Posts: 125
- Liked: 27 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Be aware of KB4013429 with clusters
Hi,
I've had a ticket this week (02100004) which was about Veeam failing with WMI errors on our 2016 Datacenter Hyper-V cluster after installing KB4013429. Uninstalling that patch resulted in a BSOD bootloop, so the host had to be reinstalled. I've been able to reproduce that issue several times. On some hosts uninstalling works fine, and after uninstalling WMI / Veeam works fine again. This ticket was closed as it's clearly caused by KB4013429, so it's a MS thing initially. However, today I stumbled on this post where someone has the exact same issues after KB4013429. This also happens to be on Datacenter, not sure if that would be the common denominator here. Also, on my site it only happened on clustered nodes, we also have standalone hosts, and they are still fine after installing KB4013429.
So I just want to warn you, do proper testing before rolling out KB4013429.
I've had a ticket this week (02100004) which was about Veeam failing with WMI errors on our 2016 Datacenter Hyper-V cluster after installing KB4013429. Uninstalling that patch resulted in a BSOD bootloop, so the host had to be reinstalled. I've been able to reproduce that issue several times. On some hosts uninstalling works fine, and after uninstalling WMI / Veeam works fine again. This ticket was closed as it's clearly caused by KB4013429, so it's a MS thing initially. However, today I stumbled on this post where someone has the exact same issues after KB4013429. This also happens to be on Datacenter, not sure if that would be the common denominator here. Also, on my site it only happened on clustered nodes, we also have standalone hosts, and they are still fine after installing KB4013429.
So I just want to warn you, do proper testing before rolling out KB4013429.
-
- Influencer
- Posts: 12
- Liked: 1 time
- Joined: Jan 31, 2017 12:40 am
- Full Name: Andraz Kopac
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup
Updates from support:
My plan is to wait a while more on a fix/workaround and use the remaining host for backup of most critical VMs, others can wait a while without Veeam backup (have also alternative backups). I don't plan uninstalling KB4013429 just yet.
When asked about number of users affected and possible timeline on MS part, no specific answer was possible. Case is in research/testing status though and I hope for more information to come.KB4013429 is causing this issue. I would suggest removing it, but there are reports that after removal update host might get BSOD bootloop:
https://social.technet.microsoft.com/Fo ... f=required
My plan is to wait a while more on a fix/workaround and use the remaining host for backup of most critical VMs, others can wait a while without Veeam backup (have also alternative backups). I don't plan uninstalling KB4013429 just yet.
-
- Expert
- Posts: 125
- Liked: 27 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup
Yes, same issue here. I've had ticket 02100004. As the issue clearly is the MS KB, the ticket was closed quite early. Today I've seen on Technet forums that there are more people with the same issues, so I opened up another case, just to allow Veeam to inform other users, as this might have extreme big impact. Also uninstalling KB4013429 on about half of the machines results in a BSOD loop.
see microsoft-hyper-v-f25/be-aware-of-kb401 ... 41788.html and https://social.technet.microsoft.com/Fo ... f=required as well.
This is potientially a very, very dangerous update.
see microsoft-hyper-v-f25/be-aware-of-kb401 ... 41788.html and https://social.technet.microsoft.com/Fo ... f=required as well.
This is potientially a very, very dangerous update.
-
- Influencer
- Posts: 12
- Liked: 1 time
- Joined: Jan 31, 2017 12:40 am
- Full Name: Andraz Kopac
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
We have a cluster of Windows Server 2016 Standard, so it is not just Datacenter issue.
-
- Expert
- Posts: 125
- Liked: 27 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Veeam now has a KB for this:
https://www.veeam.com/kb2262
Bottom line; you should install this update on ALL hosts in your cluster. Usually we deliberately don't do that, in order to be able to move VM's to another host if issues arise.
https://www.veeam.com/kb2262
Bottom line; you should install this update on ALL hosts in your cluster. Usually we deliberately don't do that, in order to be able to move VM's to another host if issues arise.
-
- Influencer
- Posts: 12
- Liked: 1 time
- Joined: Jan 31, 2017 12:40 am
- Full Name: Andraz Kopac
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Anyone with an updated 2016 cluster, that has KB4013429 and Veeam works OK? Ideally one that had the same issue during updates...
I don't want to loose my last backup-working host blindly trusting this KB
I don't want to loose my last backup-working host blindly trusting this KB
-
- Chief Product Officer
- Posts: 31780
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
We have one in our QC lab for sure
-
- Novice
- Posts: 4
- Liked: never
- Joined: Jun 04, 2014 10:52 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
We have applied MS re-issued Update yesterday: https://support.microsoft.com/en-us/help/4015438
The backups have run successfully overnight. 2016 Hyper-V Datacenter Clusters
The backups have run successfully overnight. 2016 Hyper-V Datacenter Clusters
-
- Product Manager
- Posts: 8189
- Liked: 1320 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
thanks for letting us know @alek14!
Please keep us informed if there still would be issues
Mike
Please keep us informed if there still would be issues
Mike
-
- Chief Product Officer
- Posts: 31780
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Hmm, it does not look like "re-issued" to me, but rather this is some new and different update? Or is it cumulative?
-
- Expert
- Posts: 125
- Liked: 27 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
I took the plunge and updated all my hosts with KB4015438 and Veeam still works - kinda. One VM still gives errors: 22-3-2017 13:32:13 :: Error: [WMI] Empty result.
Failed to get information about VM config path 'f3c7ba62-e30b-48fa-acff-0fcb968229d1'.
So most of it is ok now - not all. I'll hold my breath for tonights backups and let you know.
Failed to get information about VM config path 'f3c7ba62-e30b-48fa-acff-0fcb968229d1'.
So most of it is ok now - not all. I'll hold my breath for tonights backups and let you know.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Jun 04, 2014 10:52 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
What I meant was, probably the replacement for KB4013429. Cause when we built new 2016 this update was not in the list of available on MS Update but 4015438. Some call it "emergency fix" . Either way, two successful nights of backups
-
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Looks like another cumulative update came out today: https://support.microsoft.com/en-us/help/4016635
The previous update (KB4015438) wasn't on WSUS but this one is.
The previous update (KB4015438) wasn't on WSUS but this one is.
-
- Influencer
- Posts: 12
- Liked: 1 time
- Joined: Jan 31, 2017 12:40 am
- Full Name: Andraz Kopac
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
I've successfully upgraded the whole cluster and Veeam backup is now working fine.
I had troubles with KB4015438 on one host (the one that was not upgraded last weekend). It was failing with 0x800705b4 after 30-60 minutes of "preparing the updates". Restart didn't help and I retried 4 times (once with an "upgrade and shutdown").
I solved it by manually downloading KB4015438 for server 2016 (1.055GB in size), which installed successfully.
Thank you for feedback on the original problem and best regards
I had troubles with KB4015438 on one host (the one that was not upgraded last weekend). It was failing with 0x800705b4 after 30-60 minutes of "preparing the updates". Restart didn't help and I retried 4 times (once with an "upgrade and shutdown").
I solved it by manually downloading KB4015438 for server 2016 (1.055GB in size), which installed successfully.
Thank you for feedback on the original problem and best regards
-
- Product Manager
- Posts: 8189
- Liked: 1320 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Andraz,
Thanks for the additional input. One note for all of you: The reason why the manual download is so large is because it contains all updates from GA till now (so be prepared, it will only get bigger ). If you do it through windows update, it actually looks what you already have and then only downloads the bits needed for your server, so size will be less. In the manual mode that unfortunately doesn't work
So for those who want to do the manual method, keep that in mind so you don't download it manually on each server
Thanks for the additional input. One note for all of you: The reason why the manual download is so large is because it contains all updates from GA till now (so be prepared, it will only get bigger ). If you do it through windows update, it actually looks what you already have and then only downloads the bits needed for your server, so size will be less. In the manual mode that unfortunately doesn't work
So for those who want to do the manual method, keep that in mind so you don't download it manually on each server
-
- Expert
- Posts: 125
- Liked: 27 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Not entirely true; MS is working on delta-updates for 'standalone' as well. For example this one has them already available: http://www.catalog.update.microsoft.com ... =KB4013429. Still around 700MB which for some is a pain in the @ss but still better than 1GB+. Personnaly I not in favour of this constant cumulative update system. Every month something breaks. Then it's AppV. Then it's WMI. Then it's MS Dynamics. It's nearly always their own software that breaks by the way in our environment. In t he past we could uninstall the specific patch and keep secure with the rest. Now it's all or nothing. I can also see the benefit of it, but the drawbacks weigh more for me.
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
I think the cumulative update approach is far superior to what they were doing with Windows 7/Windows Server 2008 R2. They would diagnose and fix the bug and then release it as a hotfix only. That means many systems out there are running with numerous known bugs.RGijsen wrote:...Personnaly I not in favour of this constant cumulative update system. Every month something breaks...
Here is an example: The task scheduler has a bug in which the "Run task as soon as possible after a scheduled start is missed" option doesn't work. This would be a good fix to release into Windows Update.
https://support.microsoft.com/en-us/hel ... er-2008-r2
This situation was created because the support model used to be that they released hotfixes for the immediate need and then later released a service pack containing all the fixes. Microsoft stopped releasing services packs, but didn't correct the support model to actually deploy the bug fixes to regular systems until Windows 8 was released.
-
- Product Manager
- Posts: 8189
- Liked: 1320 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
@RGijsen,
I didn't saw that delta update yet. That is interesting so thanks for that. I'm going to look at it. 700 MB is still a lot so I am not sure what is exactly in there.
@Mark
I agree with you, and it is just a matter of getting used to it (And yes @Rgijsen, MSFT does need to step up its game in releasing better updates but the thought is good). I still see too many 2012R2 hosts out there where the admin claims that they are fully up-to-date but backup fails. Simply checking if the hotfixes are installed mostly solves the issue, but those hotfixes never made it to Windows Update so it is a manual process...
I didn't saw that delta update yet. That is interesting so thanks for that. I'm going to look at it. 700 MB is still a lot so I am not sure what is exactly in there.
@Mark
I agree with you, and it is just a matter of getting used to it (And yes @Rgijsen, MSFT does need to step up its game in releasing better updates but the thought is good). I still see too many 2012R2 hosts out there where the admin claims that they are fully up-to-date but backup fails. Simply checking if the hotfixes are installed mostly solves the issue, but those hotfixes never made it to Windows Update so it is a manual process...
-
- Expert
- Posts: 125
- Liked: 27 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Sure, I agree the idea behind cumulative updates are good, but then they shouldn't break our environment every single month. That's my main complaint. Every single month something breaks. With KB3216755 it was SQL. With KB4013429 it was Veeam and MS Dynamics CRM. Not fixed with the next KB4015438. Dynamics (an MS product) was only fixed in KB4016635. That's another 2GB of updates further. Now I don't mind bandwidth or anything, but these cumulative updates take ages to install, and after that TiWorker eats one cpu for anything between 0 and two hours or so. And apart from one or two of the cumulative fixed we've had something break every single month, and usually it's MS own programs that stop working, like AppV.
Now while I like the idea of 'when they now fix it, at least the fix ends up on all machines'. True, but we are a very small company. No MS Premier account. That means when I log a case with MS, we first have to answer a buttload of useless questions over and over again and upload several gigabytes of logfiles. Maybe the issue then gets accepted, sometimes they even state the businesscase is too small to fix it for us. That means even if the issue is accepted, it takes months before a fix is released. We have several MS hotfixes on our account, but most of them took 9 - 15 months to create.
So yeah, for us it was definitely better to be able to uninstall a specific patch, and keep the rest updated.
Now ontopic again
Now while I like the idea of 'when they now fix it, at least the fix ends up on all machines'. True, but we are a very small company. No MS Premier account. That means when I log a case with MS, we first have to answer a buttload of useless questions over and over again and upload several gigabytes of logfiles. Maybe the issue then gets accepted, sometimes they even state the businesscase is too small to fix it for us. That means even if the issue is accepted, it takes months before a fix is released. We have several MS hotfixes on our account, but most of them took 9 - 15 months to create.
So yeah, for us it was definitely better to be able to uninstall a specific patch, and keep the rest updated.
Now ontopic again
-
- Novice
- Posts: 9
- Liked: 1 time
- Joined: Nov 23, 2016 7:41 pm
- Full Name: Brad Jackson
- Contact:
Re: KB4013429 breaks Hyper-V 2016 backup, BSOD loop on unins
Not to beat a dead horse, but I wanted to share that I've patched all of my 2016 cluster nodes with KB4013429. No problems performing backups initially.
Who is online
Users browsing this forum: Google [Bot] and 10 guests