-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Dec 20, 2017 11:40 pm
- Full Name: Anthony
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Hi all,
After fighting this issue for months, I have found a 'solution' that works. The behavior we were experiencing is that Veeam would only work for two days in a row. After the second day, Hyper-V would fail to merge the snapshot. We had to shutdown the VMs and then restart the VMMS service to get a successful merge (and therefore a successful backup).
From this site: https://social.technet.microsoft.com/Fo ... rverhyperv
I removed the GPO: User rights assignment/Log on as a Service, followed by a GPUpdate / Force on the Hyper-V machine. The stopped all VM's, rebooted the server - and the problem did not surface again.
Instead of just disabling this, you could also just add the proper user needed by Hyper-V to the logon as a service GPO. However, I am not sure what it is. 'Virtual Machines' alone doesn't solve this issue.
Hope this helps someone!
After fighting this issue for months, I have found a 'solution' that works. The behavior we were experiencing is that Veeam would only work for two days in a row. After the second day, Hyper-V would fail to merge the snapshot. We had to shutdown the VMs and then restart the VMMS service to get a successful merge (and therefore a successful backup).
From this site: https://social.technet.microsoft.com/Fo ... rverhyperv
I removed the GPO: User rights assignment/Log on as a Service, followed by a GPUpdate / Force on the Hyper-V machine. The stopped all VM's, rebooted the server - and the problem did not surface again.
Instead of just disabling this, you could also just add the proper user needed by Hyper-V to the logon as a service GPO. However, I am not sure what it is. 'Virtual Machines' alone doesn't solve this issue.
Hope this helps someone!
-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Dec 20, 2017 11:40 pm
- Full Name: Anthony
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
As an update to the above, the proper user is 'NT VIRTUAL MACHINE\Virtual Machines'. However, when trying to add this to logon as a service using GPEdit on a domain controller, it will say not found. The remedy is to install the 'group policy management' role on the Hyper-V host and then add the user. Have not tested yet but this should work.
-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
All,
For those having trouble backup with non-DC VMs with 2016 Hyper-V Host with error messages with failing to create Production checkpoint like
Basically all you have to do is Grab a VMGuest.iso from Windows Server 2012 R2 hyper-v from %windir%\System32.
Update your VM's Integration Services version to 6.3.9600.x using the ISO and reboot.
This took care of all my non-DC VMs and I'm able to backup using Veeam AAP.
Regards,
MP
For those having trouble backup with non-DC VMs with 2016 Hyper-V Host with error messages with failing to create Production checkpoint like
I found a working solution.Failed to create VM recovery checkpoint (mode: Veeam application-aware processing)... Error code 32770
Basically all you have to do is Grab a VMGuest.iso from Windows Server 2012 R2 hyper-v from %windir%\System32.
Update your VM's Integration Services version to 6.3.9600.x using the ISO and reboot.
This took care of all my non-DC VMs and I'm able to backup using Veeam AAP.
Regards,
MP
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
So...Did the fix make it into this month's update?nmdange wrote:Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
https://support.microsoft.com/en-us/help/4057142
I can't see it in the list of addressed issues, but then again, that list has never been complete so it still can be there
I can't see it in the list of addressed issues, but then again, that list has never been complete so it still can be there
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jan 30, 2018 5:23 pm
- Full Name: Avery Pierce
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
I just tested the 2016 host and 2008 R2 DC guest scenario and the problem is still not fixed with KB4057142. It's getting hard to sell a product that has been broken for so long.
-
- Veeam Software
- Posts: 39
- Liked: 21 times
- Joined: May 17, 2010 6:49 pm
- Full Name: Rustam
- Location: hockey night in canada
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
per Microsoft premier support 2008R2/2012R2 DC bug is going to be fixed in 3rd week of FEB
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jan 30, 2018 5:23 pm
- Full Name: Avery Pierce
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Looks like the fix did not make it this month. KB4074590 - https://support.microsoft.com/en-us/hel ... -kb4074590
I installed and tested anyways and can confirm it's not fixed yet.
I installed and tested anyways and can confirm it's not fixed yet.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Seeing that Rustam said the fix will be there the 3rd week of FEB means most probably it will be a private fix first and only be in the MARCH update soonest. At least that is my guess here
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
well, if someone gets that private fix, make it public so we can start do backup again for our customer..
-
- Service Provider
- Posts: 116
- Liked: 5 times
- Joined: Jan 02, 2017 7:31 am
- Full Name: JTT
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
I second to that, this is such a absurd situation i just do not get it what has taken so long, to fix it.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Feb 27, 2018 8:47 am
- Full Name: Jesper Thomsen Grauenkjær
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Installed 2018-02 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4074590) on my HyperV 2016 hosts on 23/2-2018. Since then I've been able to backup my Windows Server 2012 R2 AD servers without warnings. I still have problems with my windows 2008R2 AD VM's.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Feb 27, 2018 2:50 pm
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
hi,
installed this update KB4072650 (Hyper-V integration components update for Windows virtual machines)
no more warnings for me.
installed this update KB4072650 (Hyper-V integration components update for Windows virtual machines)
no more warnings for me.
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Did you install KB4072650 (Hyper-V integration components update for Windows virtual machines) on any of the guests?
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Jan 30, 2018 8:11 pm
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Installing KB4072650 on my 2012 R2 Domain Controller VM, running on a 2016 host, fixed the problem for me.
-
- Service Provider
- Posts: 116
- Liked: 5 times
- Joined: Jan 02, 2017 7:31 am
- Full Name: JTT
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Yep, same here, but for Exchange 2013 on 2012R2 VM, backup still fails.jsull wrote:Installing KB4072650 on my 2012 R2 Domain Controller VM, running on a 2016 host, fixed the problem for me.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
JIT,
Exchange 2013 on a 2012R2 VM, that actually should have been solved already (unless I am mistaken). Did you ever created a support call for it?
Exchange 2013 on a 2012R2 VM, that actually should have been solved already (unless I am mistaken). Did you ever created a support call for it?
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Dec 19, 2014 3:34 pm
- Full Name: Jeff Kuhn
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
I have this exact same issue now. If I shut these machines down, I'm positive they won't start again because checkpoints exist (based on the disk names), but Hyper-V doesn't know about them.Tallwood wrote:Also having issues with backing up 2016 DC's on 2016 hyper v. I must say Veeam supports response is dreadful, basically just fobbed it off as an MS issue, no evidence that is in fact an MS issue no internal KB that this is a known issue with veeam and no links to bug ID's with MS either basically just left to sort it out ourselves.
Worst of all when veeam has attempted to backup some of these VM's its created checkpoints which then fail to merge when the VM is shutdown and then completely bork the vm itself.
Backup software that doesnt backup supported applications, corrupts VM's and support which isnt particularly proactive, nice one veeam!
Support ID's 02218107 & 02213337
I just installed the latest updates (as of 2/28/18) and 4 backups ran fine, but now they're hosed again. This is bad. My client just moved his ERP system over to these machines and now I can't back them up.
My support case number is: 02642564.
I ran the updates on 3/1 and thought I was done with this. Very disappointing.
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Dec 19, 2014 3:34 pm
- Full Name: Jeff Kuhn
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Update - Tier 1 of no use:
From: Veeam Support [mailto:support@veeam.com]
Sent: Sunday, March 04, 2018 4:45 PM
To: Jeff Kuhn <jkuhn@edsi.us.com>
Subject: RE: Backups fail on 2016 Hyper-V host and leave unregistered checkpoints on VMs. - Case # 02642564 [ ref:_00D30RWR._5000e1JBsyR:ref ]
Hello Jeff,
Unfortunately Veeam does not create or delete checkpoints. We only send WMI calls to the Hyper-V host to do that.
This does mean that if you have left over checkpoints or are unable to create/delete them, you'll need to reach out to Microsoft Support.
Thanks,
Mike
Veeam Support
Escalated now.
From: Veeam Support [mailto:support@veeam.com]
Sent: Sunday, March 04, 2018 4:45 PM
To: Jeff Kuhn <jkuhn@edsi.us.com>
Subject: RE: Backups fail on 2016 Hyper-V host and leave unregistered checkpoints on VMs. - Case # 02642564 [ ref:_00D30RWR._5000e1JBsyR:ref ]
Hello Jeff,
Unfortunately Veeam does not create or delete checkpoints. We only send WMI calls to the Hyper-V host to do that.
This does mean that if you have left over checkpoints or are unable to create/delete them, you'll need to reach out to Microsoft Support.
Thanks,
Mike
Veeam Support
Escalated now.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Support is right in saying that we don't create the checkpoints ourselves. We do sent WMI calls to the Hyper-V backup API's. Can you see those checkpoints in Hyper-V manager?
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Dec 19, 2014 3:34 pm
- Full Name: Jeff Kuhn
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Mike thank you. I see no checkpoints in the GUI, but the file names of the disks in settings all end in avhdx. I have a theory, albeit a novice one - Veeam works for the first few runs, then on the last successful run it doesn't properly merge the checkpoint (or Hyper-V doesn't properly merge it). That's why the next backup run fails - that checkpoint is there.
The only thing I see in the event logs are VSS writer time out errors.
The only thing I see in the event logs are VSS writer time out errors.
-
- Lurker
- Posts: 1
- Liked: 1 time
- Joined: Mar 06, 2018 12:24 pm
- Full Name: Thomas JOSEF
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.
-
- Novice
- Posts: 3
- Liked: never
- Joined: May 30, 2016 2:58 pm
- Full Name: Steven Ens
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
Following this thread...same boat, different paddle.
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Dec 19, 2014 3:34 pm
- Full Name: Jeff Kuhn
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
The behavior is that an .avhdx file is created and then not removed after the backup. It is not listed as a checkpoint in Hyper-V, but it's there and the VM points to it in its settings.ThomasJOSEF wrote:Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.
Strangely, when I power down the VM, eventually the .avhdx file disappears and the VM then points back to its correct file.
Is this a merge that happens after power down? Why can't it merge after the backup is complete?
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
I wouldn't be surprised if these issues really are Microsoft issues. (It's still poor for support to tell a customer "it's not our problem, go call Microsoft", though.) Have we not learned our lesson to wait for the "R2" release? Windows Server 2003, Windows Server 2008, Windows Server 2012 were all much better in the R2 flavor. And after R2 came out, the original release got treated very poorly. A very recent example is KB4072650. These new integration components were released for Windows Server 2008 R2 and Windows Server 2012 R2 leaving the R1 releases high and dry. I bet Windows Server 2016 R2 will be rock-solid!
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
none of my 2008 R2 servers can find (or have installed) the kb4072650.
anyone have a link to the .cab file and how to install it manually?
anyone have a link to the .cab file and how to install it manually?
-
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
With the .avhdx files these links might be relevant
https://blog.workinghardinit.work/2015/ ... l-machine/
https://blog.workinghardinit.work/2016/ ... rver-2016/
https://blog.workinghardinit.work/2017/ ... rver-2016/
https://blog.workinghardinit.work/2015/ ... l-machine/
https://blog.workinghardinit.work/2016/ ... rver-2016/
https://blog.workinghardinit.work/2017/ ... rver-2016/
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Dec 31, 2014 5:52 am
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
same for me ... I only have 2 vm who finds this KBfrankive wrote:none of my 2008 R2 servers can find (or have installed) the kb4072650.
anyone have a link to the .cab file and how to install it manually?
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
If you are updating your servers via WSUS, have you tried the "Check online for updates from Windows Update" option directly from Windows Update on the VM? I did this by chance and found KB4072650. For some reason, it wasn't showing on my WSUS server as needing approval. So, I manually searched for it and approved it once the search found it to make it available for the rest of the systems.
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Dec 31, 2014 5:52 am
- Contact:
Re: 9.5 Hyper-V 2016 Known Issues
i check on windows update not WSUS. I do not understand why it is not possible to download it separately
Who is online
Users browsing this forum: No registered users and 31 guests