Host-based backup of Microsoft Hyper-V VMs.
Post Reply
atid01
Lurker
Posts: 2
Liked: 1 time
Joined: Dec 20, 2017 11:40 pm
Full Name: Anthony
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by atid01 »

Hi all,

After fighting this issue for months, I have found a 'solution' that works. The behavior we were experiencing is that Veeam would only work for two days in a row. After the second day, Hyper-V would fail to merge the snapshot. We had to shutdown the VMs and then restart the VMMS service to get a successful merge (and therefore a successful backup).


From this site: https://social.technet.microsoft.com/Fo ... rverhyperv

I removed the GPO: User rights assignment/Log on as a Service, followed by a GPUpdate / Force on the Hyper-V machine. The stopped all VM's, rebooted the server - and the problem did not surface again.



Instead of just disabling this, you could also just add the proper user needed by Hyper-V to the logon as a service GPO. However, I am not sure what it is. 'Virtual Machines' alone doesn't solve this issue.

Hope this helps someone!
atid01
Lurker
Posts: 2
Liked: 1 time
Joined: Dec 20, 2017 11:40 pm
Full Name: Anthony
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by atid01 » 1 person likes this post

As an update to the above, the proper user is 'NT VIRTUAL MACHINE\Virtual Machines'. However, when trying to add this to logon as a service using GPEdit on a domain controller, it will say not found. The remedy is to install the 'group policy management' role on the Hyper-V host and then add the user. Have not tested yet but this should work.
mpark
Enthusiast
Posts: 32
Liked: 4 times
Joined: Jan 20, 2016 4:43 am
Full Name: Min Park
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mpark »

All,

For those having trouble backup with non-DC VMs with 2016 Hyper-V Host with error messages with failing to create Production checkpoint like
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing)... Error code 32770
I found a working solution.
Basically all you have to do is Grab a VMGuest.iso from Windows Server 2012 R2 hyper-v from %windir%\System32.
Update your VM's Integration Services version to 6.3.9600.x using the ISO and reboot.

This took care of all my non-DC VMs and I'm able to backup using Veeam AAP.

Regards,
MP
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

nmdange wrote:Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.
So...Did the fix make it into this month's update?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

https://support.microsoft.com/en-us/help/4057142

I can't see it in the list of addressed issues, but then again, that list has never been complete so it still can be there
apierce
Lurker
Posts: 2
Liked: never
Joined: Jan 30, 2018 5:23 pm
Full Name: Avery Pierce
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by apierce »

I just tested the 2016 host and 2008 R2 DC guest scenario and the problem is still not fixed with KB4057142. It's getting hard to sell a product that has been broken for so long.
rkovhaev
Veeam Software
Posts: 39
Liked: 21 times
Joined: May 17, 2010 6:49 pm
Full Name: Rustam
Location: hockey night in canada
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by rkovhaev »

per Microsoft premier support 2008R2/2012R2 DC bug is going to be fixed in 3rd week of FEB
apierce
Lurker
Posts: 2
Liked: never
Joined: Jan 30, 2018 5:23 pm
Full Name: Avery Pierce
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by apierce »

Looks like the fix did not make it this month. KB4074590 - https://support.microsoft.com/en-us/hel ... -kb4074590

I installed and tested anyways and can confirm it's not fixed yet.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Seeing that Rustam said the fix will be there the 3rd week of FEB means most probably it will be a private fix first and only be in the MARCH update soonest. At least that is my guess here
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive » 1 person likes this post

well, if someone gets that private fix, make it public so we can start do backup again for our customer.. :)
JTT
Service Provider
Posts: 99
Liked: 2 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by JTT »

I second to that, this is such a absurd situation i just do not get it what has taken so long, to fix it.
JesperTG
Lurker
Posts: 1
Liked: never
Joined: Feb 27, 2018 8:47 am
Full Name: Jesper Thomsen Grauenkjær
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by JesperTG »

Installed 2018-02 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4074590) on my HyperV 2016 hosts on 23/2-2018. Since then I've been able to backup my Windows Server 2012 R2 AD servers without warnings. I still have problems with my windows 2008R2 AD VM's.
xbourcier
Lurker
Posts: 1
Liked: never
Joined: Feb 27, 2018 2:50 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by xbourcier »

hi,

installed this update KB4072650 (Hyper-V integration components update for Windows virtual machines)

no more warnings for me.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

Did you install KB4072650 (Hyper-V integration components update for Windows virtual machines) on any of the guests?
jsull
Influencer
Posts: 12
Liked: never
Joined: Jan 30, 2018 8:11 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by jsull »

Installing KB4072650 on my 2012 R2 Domain Controller VM, running on a 2016 host, fixed the problem for me.
JTT
Service Provider
Posts: 99
Liked: 2 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by JTT »

jsull wrote:Installing KB4072650 on my 2012 R2 Domain Controller VM, running on a 2016 host, fixed the problem for me.
Yep, same here, but for Exchange 2013 on 2012R2 VM, backup still fails.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

JIT,

Exchange 2013 on a 2012R2 VM, that actually should have been solved already (unless I am mistaken). Did you ever created a support call for it?
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

Tallwood wrote:Also having issues with backing up 2016 DC's on 2016 hyper v. I must say Veeam supports response is dreadful, basically just fobbed it off as an MS issue, no evidence that is in fact an MS issue no internal KB that this is a known issue with veeam and no links to bug ID's with MS either basically just left to sort it out ourselves.

Worst of all when veeam has attempted to backup some of these VM's its created checkpoints which then fail to merge when the VM is shutdown and then completely bork the vm itself.
Backup software that doesnt backup supported applications, corrupts VM's and support which isnt particularly proactive, nice one veeam!

Support ID's 02218107 & 02213337
I have this exact same issue now. If I shut these machines down, I'm positive they won't start again because checkpoints exist (based on the disk names), but Hyper-V doesn't know about them.

I just installed the latest updates (as of 2/28/18) and 4 backups ran fine, but now they're hosed again. This is bad. My client just moved his ERP system over to these machines and now I can't back them up.

My support case number is: 02642564.

I ran the updates on 3/1 and thought I was done with this. Very disappointing.
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

Update - Tier 1 of no use:

From: Veeam Support [mailto:support@veeam.com]
Sent: Sunday, March 04, 2018 4:45 PM
To: Jeff Kuhn <jkuhn@edsi.us.com>
Subject: RE: Backups fail on 2016 Hyper-V host and leave unregistered checkpoints on VMs. - Case # 02642564 [ ref:_00D30RWR._5000e1JBsyR:ref ]

Hello Jeff,

Unfortunately Veeam does not create or delete checkpoints. We only send WMI calls to the Hyper-V host to do that.

This does mean that if you have left over checkpoints or are unable to create/delete them, you'll need to reach out to Microsoft Support.


Thanks,
Mike
Veeam Support


Escalated now.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Support is right in saying that we don't create the checkpoints ourselves. We do sent WMI calls to the Hyper-V backup API's. Can you see those checkpoints in Hyper-V manager?
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

Mike thank you. I see no checkpoints in the GUI, but the file names of the disks in settings all end in avhdx. I have a theory, albeit a novice one - Veeam works for the first few runs, then on the last successful run it doesn't properly merge the checkpoint (or Hyper-V doesn't properly merge it). That's why the next backup run fails - that checkpoint is there.

The only thing I see in the event logs are VSS writer time out errors.
ThomasJOSEF
Lurker
Posts: 1
Liked: 1 time
Joined: Mar 06, 2018 12:24 pm
Full Name: Thomas JOSEF
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ThomasJOSEF » 1 person likes this post

Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.
Ensposito
Novice
Posts: 3
Liked: never
Joined: May 30, 2016 2:58 pm
Full Name: Steven Ens
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Ensposito »

Following this thread...same boat, different paddle.
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

ThomasJOSEF wrote:Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.
The behavior is that an .avhdx file is created and then not removed after the backup. It is not listed as a checkpoint in Hyper-V, but it's there and the VM points to it in its settings.

Strangely, when I power down the VM, eventually the .avhdx file disappears and the VM then points back to its correct file.

Is this a merge that happens after power down? Why can't it merge after the backup is complete?
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

I wouldn't be surprised if these issues really are Microsoft issues. (It's still poor for support to tell a customer "it's not our problem, go call Microsoft", though.) Have we not learned our lesson to wait for the "R2" release? Windows Server 2003, Windows Server 2008, Windows Server 2012 were all much better in the R2 flavor. And after R2 came out, the original release got treated very poorly. A very recent example is KB4072650. These new integration components were released for Windows Server 2008 R2 and Windows Server 2012 R2 leaving the R1 releases high and dry. I bet Windows Server 2016 R2 will be rock-solid!
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

none of my 2008 R2 servers can find (or have installed) the kb4072650.
anyone have a link to the .cab file and how to install it manually?
houstek
Novice
Posts: 5
Liked: 1 time
Joined: Dec 31, 2014 5:52 am
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by houstek »

frankive wrote:none of my 2008 R2 servers can find (or have installed) the kb4072650.
anyone have a link to the .cab file and how to install it manually?
same for me ... I only have 2 vm who finds this KB
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

If you are updating your servers via WSUS, have you tried the "Check online for updates from Windows Update" option directly from Windows Update on the VM? I did this by chance and found KB4072650. For some reason, it wasn't showing on my WSUS server as needing approval. So, I manually searched for it and approved it once the search found it to make it available for the rest of the systems.
houstek
Novice
Posts: 5
Liked: 1 time
Joined: Dec 31, 2014 5:52 am
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by houstek »

i check on windows update not WSUS. I do not understand why it is not possible to download it separately
Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests