9.5 Hyper-V 2016 Known Issues

[atid01]

Hi all,

After fighting this issue for months, I have found a 'solution' that works. The behavior we were experiencing is that Veeam would only work for two days in a row. After the second day, Hyper-V would fail to merge the snapshot. We had to shutdown the VMs and then restart the VMMS service to get a successful merge (and therefore a successful backup).


From this site: https://social.technet.microsoft.com/Fo ... rverhyperv

I removed the GPO: User rights assignment/Log on as a Service, followed by a GPUpdate / Force on the Hyper-V machine. The stopped all VM's, rebooted the server - and the problem did not surface again.



Instead of just disabling this, you could also just add the proper user needed by Hyper-V to the logon as a service GPO. However, I am not sure what it is. 'Virtual Machines' alone doesn't solve this issue.

Hope this helps someone!

[atid01]

As an update to the above, the proper user is 'NT VIRTUAL MACHINE\Virtual Machines'. However, when trying to add this to logon as a service using GPEdit on a domain controller, it will say not found. The remedy is to install the 'group policy management' role on the Hyper-V host and then add the user. Have not tested yet but this should work.

[mpark]

All,

For those having trouble backup with non-DC VMs with 2016 Hyper-V Host with error messages with failing to create Production checkpoint like

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing)... Error code 32770

I found a working solution.
Basically all you have to do is Grab a VMGuest.iso from Windows Server 2012 R2 hyper-v from %windir%\System32.
Update your VM's Integration Services version to 6.3.9600.x using the ISO and reboot.

This took care of all my non-DC VMs and I'm able to backup using Veeam AAP.

Regards,
MP

[mkaec]

Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.

So...Did the fix make it into this month's update?

[Mike Resseler]

https://support.microsoft.com/en-us/help/4057142

I can't see it in the list of addressed issues, but then again, that list has never been complete so it still can be there

[apierce]

I just tested the 2016 host and 2008 R2 DC guest scenario and the problem is still not fixed with KB4057142. It's getting hard to sell a product that has been broken for so long.

[rkovhaev]

per Microsoft premier support 2008R2/2012R2 DC bug is going to be fixed in 3rd week of FEB

[apierce]

Looks like the fix did not make it this month. KB4074590 - https://support.microsoft.com/en-us/hel ... -kb4074590

I installed and tested anyways and can confirm it's not fixed yet.

[Mike Resseler]

Seeing that Rustam said the fix will be there the 3rd week of FEB means most probably it will be a private fix first and only be in the MARCH update soonest. At least that is my guess here

[frankive]

well, if someone gets that private fix, make it public so we can start do backup again for our customer..

[JTT]

I second to that, this is such a absurd situation i just do not get it what has taken so long, to fix it.

[JesperTG]

Installed 2018-02 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4074590) on my HyperV 2016 hosts on 23/2-2018. Since then I've been able to backup my Windows Server 2012 R2 AD servers without warnings. I still have problems with my windows 2008R2 AD VM's.

[xbourcier]

hi,

installed this update KB4072650 (Hyper-V integration components update for Windows virtual machines)

no more warnings for me.

[mkaec]

Did you install KB4072650 (Hyper-V integration components update for Windows virtual machines) on any of the guests?

[jsull]

Installing KB4072650 on my 2012 R2 Domain Controller VM, running on a 2016 host, fixed the problem for me.

[JTT]

Installing KB4072650 on my 2012 R2 Domain Controller VM, running on a 2016 host, fixed the problem for me.

Yep, same here, but for Exchange 2013 on 2012R2 VM, backup still fails.

[Mike Resseler]

JIT,

Exchange 2013 on a 2012R2 VM, that actually should have been solved already (unless I am mistaken). Did you ever created a support call for it?

[Kuhnster]

Also having issues with backing up 2016 DC's on 2016 hyper v. I must say Veeam supports response is dreadful, basically just fobbed it off as an MS issue, no evidence that is in fact an MS issue no internal KB that this is a known issue with veeam and no links to bug ID's with MS either basically just left to sort it out ourselves.

Worst of all when veeam has attempted to backup some of these VM's its created checkpoints which then fail to merge when the VM is shutdown and then completely bork the vm itself.
Backup software that doesnt backup supported applications, corrupts VM's and support which isnt particularly proactive, nice one veeam!

Support ID's 02218107 & 02213337

I have this exact same issue now. If I shut these machines down, I'm positive they won't start again because checkpoints exist (based on the disk names), but Hyper-V doesn't know about them.

I just installed the latest updates (as of 2/28/18) and 4 backups ran fine, but now they're hosed again. This is bad. My client just moved his ERP system over to these machines and now I can't back them up.

My support case number is: 02642564.

I ran the updates on 3/1 and thought I was done with this. Very disappointing.

[Kuhnster]

Update - Tier 1 of no use:

From: Veeam Support [mailto:support@veeam.com]
Sent: Sunday, March 04, 2018 4:45 PM
To: Jeff Kuhn <jkuhn@edsi.us.com>
Subject: RE: Backups fail on 2016 Hyper-V host and leave unregistered checkpoints on VMs. - Case # 02642564 [ ref:_00D30RWR._5000e1JBsyR:ref ]

Hello Jeff,

Unfortunately Veeam does not create or delete checkpoints. We only send WMI calls to the Hyper-V host to do that.

This does mean that if you have left over checkpoints or are unable to create/delete them, you'll need to reach out to Microsoft Support.


Thanks,
Mike
Veeam Support


Escalated now.

[Mike Resseler]

Support is right in saying that we don't create the checkpoints ourselves. We do sent WMI calls to the Hyper-V backup API's. Can you see those checkpoints in Hyper-V manager?

[Kuhnster]

Mike thank you. I see no checkpoints in the GUI, but the file names of the disks in settings all end in avhdx. I have a theory, albeit a novice one - Veeam works for the first few runs, then on the last successful run it doesn't properly merge the checkpoint (or Hyper-V doesn't properly merge it). That's why the next backup run fails - that checkpoint is there.

The only thing I see in the event logs are VSS writer time out errors.

[ThomasJOSEF]

Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.

[Ensposito]

Following this thread...same boat, different paddle.

[Kuhnster]

Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.

The behavior is that an .avhdx file is created and then not removed after the backup. It is not listed as a checkpoint in Hyper-V, but it's there and the VM points to it in its settings.

Strangely, when I power down the VM, eventually the .avhdx file disappears and the VM then points back to its correct file.

Is this a merge that happens after power down? Why can't it merge after the backup is complete?

[mkaec]

I wouldn't be surprised if these issues really are Microsoft issues. (It's still poor for support to tell a customer "it's not our problem, go call Microsoft", though.) Have we not learned our lesson to wait for the "R2" release? Windows Server 2003, Windows Server 2008, Windows Server 2012 were all much better in the R2 flavor. And after R2 came out, the original release got treated very poorly. A very recent example is KB4072650. These new integration components were released for Windows Server 2008 R2 and Windows Server 2012 R2 leaving the R1 releases high and dry. I bet Windows Server 2016 R2 will be rock-solid!

[frankive]

none of my 2008 R2 servers can find (or have installed) the kb4072650.
anyone have a link to the .cab file and how to install it manually?

[nmdange]

With the .avhdx files these links might be relevant
https://blog.workinghardinit.work/2015/ ... l-machine/

https://blog.workinghardinit.work/2016/ ... rver-2016/
https://blog.workinghardinit.work/2017/ ... rver-2016/

[houstek]

none of my 2008 R2 servers can find (or have installed) the kb4072650.
anyone have a link to the .cab file and how to install it manually?

same for me ... I only have 2 vm who finds this KB

[mkaec]

If you are updating your servers via WSUS, have you tried the "Check online for updates from Windows Update" option directly from Windows Update on the VM? I did this by chance and found KB4072650. For some reason, it wasn't showing on my WSUS server as needing approval. So, I manually searched for it and approved it once the search found it to make it available for the rest of the systems.

[houstek]

i check on windows update not WSUS. I do not understand why it is not possible to download it separately