9.5 Hyper-V 2016 Known Issues

[jazzoberoi]

Hi Guys,

I was reading the release notes of the 9.5 version and noticed the following Known Issues with Hyper-V :

Resilient Change Tracking (RCT) returns incorrect changes for VMs which had their virtual disks moved with Storage Live Migration, resulting in corrupt backups. This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update. Until then, disable the use of changed block tracking for any jobs processing such VMs.
• Application-aware processing of VMs with guest OS other than Windows Server 2016 fails with the “Failed to take in-guest VSS snapshot COM error: Code: 0x80042308” error. This issue is fixed in the recent Windows Updates for the corresponding operating systems.
• Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.
• Backing up VMs from Hyper-V cluster in rolling upgrade is supported, however RCT will not be leveraged until the upgrade is completed for all nodes and cluster functional level is upgraded to Windows Server 2016. Keep in mind that VMs virtual hardware version must be upgraded to version 8.0 before RCT can be leveraged on the VM.

Is there a way to track or find out if and when these bugs will be resolved ?

The Resilient Change Tracking (RCT) was a big move forward for Hyper-V as it provided the baseline for change tracking in Hyper-V VM's instead of relying on the CBT driver provided by the backup provider (VEEAM in this case).

However, this bug is proving that we now will have to rely on the Microsoft QA to ensure no major bugs surface with the RCT, otherwise there is a strong possibility of backup corruption ! Unfortunately this bug proves that it could be a bumpy ride ahead

[Mike Resseler]

Jazz,

Tracking when this bug is resolved is done with Microsoft. And I would have no idea how to track their bugs. The only thing we can do is keep putting pressure on MSFT (and we do, and with us the entire community ) so that they put priority on this one. As you can see from the RN, MSFT is aware of it and they are working on it, and the moment we know it is fixed, we will certainly advise to install the necessary update

Cheers
Mike

[Thomas Hietschold]

The problem comes up after the live migration of a Hyper-V VM. We also tested it with manual copy of the .vhdx (because the the orgininal problem was described as "...virtual disks moved with Storage Live Migration"), without succes. But now we found a simple solution: delete the machine definition in Hyper-V, add a new machine with correct parameters, add the still available .vhdx of the deleted machine. Now the Veeam job works correct without error.

[Mike Resseler]

Thomas,

That is interesting. So you delete the VM but not the disks, recreate a VM (with same parameters) and attach the disks and then it works again? Did you try to restore also? (Use Surebackup for the test to be sure that everything works)

Thanks
Mike

[Gostev]

I would definitely expect this to work, because doing so resets CBT data.

[grazor]

Any idea on which update is needed to resolve the “Failed to take in-guest VSS snapshot COM error: Code: 0x80042308” error?

I've tried creating a support case with Veeam as our backups are a complete mess right now, but they haven't been helpful, other than suggesting I contact Microsoft and find out why Veeam isn't backing anything up.

[Gostev]

Indeed, Hyper-V 2016 is extremely raw at the moment.
As to your question, try installing KB3063109.

[sbradbury]

Just an FYI, hopefully save some people from down time, do NOT install KB3063109 on an exchange 2013/2016 server, it's no good.

https://social.technet.microsoft.com/Fo ... nservergen

[mkaec]

That was my question, too, when I read the 9.5 release notes. Why isn't there a link to an MS KB article that I can follow? Well, maybe a KB article does not exist. I don't know.

One of the things I'll remember about 2016 is how many problems VMware had with CBT and how happy I was to be on Hyper-V. I hope Microsoft doesn't turn the tide in 2017 with a lot of RTC bugs.

[Mike Resseler]

Marc,

We hope the same (obviously) and hope that they will put a lot of effort in it. But there is not only this change, there is also the change the way they deliver their updates which makes it confusing for many people. I guess we all will need to learn these new methods with update rollups and get used to it

[JTT]

Does anybody have an update of the known issues?

[Mike Resseler]

Not for the moment. Soon we will have UR1 and some more information. I did find a link to track the updates for server 2016: https://support.microsoft.com/en-us/hel ... te-history

[mkaec]

I'm OK with the update rollup methodology. It's much better than the "release hotfixes that are hard to find and are not released via Windows Update until we release a service pack except we no longer release service packs" methodology that they used to use.

[Mike Resseler]

Marc,

Certainly not saying that it is a bad system. Just saying we need to get used to it. The only thing that really bothers me is the lack of information that you get per rollup update. It does state a few things what is fixed and enhanced but not really a lot more information. I'm hoping that they will increase their effort for that documentation so that we are better aware what is fixed and in what cases

[AlexLeadingEdge]

I believe I am having this same issue:

Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.

Case # 02039027.

Windows Server 2016 Hyper-V + Veeam B&R 9.5 SP1 + SBS2011 (Windows Server 2008 R2 + Domain Controller + Exchange)

These are the error messages I'm seeing:

31/01/2017 1:56:16 p.m. :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover) Details: Job failed ('Checkpoint operation for 'SBS-SVR' failed. (Virtual machine ID B854900D-2DA6-44D0-9E3A-24D5D4700368)'). Error code: '32768'. Failed to create VM recovery snapshot, VM ID 'b854900d-2da6-44d0-9e3a-24d5d4700368'.

31/01/2017 1:56:27 p.m. :: Retrying snapshot creation attempt (Failed to create production checkpoint.)

31/01/2017 1:56:40 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)

[Mike Resseler]

Alex,

This is indeed one of them. Unfortunately we still have no news if this is solved or not in the latest rollup update.

[mfriedl]

Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.

I am having this same issue, too.

[mkaec]

I've been randomly seeing the Failed to create VM recovery checkpoint problem on a 2012 R2 VM running on a 2016 host. However, this VM is not a DC. It is a file server. So far, when it happens, I restart the VM and backups start working again. I'm hoping whatever change is made to fix the issue for DCs also fixes it for this system. The file server does have DFS Replication and DFS Namespace services running. That's one commonality with a modern DC.

[agrob]

Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.

Have a Case open with MS Support about this issue. When i have any updates, i'll post it here.

[evolmar]

Hi guys,

Thanks for the thread. We currently have a 3 host Windows Server 2012 cluster connected to Dell Equallogic. I built up a new 3 host Hyper-V 2016 cluster connected to a Tintri VMStore T820. It's been running great but I was planning on moving VMs over to it and reading this thread has given me pause. Should I rebuild the cluster to Windows Server 2012R2 instead?

Thanks in advance!

-e

[agrob]

Well i just can speak from a single server hyperv 2016 server Installation. I "just" have troubles with 2012R2 Server with RoDC Role installed. 2012R2 without RoDC Role, 2008R2 and Windows 10 VM Backup is running fine so far

[agrob]

have opened a ticket now. ID 02060542.
Because MS has no knowledge about a fix for this issue....

[AlexLeadingEdge]

Alex, This is indeed one of them. Unfortunately we still have no news if this is solved or not in the latest rollup update.

Gostev said that there will be a patch next Tuesday but it is causing instability with VEEAM?

Regarding the Windows Server 2016 deduplication corruption issue that I mentioned in the previous digest, be aware that we have one report of KB3216755 (which supposedly fixes the issues) introducing system instability and causes Veeam services to crash on Windows Server 2016. We have the crash dump and are currently investigating - meanwhile, uninstalling the update helps. At this time, the update is only available on Microsoft Update Catalog web site, but presumably will become widely available during the next Patch Tuesday.

[Gostev]

The instability should be fixed by Microsoft before the patch goes GA, see thread on this in the main forum.

[AlexLeadingEdge]

The instability should be fixed by Microsoft before the patch goes GA, see thread on this in the main forum.

Sorry to be a pain Gostev but can you provide a link to where you want me to look?

[Mike Resseler]

I believe Gostev refers to this thread > Veeam services crash after installing KB3216755, KB4010672

[StoneFritz]

Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.

The same error here.

[frankive]

We also see this error after repliacted from a 2012 hyper-v to hyper-v 2016 with Veeam.
Guest os 2012.

Only running 2 vms. The best and easy fix is to delete the vm and attach the .vhdx to a new vm?

[frankive]

well, it didn't work to create a new vm (gen . Turned off application aware for now and backup runs fine.
This is a 2012 domain controller VM.

[Salm]

Hey there !

I was recently given a workaround for backing up DCs with AAP enabled.

I tried it on 2012 R2, but it should work on 2012 (haven't tried it).

What you do is :

Stop the "COM+ Event System" service (be careful, it has a few dependencies, such as DHCP Server, so make sure to restart it after).
Restart the "Volume Shadow Copy" service
Start the "COM+ Event System" again (and all it's dependencies).

You have to do this every time the DC reboots.

I wrote a script that is started by a scheduled task, scheduled to run "at startup", which works fairly well.

Use at your own risk (I wrote it, but I can't guarantee it will work for your system), there is no error check on this script, and it was only tested on 2012 R2 with powershell 4.0

Code: Select all

function fStop-Services([string]$serviceToStop)
{
    #Get running dependencies
    $dependent_services = (get-service $ServiceToStop).dependentservices | get-service |Where-Object {$_.Status -eq "Running"}

    #stop dependent services
    foreach ($dependent_service in $dependent_services)
    {
        Write-Host "Stopping " $dependent_service
        stop-service $dependent_service
    }
    #stop service
    Write-Host "Stopping " $ServiceToStop
    stop-service $ServiceToStop

    return $dependent_services
}


function fStart-Services($serviceToStart)
{
    Write-Host "Starting " $serviceToStart
    start-service $serviceToStart

    foreach ($service in $services)
    {
        Write-Host "Starting " $service
        start-service $service
    }

}

$log="c:\scripts\log.txt"

"start" |out-file $log
$services = fStop-Services("eventsystem")


sleep -s 5

#restart VSS
Write-Host "Restarting VSS"
Restart-Service VSS

sleep -s 5

fStart-Services("eventsystem")

sleep -s 5

"end" |out-file $log -append

exit 0

Change $log to a path that will suit you (it only writes start and end when the script runs, erases the file on each run).

Scheduled task configuration
Trigger : "At Startup"
Action : Program/script C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Arguments : -ExecutionPolicy Bypass -file c:\Scripts\Restart-DCOM-VSS.ps1
Change path to actual script location.

Hope that helps.