9.5 Hyper-V 2016 Known Issues

[Kuhnster]

Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.

The issue I'm having is with WINDOWS SERVER 2016 guests running on a WINDOWS SERVER 2016 Hyper-V host.

[dcolpitts]

The behavior is that an .avhdx file is created and then not removed after the backup. It is not listed as a checkpoint in Hyper-V, but it's there and the VM points to it in its settings.
Strangely, when I power down the VM, eventually the .avhdx file disappears and the VM then points back to its correct file.
Is this a merge that happens after power down? Why can't it merge after the backup is complete?

I'm seeing **exactly** the same thing. Power off the VM, and it does some sort of merge in the background. Power the VM back on and it's good to again for a backup or two. I should note that you can't take checkpoint either when this happens. And it doesn't matter which guest it is. W7, 2008, 2008R2, 2012R2 and 2016 guest are all affected randomly on a 2016 host.

dcc

[Mike Resseler]

(Sorry for the late reply... Travel)

So both of you are seeing orphaned snapshots. Do you both already have a support call? (if you already posted it here, sorry about that, but the thread becomes a little bit too long to find it ). I want to follow on that call and try to understand why they get orphaned. veeam does give the WMI instruction to do the merge to Hyper-V so from our point of view, this should be merged after backup, and not the behavior what you are seeing (unless you are using VHD-SET, we are aware that there is a bug with MSFT that they don't seem to be able to solve)

[dcolpitts]

I haven't opened a ticket yet - this is a new install I only completed about a week ago for the customer and have slowly been working the bugs out (there was a massive amount of things done all at once in the environment, and not just in regards to P2V, Hyper-V and Veeam) and this so far has been low on the priority list. That said, I'll get a ticket opened today once I finish restoring one of the support VMs (WSUS) from last night's backup (I was messing with manual merges this morning to see if I could figure out the issue and it didn't go well).

dcc

[dcolpitts]

Mike - see Veeam Support - Case # 02674616 - avdhx not merged into parent vhdx after backup job completes on HyperV 2016.

Thanks

dcc

[mkaec]

I've had problems with avhdx files a couple of times. There were some instances in which it appeared that the avhdx files were leftover after a merge as their last modified date was quite old and I was able to delete them while the VM was running. Other times, the snapshot was in place. For that I issue, I use the Remove-VMSnapshot Powershell command to resolve. I've been having some problems with this system and the DFS-R service sometimes locking up during VSS operations. I had assumed the avhdx problems were related to that. Server 2016 has been quite a bit of trouble for me.

[dcolpitts]

I've had problems with avhdx files a couple of times. There were some instances in which it appeared that the avhdx files were leftover after a merge as their last modified date was quite old and I was able to delete them while the VM was running. Other times, the snapshot was in place. For that I issue, I use the Remove-VMSnapshot Powershell command to resolve. I've been having some problems with this system and the DFS-R service sometimes locking up during VSS operations. I had assumed the avhdx problems were related to that. Server 2016 has been quite a bit of trouble for me.

If only it were that simple. This is the only Hyper-V server in the organization, fresh install of Windows 2016 1607 DC three weeks ago. As you see from the output below, there are no snapshots, yet the VMs are using differencing disk.

Code: Select all

PS C:\Windows\system32> Get-VM | Get-VMSnapshot
PS C:\Windows\system32> Get-VM | Select-Object VMId | Get-VHD | ft path, @{label=’Size(GB)’;expression={$_.filesize/1gb -as [int]}}, vhdtype

Path                                                                                   Size(GB)      VhdType
----                                                                                   --------      -------
D:\Hyper-V\Virtual Hard Disks\2012R2-TEMPL01.vhdx                                            18      Dynamic
D:\Hyper-V\Virtual Hard Disks\2016-TEMPL01.vhdx                                              21      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive_3062F9DF-AC72-4829-A4B8-0DFD35EE8618.avhdx        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_E_Drive_E352AF89-320D-4459-808C-7BBA29A053F3.avhdx        2 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_F_Drive_F3D4C0F3-2DE4-46FD-ABDD-FEECF40D53F2.avhdx        1 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST2_C_Drive_F35990A5-5500-4E55-8949-BC276B25843C.avhdx        1 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST3_C_Drive_418B37A0-C093-46AE-B60C-49E5034CD6B0.avhdx        7 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST5_C_Drive_D09A1EF2-4CA8-4F4A-8B2F-5B2944F26014.avhdx        0 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST5_E_Drive_54253DB7-EFD8-4420-8AB8-5363F486B922.avhdx       16 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST9_C_Drive_F8FE472C-9323-45B3-8B05-D04D43451832.avhdx        1 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST9_E_Drive_BFE65A50-CE2A-486A-BCEF-5D0557A6FF41.avhdx        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST-AD01_C_Drive.vhdx                                         25      Dynamic
E:\Hyper-V\Virtual Hard Disks\CUST-AD02_C_Drive.vhdx                                         31      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST-DOMINO01_C_Drive_7BFF1293-E6B1-4FF0-96F6-4CB32FE...        0 Differencing
E:\Hyper-V\Virtual Hard Disks\CUSTEAC_C_Drive.vhdx                                           94      Dynamic
E:\Hyper-V\Virtual Hard Disks\CUST-EXC01_C_Drive_33A1932D-6B2E-459B-8CCD-903808502D...        2 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST-EXC01_D_Drive_09A64A18-2910-452B-80AB-BF87683B07...       10 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST-LIC01_C_Drive_9C0646E5-4951-4309-8A03-2B03DFB530...        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST-VEEAM01_C_Drive.vhdx                                      31      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST-WSUS01_C_Drive.vhdx                                       34      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST-WSUS01_D_Drive.vhdx                                       59      Dynamic
D:\Hyper-V\Virtual Hard Disks\W7Px64-OFF2K10_C_Drive.vhdx                                    29      Dynamic
D:\Hyper-V\Virtual Hard Disks\W7Px64-TEMPL01_C_Drive.vhdx                                    26      Dynamic
D:\Hyper-V\Virtual Hard Disks\WG-DIMENSION_BOOT01.vhd                                         2      Dynamic
D:\Hyper-V\Virtual Hard Disks\WG-DIMENSION_DATA01.vhdx                                       23      Dynamic

dcc

[Mike Resseler]

Can you do Get-VHD 'location\filename.vhdx' on some of those VHDX files? Not sure but it looks these are manual differencing disks?

[dcolpitts]

Here you go Mike. Again, there were never any manual checkpoints taken, only recovery checkpoints on behalf of Veeam during the backups.

Code: Select all

PS C:\Windows\system32> Get-VHD "D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive_3062F9DF-AC72-4829-A4B8-0DFD35EE8618.avhdx"

ComputerName            : CUST-HV01
Path                    : d:\hyper-v\virtual hard disks\CUST1_c_drive_3062f9df-ac72-4829-a4b8-0dfd35ee8618.avhdx
VhdFormat               : VHDX
VhdType                 : Differencing
FileSize                : 309329920
Size                    : 107374182400
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 512
BlockSize               : 2097152
ParentPath              : D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive.vhdx
DiskIdentifier          : 95D8B6A2-4B89-4A2B-BCD2-68C57F25ED17
FragmentationPercentage :
Alignment               : 1
Attached                : False
DiskNumber              :
Number                  :

PS C:\Windows\system32> Get-VHD "D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive.vhdx"

ComputerName            : CUST-HV01
Path                    : d:\hyper-v\virtual hard disks\CUST1_c_drive.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 62352523264
Size                    : 107374182400
MinimumSize             : 105005449216
LogicalSectorSize       : 512
PhysicalSectorSize      : 512
BlockSize               : 2097152
ParentPath              :
DiskIdentifier          : 95D8B6A2-4B89-4A2B-BCD2-68C57F25ED17
FragmentationPercentage : 10
Alignment               : 1
Attached                : True
DiskNumber              :
Number                  :

[Mike Resseler]

OK. Let's look at what support says. I think I am out of ideas at this moment. Hopefully they see more in the logs

[dcolpitts]

Long night (it's almost 11 pm now) - I shutdown one of my affected VMs at 6:30 pm (a 500GB SQL database for a ERP app that I couldn't backup because I couldn't take a checkpoint) only to find my checkpoint chain was corrupt and the VM would not boot... After about 3 3/4 hours of darn near heart failure (who expects disk corruption rebooting a server - oh silly me - it was Hyper-V - I should have known), with help of Veeam support (go Veeam!!!) I managed to get my checkpoint chain put back together without data lost but I definitely lost at least 1 year off my life and added many many grey hairs.

Anyways - I'm not here to rant about Hyper-V (I'll do that on my own blog), but what I am here to say is that I discover the kernel System process (PID 4) is/was holding some of the vhds opened for that VM and one other even after the VMs were powered off and the HyperV services were restarted. I found this out while trying to manually make a copy of my vhds before trying to piece the checkpoint chain back together. I got an access is denied because the file was opened when I tried to move the files (after making a copy to a 2nd SAN volume, I was just going to move the original files to a temp location to work on them). Once I got this, I wanted to know what was holding the files opened so I run two Sysinternal Suites tools - psexec and procexp64 (both of which are in the path of every Windows box I deploy). To do so I opened an Administrative command prompt and ran:

psexec -i -d -s procexp64.exe

This launches Process Explorer as "NT Authority\System". Then, I hit CTRL+F to "Find Handle or DLL" and entered a wildcard search for the vhd name (i.e. CUST1*). This returns all the handles that have the files named CUST1* opened. Sure enough, it was PID 4, System.

Now what is significant about this is that we've been having issues ejecting RDX cartridges at two other client locations after the backup completes (mind you both those locations are VMware, not Hyper-V, and VBR servers there are 2012 R2 and not 2016). Using the same steps above, I had already determined it was PID 4, System that is holding the cartridges opened which prevents Windows from ejecting them - but I didn't have enough information to put 2 + 2 together. Most of my customers' setups are cookie cutter - so the first thing that popped into my head that was common was they were all HPE DL380s (one Gen9, and two Gen10). Two were fresh windows loads, and one was a migration from a Gen9. They all had the current HPE Support Pack for Proliant deployed to them. So the next thing in common was the RDX drives (the checkpoint issues only occur during backup to RDX for me, not when going to the StoreOnce Catalyst share). So I check if we had loaded the HPE RDX Tools on those other two servers - yup - same version as on the Hyper-V host - HPE RDX Tools 1.59. A quick survey of all my clients reveals that the **ONLY** three locations that have the service RDXmon 1.51 installed and running (which comes from HPE RDX Tools 1.59 - go figure) are these three systems - the three systems that seem to be having file lock issues after dealing with the RDX drives. All the other sites either don't have RDX Tools installed (even if they do have an RDX drive) or they have older version of HPE RDX Tools (i.e. 1.56) installed.

So I stopped the RDXmon 1.51 service and the RDXSoftEjectService and set both disabled. We'll see what happens tomorrow I guess (only 7 hours away until my alarm goes off!)

BTW - one comment about my support experience tonight for you Veeam guys that monitor these threads. The engineer I was talking was smart, polite, and seemed pretty confident in that fact that it was just a matter of stitching the checkpoints back together based on what I described to him, but I was very disappoint with the reference material I was given to do it with - a 3rd party blog and a separate YouTube video from someone else for Hyper-V on 2008 using a 3rd party tool. I really had expected Veeam to have formal documentation on using get-vhd to determine the parent path, set-vhd to reconnect them, and mount-vhd to verify them afterwards. After I went through what I was sent by the engineer, I did some additional research and PowerShell was much simpler (and quicker) than the Hyper-V GUI and 3rd party tool that I was directed to in the video. I guess on the flip side of the coin, next week maybe Veeam support can reference my more relevant blog after I write up my "HOWTO: Piece checkpoints back together again after failed merges" entry and post it!

dcc

[Mike Resseler]

I can only say thanks for still writing up such a great post even if it was late already! You are Veeam material

PM me the name of the support engineer please, and thanks for the good feedback on the reference material. I like it and will discuss this (to be honest, low priority) with the VP's of support to see what we can improve.

And I am waiting for your response to see if disabling the service will help.

[dcolpitts]

Update #1 - I just got to my desk - we got close to a foot of snow overnight (another 6" predicted for today) with high winds - power went out at the site that had the corrupt checkpoints at around 2am and only came back on 6:30am and the servers shutdown, so no update from there if that fixed the issue or not yet.

dcc

[dcolpitts]

Disabling RDXmon didn't help with the failed merges. And just shutting down the VM is not always enough to get the merge to complete (sometimes it still fails). But if I restart the "Hyper-V Virtual Machine Management" service with the VMs powered off, they immediately merge (and auto-poweron if so configured). And then any other VM that is shutdown afterwards that has failed to merge after the backup then merges properly on shutdown. I don't know how long this remedy works - the host was rebooted 12 hours ago at 7:30p... The last successful merge I see in the event log is at 10p, and then next attempted merge was at 11:53p (which failed). Actually, I've noticed a couple of times in the short history of this host that I've had to restart VMMS to do certain things (like create a new VM), otherwise I get an error and the task doesn't complete.

I'm half tempted to add a scheduled task to restart VMMS every hour or two at this point and see what happens.

dcc

[Mike Resseler]

What scares me is that you say that you need to restart VMMS to do certain things such as creating a new VM. This leads me in the direction that there is something seriously wrong with that host and (Yes, I can be wrong ) it actually has nothing to do with Veeam, but with the hyper-v itself. Besides Veeam, is there something else on those hosts? AV? Management software? ...

[Kuhnster]

My case number is 02642564.

It has since been closed because this is a MSFT problem.

I still get the "Operation cannot be performed while the object is in use" error when taking a checkpoint.

I did see this interesting post on Veeam.com: https://www.veeam.com/blog/hyperv-2016- ... dates.htmland this interesting sentence:

"Once you've upgraded an environment to Veeam Backup & Replication 9.5, Hyper-V 2016 and VM Hardware version 8.0 with the latest integration services, you'll be able to fully benefit from the points discussed here." (Production checkpoints).

All 3 of my troubled VMs are generation 1, Server 2016, and are sitting on a Server 2016 Hyper-V host. The Ubuntu machine backs up just fine.

I am going to create some generation 2 test VMs this weekend and swing the hard drives (copies of them) over to these machines and see if they will back up. Maybe the whole problem is that I am using the wrong hardware. Admittedly, I have no idea what hardware version I am running, only that it is generation 1.

[mkaec]

It's always sad when a support case is closed with "Not our problem. Contact Microsoft." given how difficult it can be to get through to good support at MS.

In my experience, it's very important for Veeam that the configuration version be 8.0. But, generation 1 should be OK. However, I've never tried long-term use of leaving a VM at generation 1 that could be generation 2.

I'm not sure how well it would work to move vhdx files from a generation 1 system to a generation 2 system. If you have a problem with that, there is an unofficial conversion tool. I had good success with it, but Windows Server 2016 was not out when I used it.

https://code.msdn.microsoft.com/windows ... n-81ddafa2

[dcolpitts]

McAfee Enterprise AV was added to it to **after** the issues were already occurring. Just 2016 with the HyperV role, SQL Express 2014 and VBR on it. Other than that, it's a pretty vanilla box.

Last night (err - this morning) was better - only one VM with a differencing disk left over, but I did add a scheduled task to restart VMMS every two hours.

dcc

[Mike Resseler]

It still scares me that you need to restart VMMS so often. Do you have error notifications in eventviewer for Hyper-V? For example, do you have something logged when you create a VM and it fails?

[Kuhnster]

Per this web page, I turned off Backup in Hyper-V Integration Services and now my 3 Server 2016 VMs back up just fine. I can't do AAIP, but it's better than nothing. How bizzare.

http://backupchain.com/i/hyper-v-backup ... 0x80070490

[Mike Resseler]

And did you try the end-suggestion in the article? After turning it off, turn it back on again? (Seems like really bizarre indeed)

[dcolpitts]

I split the backup job to RDX into two... Last night worked like it should and no differencing disk this morning.

dcc

[Kuhnster]

And did you try the end-suggestion in the article? After turning it off, turn it back on again? (Seems like really bizarre indeed)

I think I missed that!

I ran 6 backups in a row before it FAILED AGAIN with 32774 error.

I will turn it back on tonight (shutting everything down to merge the .avhdx files anyway) and monitor things again.

Thanks Mike.

Jeff

[jed-hyper]

I also have this issue.
Running Windows Server 2016 Hyper-V Host with Windows 2012 R2 guest which is a DC
Veeam error: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'.
Can someone please post here when Microsoft have a fix for this issue ?

FYI installing KB4072650 on my Windows 2012 R2 guest virtual machine (Which is a domain controller) fixed this issue FINALLY...

[Mike Resseler]

Hi Jed,

Thanks for letting us know. We do get more and more notes from the field that this specific update fixes indeed the issue. Just like you, we are very happy about that

[Kuhnster]

And did you try the end-suggestion in the article? After turning it off, turn it back on again? (Seems like really bizarre indeed)

I'm still having the issues, Mike. Error 32774 when taking checkpoints on 2016 VMs running in a 2016 HV host.

[collinp]

I'm also getting 32768 with an error of "Failed to create VM recovery checkpoint.." using Hyper-v 2016. It's a widespread issue. The only fix that we have found is to shut down the affected vm, move it to another Hyper-v host and then power the vm on again. We are using SMB 3 and NetApp storage. The issue is only happening on Hyper-v 2016, but the other factor is that Hyper-v 2016 is using SMB 3 whereas our 2012 clusters are using CSV's.

Veeam support asked me to manually create a Hyper-v checkpoint which would point to Hyper-v being the issue (and not Veeam) but this is still an issue whether it is on the Hyper-v or Veeam side. The issue does not happen in our identical test environment which isn't being backed up.

I sent all the logs to Veeam within case 02638159. We aren't using the application aware options and these aren't domain controllers.

I wonder is this is a load issue, especially with the allow volume snapshot checked. We throttled the number of snapshots per Hyper-v host and it is happening less.

Another symptom of this issue is when this error is occurring, you are not able to live migrate the VM.

Hope I posed to the correct thread. The beginning of the thread talks about error 32768 (my issue) but later within the thread it discusses 32774

[mkaec]

Code: Select all

Path                                                                                   Size(GB)      VhdType
----                                                                                   --------      -------
D:\Hyper-V\Virtual Hard Disks\2012R2-TEMPL01.vhdx                                            18      Dynamic
D:\Hyper-V\Virtual Hard Disks\2016-TEMPL01.vhdx                                              21      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive_3062F9DF-AC72-4829-A4B8-0DFD35EE8618.avhdx        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_E_Drive_E352AF89-320D-4459-808C-7BBA29A053F3.avhdx        2 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_F_Drive_F3D4C0F3-2DE4-46FD-ABDD-FEECF40D53F2.avhdx        1 Differencing
...

It looks like this occurred on one of my VMs. I was in the processes of moving it to a new host when a backup started. I canceled the backup, but one of the disks came out like this with the avhdx assigned as the disk. But, it looks like I got lucky. I manually created a checkpoint and deleted it, and all of the avhdx files merged back into the main.

[Line_Rider]

I'm experiencing a similar issue with only one 2012 R2 vm on my 2016 S2D cluster if I have AAP enabled, can't create the checkpoint and thus of course the backup fails.
Integration services version is 6.3.9600.18907 and it's a VM version 8. That particular hotfix KB4072650, is installed.

If I leave the checkpoint option at production, I can not manually create a checkpoint either. If I change it to standard checkpoints I can then successfully create checkpoints and deleting them works fine as well, no left over .avhdx files.

But, if I leave it at standard checkpoints and then run the backup job, it still fails to create the checkpoint.

All my other vm's in the cluster work fine with AAP and it's a mix of 08R2, 2012, 2012 R2 and 2016 vm's.

[ss@ctstechs.com]

Microsoft released an update to the Hyper-V integration components in February 2018 that fixes this issue, but it's recommended update so it won't download automatically. KB 4072650

https://support.microsoft.com/en-ie/hel ... l-machines