9.5 Hyper-V 2016 Known Issues

Discussions specific to Microsoft Hyper-V hypervisor

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Kuhnster » Mon Mar 12, 2018 4:47 pm

ThomasJOSEF wrote:Had the same Problem with Hyperv 2016.
After a remote session with veeam Support, i had to upgrade the Integration services in the vm.
(no Windows updates were available for the vm)
you can download it under https://support.microsoft.com/en-us/hel ... l-machines
You have to install the cab file: DISM.exe /Online /Add-Package /PackagePath:path to cab
After reboot the backup was sucessfull.


The issue I'm having is with WINDOWS SERVER 2016 guests running on a WINDOWS SERVER 2016 Hyper-V host.
Kuhnster
Influencer
 
Posts: 12
Liked: never
Joined: Fri Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Mon Mar 12, 2018 5:00 pm

Kuhnster wrote:The behavior is that an .avhdx file is created and then not removed after the backup. It is not listed as a checkpoint in Hyper-V, but it's there and the VM points to it in its settings.
Strangely, when I power down the VM, eventually the .avhdx file disappears and the VM then points back to its correct file.
Is this a merge that happens after power down? Why can't it merge after the backup is complete?


I'm seeing **exactly** the same thing. Power off the VM, and it does some sort of merge in the background. Power the VM back on and it's good to again for a backup or two. I should note that you can't take checkpoint either when this happens. And it doesn't matter which guest it is. W7, 2008, 2008R2, 2012R2 and 2016 guest are all affected randomly on a 2016 host.

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Tue Mar 13, 2018 6:13 am

(Sorry for the late reply... Travel)

So both of you are seeing orphaned snapshots. Do you both already have a support call? (if you already posted it here, sorry about that, but the thread becomes a little bit too long to find it :-)). I want to follow on that call and try to understand why they get orphaned. veeam does give the WMI instruction to do the merge to Hyper-V so from our point of view, this should be merged after backup, and not the behavior what you are seeing (unless you are using VHD-SET, we are aware that there is a bug with MSFT that they don't seem to be able to solve)
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Tue Mar 13, 2018 12:38 pm

I haven't opened a ticket yet - this is a new install I only completed about a week ago for the customer and have slowly been working the bugs out (there was a massive amount of things done all at once in the environment, and not just in regards to P2V, Hyper-V and Veeam) and this so far has been low on the priority list. That said, I'll get a ticket opened today once I finish restoring one of the support VMs (WSUS) from last night's backup (I was messing with manual merges this morning to see if I could figure out the issue and it didn't go well).

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Tue Mar 13, 2018 1:49 pm

Mike - see Veeam Support - Case # 02674616 - avdhx not merged into parent vhdx after backup job completes on HyperV 2016.

Thanks

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby mkaec » Tue Mar 13, 2018 2:20 pm

I've had problems with avhdx files a couple of times. There were some instances in which it appeared that the avhdx files were leftover after a merge as their last modified date was quite old and I was able to delete them while the VM was running. Other times, the snapshot was in place. For that I issue, I use the Remove-VMSnapshot Powershell command to resolve. I've been having some problems with this system and the DFS-R service sometimes locking up during VSS operations. I had assumed the avhdx problems were related to that. Server 2016 has been quite a bit of trouble for me.
mkaec
Expert
 
Posts: 222
Liked: 49 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Tue Mar 13, 2018 2:28 pm

mkaec wrote:I've had problems with avhdx files a couple of times. There were some instances in which it appeared that the avhdx files were leftover after a merge as their last modified date was quite old and I was able to delete them while the VM was running. Other times, the snapshot was in place. For that I issue, I use the Remove-VMSnapshot Powershell command to resolve. I've been having some problems with this system and the DFS-R service sometimes locking up during VSS operations. I had assumed the avhdx problems were related to that. Server 2016 has been quite a bit of trouble for me.


If only it were that simple. This is the only Hyper-V server in the organization, fresh install of Windows 2016 1607 DC three weeks ago. As you see from the output below, there are no snapshots, yet the VMs are using differencing disk.

Code: Select all
PS C:\Windows\system32> Get-VM | Get-VMSnapshot
PS C:\Windows\system32> Get-VM | Select-Object VMId | Get-VHD | ft path, @{label=’Size(GB)’;expression={$_.filesize/1gb -as [int]}}, vhdtype

Path                                                                                   Size(GB)      VhdType
----                                                                                   --------      -------
D:\Hyper-V\Virtual Hard Disks\2012R2-TEMPL01.vhdx                                            18      Dynamic
D:\Hyper-V\Virtual Hard Disks\2016-TEMPL01.vhdx                                              21      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive_3062F9DF-AC72-4829-A4B8-0DFD35EE8618.avhdx        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_E_Drive_E352AF89-320D-4459-808C-7BBA29A053F3.avhdx        2 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_F_Drive_F3D4C0F3-2DE4-46FD-ABDD-FEECF40D53F2.avhdx        1 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST2_C_Drive_F35990A5-5500-4E55-8949-BC276B25843C.avhdx        1 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST3_C_Drive_418B37A0-C093-46AE-B60C-49E5034CD6B0.avhdx        7 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST5_C_Drive_D09A1EF2-4CA8-4F4A-8B2F-5B2944F26014.avhdx        0 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST5_E_Drive_54253DB7-EFD8-4420-8AB8-5363F486B922.avhdx       16 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST9_C_Drive_F8FE472C-9323-45B3-8B05-D04D43451832.avhdx        1 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST9_E_Drive_BFE65A50-CE2A-486A-BCEF-5D0557A6FF41.avhdx        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST-AD01_C_Drive.vhdx                                         25      Dynamic
E:\Hyper-V\Virtual Hard Disks\CUST-AD02_C_Drive.vhdx                                         31      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST-DOMINO01_C_Drive_7BFF1293-E6B1-4FF0-96F6-4CB32FE...        0 Differencing
E:\Hyper-V\Virtual Hard Disks\CUSTEAC_C_Drive.vhdx                                           94      Dynamic
E:\Hyper-V\Virtual Hard Disks\CUST-EXC01_C_Drive_33A1932D-6B2E-459B-8CCD-903808502D...        2 Differencing
E:\Hyper-V\Virtual Hard Disks\CUST-EXC01_D_Drive_09A64A18-2910-452B-80AB-BF87683B07...       10 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST-LIC01_C_Drive_9C0646E5-4951-4309-8A03-2B03DFB530...        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST-VEEAM01_C_Drive.vhdx                                      31      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST-WSUS01_C_Drive.vhdx                                       34      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST-WSUS01_D_Drive.vhdx                                       59      Dynamic
D:\Hyper-V\Virtual Hard Disks\W7Px64-OFF2K10_C_Drive.vhdx                                    29      Dynamic
D:\Hyper-V\Virtual Hard Disks\W7Px64-TEMPL01_C_Drive.vhdx                                    26      Dynamic
D:\Hyper-V\Virtual Hard Disks\WG-DIMENSION_BOOT01.vhd                                         2      Dynamic
D:\Hyper-V\Virtual Hard Disks\WG-DIMENSION_DATA01.vhdx                                       23      Dynamic

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Tue Mar 13, 2018 3:27 pm

Can you do Get-VHD 'location\filename.vhdx' on some of those VHDX files? Not sure but it looks these are manual differencing disks?
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Tue Mar 13, 2018 3:32 pm

Here you go Mike. Again, there were never any manual checkpoints taken, only recovery checkpoints on behalf of Veeam during the backups.

Code: Select all
PS C:\Windows\system32> Get-VHD "D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive_3062F9DF-AC72-4829-A4B8-0DFD35EE8618.avhdx"

ComputerName            : CUST-HV01
Path                    : d:\hyper-v\virtual hard disks\CUST1_c_drive_3062f9df-ac72-4829-a4b8-0dfd35ee8618.avhdx
VhdFormat               : VHDX
VhdType                 : Differencing
FileSize                : 309329920
Size                    : 107374182400
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 512
BlockSize               : 2097152
ParentPath              : D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive.vhdx
DiskIdentifier          : 95D8B6A2-4B89-4A2B-BCD2-68C57F25ED17
FragmentationPercentage :
Alignment               : 1
Attached                : False
DiskNumber              :
Number                  :

PS C:\Windows\system32> Get-VHD "D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive.vhdx"

ComputerName            : CUST-HV01
Path                    : d:\hyper-v\virtual hard disks\CUST1_c_drive.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 62352523264
Size                    : 107374182400
MinimumSize             : 105005449216
LogicalSectorSize       : 512
PhysicalSectorSize      : 512
BlockSize               : 2097152
ParentPath              :
DiskIdentifier          : 95D8B6A2-4B89-4A2B-BCD2-68C57F25ED17
FragmentationPercentage : 10
Alignment               : 1
Attached                : True
DiskNumber              :
Number                  :
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Tue Mar 13, 2018 3:58 pm

OK. Let's look at what support says. I think I am out of ideas at this moment. Hopefully they see more in the logs
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Wed Mar 14, 2018 2:09 am

Long night (it's almost 11 pm now) - I shutdown one of my affected VMs at 6:30 pm (a 500GB SQL database for a ERP app that I couldn't backup because I couldn't take a checkpoint) only to find my checkpoint chain was corrupt and the VM would not boot... After about 3 3/4 hours of darn near heart failure (who expects disk corruption rebooting a server - oh silly me - it was Hyper-V - I should have known), with help of Veeam support (go Veeam!!!) I managed to get my checkpoint chain put back together without data lost but I definitely lost at least 1 year off my life and added many many grey hairs.

Anyways - I'm not here to rant about Hyper-V (I'll do that on my own blog), but what I am here to say is that I discover the kernel System process (PID 4) is/was holding some of the vhds opened for that VM and one other even after the VMs were powered off and the HyperV services were restarted. I found this out while trying to manually make a copy of my vhds before trying to piece the checkpoint chain back together. I got an access is denied because the file was opened when I tried to move the files (after making a copy to a 2nd SAN volume, I was just going to move the original files to a temp location to work on them). Once I got this, I wanted to know what was holding the files opened so I run two Sysinternal Suites tools - psexec and procexp64 (both of which are in the path of every Windows box I deploy). To do so I opened an Administrative command prompt and ran:

psexec -i -d -s procexp64.exe

This launches Process Explorer as "NT Authority\System". Then, I hit CTRL+F to "Find Handle or DLL" and entered a wildcard search for the vhd name (i.e. CUST1*). This returns all the handles that have the files named CUST1* opened. Sure enough, it was PID 4, System.

Now what is significant about this is that we've been having issues ejecting RDX cartridges at two other client locations after the backup completes (mind you both those locations are VMware, not Hyper-V, and VBR servers there are 2012 R2 and not 2016). Using the same steps above, I had already determined it was PID 4, System that is holding the cartridges opened which prevents Windows from ejecting them - but I didn't have enough information to put 2 + 2 together. Most of my customers' setups are cookie cutter - so the first thing that popped into my head that was common was they were all HPE DL380s (one Gen9, and two Gen10). Two were fresh windows loads, and one was a migration from a Gen9. They all had the current HPE Support Pack for Proliant deployed to them. So the next thing in common was the RDX drives (the checkpoint issues only occur during backup to RDX for me, not when going to the StoreOnce Catalyst share). So I check if we had loaded the HPE RDX Tools on those other two servers - yup - same version as on the Hyper-V host - HPE RDX Tools 1.59. A quick survey of all my clients reveals that the **ONLY** three locations that have the service RDXmon 1.51 installed and running (which comes from HPE RDX Tools 1.59 - go figure) are these three systems - the three systems that seem to be having file lock issues after dealing with the RDX drives. All the other sites either don't have RDX Tools installed (even if they do have an RDX drive) or they have older version of HPE RDX Tools (i.e. 1.56) installed.

So I stopped the RDXmon 1.51 service and the RDXSoftEjectService and set both disabled. We'll see what happens tomorrow I guess (only 7 hours away until my alarm goes off!)

BTW - one comment about my support experience tonight for you Veeam guys that monitor these threads. The engineer I was talking was smart, polite, and seemed pretty confident in that fact that it was just a matter of stitching the checkpoints back together based on what I described to him, but I was very disappoint with the reference material I was given to do it with - a 3rd party blog and a separate YouTube video from someone else for Hyper-V on 2008 using a 3rd party tool. I really had expected Veeam to have formal documentation on using get-vhd to determine the parent path, set-vhd to reconnect them, and mount-vhd to verify them afterwards. After I went through what I was sent by the engineer, I did some additional research and PowerShell was much simpler (and quicker) than the Hyper-V GUI and 3rd party tool that I was directed to in the video. I guess on the flip side of the coin, next week maybe Veeam support can reference my more relevant blog after I write up my "HOWTO: Piece checkpoints back together again after failed merges" entry and post it! :-)

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Wed Mar 14, 2018 10:25 am

I can only say thanks for still writing up such a great post even if it was late already! You are Veeam material ;-)

PM me the name of the support engineer please, and thanks for the good feedback on the reference material. I like it and will discuss this (to be honest, low priority) with the VP's of support to see what we can improve.

And I am waiting for your response to see if disabling the service will help.
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Wed Mar 14, 2018 10:30 am

Update #1 - I just got to my desk - we got close to a foot of snow overnight (another 6" predicted for today) with high winds - power went out at the site that had the corrupt checkpoints at around 2am and only came back on 6:30am and the servers shutdown, so no update from there if that fixed the issue or not yet.

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby dcolpitts » Thu Mar 15, 2018 10:53 am

Disabling RDXmon didn't help with the failed merges. And just shutting down the VM is not always enough to get the merge to complete (sometimes it still fails). But if I restart the "Hyper-V Virtual Machine Management" service with the VMs powered off, they immediately merge (and auto-poweron if so configured). And then any other VM that is shutdown afterwards that has failed to merge after the backup then merges properly on shutdown. I don't know how long this remedy works - the host was rebooted 12 hours ago at 7:30p... The last successful merge I see in the event log is at 10p, and then next attempted merge was at 11:53p (which failed). Actually, I've noticed a couple of times in the short history of this host that I've had to restart VMMS to do certain things (like create a new VM), otherwise I get an error and the task doesn't complete.

I'm half tempted to add a scheduled task to restart VMMS every hour or two at this point and see what happens.

dcc
dcolpitts
Veeam ProPartner
 
Posts: 68
Liked: 12 times
Joined: Fri Apr 01, 2011 10:36 am
Location: Atlantic coast of Canada
Full Name: Dean Colpitts

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Fri Mar 16, 2018 7:34 am

What scares me is that you say that you need to restart VMMS to do certain things such as creating a new VM. This leads me in the direction that there is something seriously wrong with that host and (Yes, I can be wrong :-)) it actually has nothing to do with Veeam, but with the hyper-v itself. Besides Veeam, is there something else on those hosts? AV? Management software? ...
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: CloudMSP and 7 guests