Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Mar 16, 2018 8:10 pm

My case number is 02642564.

It has since been closed because this is a MSFT problem.

I still get the "Operation cannot be performed while the object is in use" error when taking a checkpoint.

I did see this interesting post on Veeam.com: https://www.veeam.com/blog/hyperv-2016- ... dates.htmland this interesting sentence:

"Once you've upgraded an environment to Veeam Backup & Replication 9.5, Hyper-V 2016 and VM Hardware version 8.0 with the latest integration services, you'll be able to fully benefit from the points discussed here." (Production checkpoints).

All 3 of my troubled VMs are generation 1, Server 2016, and are sitting on a Server 2016 Hyper-V host. The Ubuntu machine backs up just fine.

I am going to create some generation 2 test VMs this weekend and swing the hard drives (copies of them) over to these machines and see if they will back up. Maybe the whole problem is that I am using the wrong hardware. Admittedly, I have no idea what hardware version I am running, only that it is generation 1.

mkaec
Expert
Posts: 289
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec » Mar 17, 2018 12:16 pm

It's always sad when a support case is closed with "Not our problem. Contact Microsoft." given how difficult it can be to get through to good support at MS.

In my experience, it's very important for Veeam that the configuration version be 8.0. But, generation 1 should be OK. However, I've never tried long-term use of leaving a VM at generation 1 that could be generation 2.

I'm not sure how well it would work to move vhdx files from a generation 1 system to a generation 2 system. If you have a problem with that, there is an unofficial conversion tool. I had good success with it, but Windows Server 2016 was not out when I used it.

https://code.msdn.microsoft.com/windows ... n-81ddafa2

dcolpitts
Veeam ProPartner
Posts: 73
Liked: 12 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by dcolpitts » Mar 17, 2018 7:57 pm

McAfee Enterprise AV was added to it to **after** the issues were already occurring. Just 2016 with the HyperV role, SQL Express 2014 and VBR on it. Other than that, it's a pretty vanilla box.

Last night (err - this morning) was better - only one VM with a differencing disk left over, but I did add a scheduled task to restart VMMS every two hours.

dcc

Mike Resseler
Product Manager
Posts: 5226
Liked: 549 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler » Mar 19, 2018 7:56 am

It still scares me that you need to restart VMMS so often. Do you have error notifications in eventviewer for Hyper-V? For example, do you have something logged when you create a VM and it fails?

Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Mar 20, 2018 12:15 pm

Per this web page, I turned off Backup in Hyper-V Integration Services and now my 3 Server 2016 VMs back up just fine. I can't do AAIP, but it's better than nothing. How bizzare.

http://backupchain.com/i/hyper-v-backup ... 0x80070490

Mike Resseler
Product Manager
Posts: 5226
Liked: 549 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler » Mar 20, 2018 12:24 pm

And did you try the end-suggestion in the article? After turning it off, turn it back on again? (Seems like really bizarre indeed)

dcolpitts
Veeam ProPartner
Posts: 73
Liked: 12 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by dcolpitts » Mar 20, 2018 12:27 pm

I split the backup job to RDX into two... Last night worked like it should and no differencing disk this morning.

dcc

Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Mar 21, 2018 6:49 pm

Mike Resseler wrote:And did you try the end-suggestion in the article? After turning it off, turn it back on again? (Seems like really bizarre indeed)
I think I missed that!

I ran 6 backups in a row before it FAILED AGAIN with 32774 error.

I will turn it back on tonight (shutting everything down to merge the .avhdx files anyway) and monitor things again.

Thanks Mike.

Jeff

jed-hyper
Enthusiast
Posts: 39
Liked: 4 times
Joined: Feb 26, 2014 4:42 am
Full Name: Jed Parkes
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by jed-hyper » Mar 28, 2018 2:41 am

jed-hyper wrote:I also have this issue.
Running Windows Server 2016 Hyper-V Host with Windows 2012 R2 guest which is a DC
Veeam error: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'.
Can someone please post here when Microsoft have a fix for this issue ?
FYI installing KB4072650 on my Windows 2012 R2 guest virtual machine (Which is a domain controller) fixed this issue FINALLY...

Mike Resseler
Product Manager
Posts: 5226
Liked: 549 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler » Mar 28, 2018 6:38 am

Hi Jed,

Thanks for letting us know. We do get more and more notes from the field that this specific update fixes indeed the issue. Just like you, we are very happy about that ;-)

Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Apr 03, 2018 12:18 pm

Mike Resseler wrote:And did you try the end-suggestion in the article? After turning it off, turn it back on again? (Seems like really bizarre indeed)
I'm still having the issues, Mike. Error 32774 when taking checkpoints on 2016 VMs running in a 2016 HV host.

collinp
Expert
Posts: 137
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by collinp » Apr 06, 2018 6:08 pm

I'm also getting 32768 with an error of "Failed to create VM recovery checkpoint.." using Hyper-v 2016. It's a widespread issue. The only fix that we have found is to shut down the affected vm, move it to another Hyper-v host and then power the vm on again. We are using SMB 3 and NetApp storage. The issue is only happening on Hyper-v 2016, but the other factor is that Hyper-v 2016 is using SMB 3 whereas our 2012 clusters are using CSV's.

Veeam support asked me to manually create a Hyper-v checkpoint which would point to Hyper-v being the issue (and not Veeam) but this is still an issue whether it is on the Hyper-v or Veeam side. The issue does not happen in our identical test environment which isn't being backed up.

I sent all the logs to Veeam within case 02638159. We aren't using the application aware options and these aren't domain controllers.

I wonder is this is a load issue, especially with the allow volume snapshot checked. We throttled the number of snapshots per Hyper-v host and it is happening less.

Another symptom of this issue is when this error is occurring, you are not able to live migrate the VM.

Hope I posed to the correct thread. The beginning of the thread talks about error 32768 (my issue) but later within the thread it discusses 32774

mkaec
Expert
Posts: 289
Liked: 61 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec » Apr 07, 2018 12:00 am

dcolpitts wrote:

Code: Select all

Path                                                                                   Size(GB)      VhdType
----                                                                                   --------      -------
D:\Hyper-V\Virtual Hard Disks\2012R2-TEMPL01.vhdx                                            18      Dynamic
D:\Hyper-V\Virtual Hard Disks\2016-TEMPL01.vhdx                                              21      Dynamic
D:\Hyper-V\Virtual Hard Disks\CUST1_C_Drive_3062F9DF-AC72-4829-A4B8-0DFD35EE8618.avhdx        0 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_E_Drive_E352AF89-320D-4459-808C-7BBA29A053F3.avhdx        2 Differencing
D:\Hyper-V\Virtual Hard Disks\CUST1_F_Drive_F3D4C0F3-2DE4-46FD-ABDD-FEECF40D53F2.avhdx        1 Differencing
...
It looks like this occurred on one of my VMs. I was in the processes of moving it to a new host when a backup started. I canceled the backup, but one of the disks came out like this with the avhdx assigned as the disk. But, it looks like I got lucky. I manually created a checkpoint and deleted it, and all of the avhdx files merged back into the main.

Line_Rider
Influencer
Posts: 21
Liked: 3 times
Joined: Mar 10, 2016 7:05 pm
Full Name: Jon
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Line_Rider » May 08, 2018 10:19 pm

I'm experiencing a similar issue with only one 2012 R2 vm on my 2016 S2D cluster if I have AAP enabled, can't create the checkpoint and thus of course the backup fails.
Integration services version is 6.3.9600.18907 and it's a VM version 8. That particular hotfix KB4072650, is installed.

If I leave the checkpoint option at production, I can not manually create a checkpoint either. If I change it to standard checkpoints I can then successfully create checkpoints and deleting them works fine as well, no left over .avhdx files.

But, if I leave it at standard checkpoints and then run the backup job, it still fails to create the checkpoint.

All my other vm's in the cluster work fine with AAP and it's a mix of 08R2, 2012, 2012 R2 and 2016 vm's.

ss@ctstechs.com
Lurker
Posts: 1
Liked: 1 time
Joined: May 21, 2018 2:53 am
Full Name: Shannon Stoker
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ss@ctstechs.com » May 21, 2018 2:55 am 1 person likes this post

Microsoft released an update to the Hyper-V integration components in February 2018 that fixes this issue, but it's recommended update so it won't download automatically. KB 4072650

https://support.microsoft.com/en-ie/hel ... l-machines

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests