Discussions specific to the Microsoft Hyper-V hypervisor
cruggeri
Novice
Posts: 4
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri » May 21, 2018 8:25 pm

There was also an update for Hyper-V 2016 just released I saw in this post:

https://social.technet.microsoft.com/Fo ... rverhyperv

Over lunch, I applied the update suggested by ss above to my VM, applied the update referred to in the technet forum to the host, rebooted. So far, 3 backups and a successful merge of the checkpoints. Not "out of the woods" yet, so to pseak, as it had begun failing after about 36 hours one time...but...I am hopeful.

cruggeri
Novice
Posts: 4
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri » May 22, 2018 1:56 pm

and this morning I came in and see the AVHDX did not merge properly. So, still stuck.

pwm001
Lurker
Posts: 1
Liked: never
Joined: May 29, 2018 7:12 pm
Full Name: Philip W Michael
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by pwm001 » May 29, 2018 7:26 pm

Have you found a solution yet? I am having the same issue. I wonder if it's some sort of rights/permissions issue. I am using the free version of Veeam for my home lab and have a script that runs from Task Scheduler each night. I have been using this setup for about a year now and never had any problems. Lately I changed some admin passwords and created a new domain admin account so I could disable the default domain admin account. I use the new account to run the scripts and sometimes it works and sometimes I have the same issue as you. I can run the backups just fine one at a time from the Veeam B&S, but my script fails most of the time. I haven't tried restarting the VMMS to see if it will merge all the VMs, but I will try that the next time it happens. Like you, some of the VMs merge after shutting them down and some fail and I have to fix them manually if I can. Powershell commands don't fix them. You are the only other person I have found with the exact same issue. Just an FYI, my script runs at 4 AM on one host to backup 6 VMs on two hosts. At this time, there is no one logged in to the server.

cruggeri
Novice
Posts: 4
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri » Jun 01, 2018 3:32 pm

I have been out of office. It is still failing; however, we have a little more information.

While out, I had someone fill in for me. We had resolved to shutting down the machine, backing it up, then immediately restoring it. When my "fill in" attempted this for the 1st time, he was unable to restore and we discovered the Hyper-V virtual machine management service needed to be restarted (I had been rebooting the host as well).

Upon my return, I was attempting to restart the VMMS service to see if the backups would succeed, which they didn't; however, I did see that when the machine was shut down to run the backup, the snapshots immediately began to merge, actually, causing the backup to fail; but, everything was then merged.

So, starting last night, I watched the backup run and create the checkpoint. Once the checkpoint creation was completed and the backup began, I restarted the VMMS service. The backup was successful and the Avhdx files merged as expected. I repeated this again this morning, with the same result. It has only been twice now, and we have had issues after successfully running a day, so I don't know if this would continue to work, but it has so far.

gorka
Enthusiast
Posts: 38
Liked: 4 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka » Jun 05, 2018 9:38 pm

Hi,
the problem is hyper-v 2016, I am also suffering these last 2 weeks without finding a solution. Every day I have to combine vhdx manually, restart the administrator of virtual machines of hyper-v etc etc, I do not know what else to try, I'm dying

greg.haberek
Lurker
Posts: 2
Liked: never
Joined: Aug 31, 2017 1:15 pm
Full Name: Greg Haberek
Location: Sterling Heights, MI
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by greg.haberek » Jun 07, 2018 1:26 pm

I was having the same problem with these .avhdx files remaining after the backup checkpoints were removed. This started happening after running Windows Updates on the hosts on May 25, 2018 (bringing them up to kb4103723). Same symptoms as described previously: there are no checkpoints listed but there are .avhdx files sitting in the Virtual Hard Disks folder for the machine.

I would have to shutdown the virtual machine, manually merge each of the .avhdx file into its parent, and then re-attach the original .vhdx and restart the machine. The guests affected were running any of CentOS 6, CentOS 7, Server 2008 R2, and Server 2016. One thing that was strange, is that it always seemed to be the same guests with the leftover .avhdx files.

I finally stumbled upon kb4103720 which is not coming down through WSUS. I downloaded it manually from the Update Catalog as described in the article and installed it manually on each of my Hyper-V hosts yesterday, before my backups started for the night. This SEEMS to have resolved the issue as of this morning. There are no .avhdx files to be found on any of my machines.

What's strange, is that KB article doesn't specifically mention this issue in its release notes. The closest it comes is this:
kb4103720 wrote:
  • Addresses an issue that makes it impossible to revert to a virtual machine checkpoint. Reapplying the checkpoint fails with an error.
We don't use Veeam Backup for Hyper-V so I can confirm that is not what's causing the issue. (We do use Veeam Backup for Office 365.) We are (unfortunately) using NetVault Backup for a while longer until we can get off some legacy systems that Veeam does not support. For me, this issue seems to be related specifically to the update kb4103723 for Windows Server 2016. However, others seem to have been having this issue since before that update was released.

I'll keep a close eye on this and report back if the situation changes. For now I'm considering this resolved on my Hyper-V hosts.

gorka
Enthusiast
Posts: 38
Liked: 4 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka » Jun 08, 2018 8:34 am

Hi
for a moment I thought I had the solution, but that kb has already been applied to my hyper-v server.
I have a temporary solution, I have created a script that restarts the virtual machine management service of hyper-v, 3 minutes before the Veeam Backup task, with this temporary solution, I avoid that the avhdx remain un-merged, and the Backups jobs are done correctly.

cruggeri
Novice
Posts: 4
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri » Jun 11, 2018 7:42 pm

Interesting Gorka....has this continued to work? I have been restarting the VMMS service after the job creates the checkpoint for about 2 weeks now, but if i can add a startup script to the job itself that would eliminate any manual effort.

gorka
Enthusiast
Posts: 38
Liked: 4 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka » Jun 11, 2018 8:09 pm

hi Cruggeri, I still use the script (.bat) 2 minutes before the backup task, I have it with the windows task scheduler and I have it execute a .bat file with the following command: net stop vmms && net start vmms . I've been using it for days and it never fails. I have configured it with the windows task scheduler, but it could also be configured with the option that Veeam Backup has for pre-script and post-script, I leave this link where it explains where to configure these parameters.
https://aprendiendoavirtualizar.com/vee ... t-timeout/

gorka
Enthusiast
Posts: 38
Liked: 4 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka » Jul 03, 2018 8:48 am

hi people,
I think the problem with the merge of vhdx is solved with this update KB4284833,I tried to do a backup job with Veeam and when finished I made the merge correctly without restarting the virtual machine manager of hyper-v, I will confirm it tomorrow

Regards

Link
Novice
Posts: 3
Liked: never
Joined: Jun 11, 2013 7:49 am
Full Name: Leigh Sleight
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Link » Jul 17, 2018 5:13 am

I dont see anything in KB424833 that indicates its resolved any issues with change block tracking in hyper-v 2016.
I only have this running in a production environment, that I am not able to test in. I cant afford another serious failure on my cluster.
I am about to downgrade to 2012 R2, and rebuild all my virtual machines with the previous VM version. I even have web servers with hundreds of IP addresses that will need re-adding and binding again, and yes I am willing to go through the pain of downgrading back to 2012 R2 to just get stability back.

Can anyone confirm if this issue has been resolved yet please?


Many thanks

Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Jul 24, 2018 3:18 pm

I just spent 3 days recovering from "orphaned" avhdx files. Last night when I was getting ready to power up the VM's on my HyperV 2016 host, I checked for updates. Lo and behold, the 2018-07 Cumulative Update was available. I ran it.

Here's a link: https://www.catalog.update.microsoft.co ... =KB4338814

I then made a copy of my smallest 2016 Server and configured it as a test vm with no network (so as to not confuse the entire network of course!).

I ran backups every 5 minutes until I had 20 successful backups.

I then disabled that backup and ran tests of the production machine. I ran those all night and have 13 good copies and no .avhdx files hanging out.

I have since enabled the other 2 production backups for the VMs on this 2016 host. They will run today at 1230, 1530, and 1830 local. I will report results here.

This is the furthest I've gotten with this issue, so I am encouraged yet skeptical. I'm waiting for the other shoe to drop and to have more failures.

Tick tock.

My case number is 03103645.

Kuhnster

Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Jul 24, 2018 9:08 pm

So the backups of the production servers worked normally on the 2 smaller VM's, but the 2.2TB SQL server failed to remove the .avhdx files, but the checkpoints are not shown in HyperV Manager. Same problem. Discussion with Veeam and a look at the event log shows the following:

'SERVERNAME' background disk merge failed to complete: Account restrictions are preventing this user from signing in. For example: blank passwords aren't allowed, sign-in times are limited, or a policy restriction has been enforced. (0x8007052F). (Virtual machine ID AE580C5A-3DCC-4D4B-A395-E9146DF8DF53)

The backup was stuck at 99 percent for a long time, perhaps 1/2 hour prior to completion, which is when I saw the .avhdx files still in the folder. It does appear to be a timeout or credential issue as I have been using a domain admin account for the credentials when adding the domain-joined hypervisor (that's the way the client wants it).

Veeam suggested I edit the properties of the hypervisor and use the local administrator account, then fix the disks and back up the machine again to see if it works this time.

3 nights in a row ...

Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster » Jul 25, 2018 11:33 am

UPDATE - Luckily when I shutdown the SQL server to edit the disks, the checkpoints merged automatically, so I didn't have that nightmare to haunt me.

What I changed: I edited the hypervisor in Veeam and added the .\administrator credentials. I also set the SQL job to run every two hours, round the clock and increased my retention points to 360 so I have a month locally. I'm hoping doing s so will alleviate the timeout issue and the credential issue all in one fell swoop. My backups will also be much faster as the amount of change should be less for each job run.

jazzoberoi
Enthusiast
Posts: 79
Liked: 16 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by jazzoberoi » Jul 31, 2018 2:08 am

Jeez, 2 years on and the issues are still going on !

Sometimes with REFS and sometimes with Hyper-V itself !!

Can anyone please confirm if the update (July 24, 2018—KB4338822 (OS Build 14393.2395)) fixes all the merge problems with Hyper-V 2016 cluster ?

So i can finally upgrade my 2012R2 cluster to 2016 :)

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests