Host-based backup of Microsoft Hyper-V VMs.
Post Reply
cruggeri
Novice
Posts: 8
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri »

There was also an update for Hyper-V 2016 just released I saw in this post:

https://social.technet.microsoft.com/Fo ... rverhyperv

Over lunch, I applied the update suggested by ss above to my VM, applied the update referred to in the technet forum to the host, rebooted. So far, 3 backups and a successful merge of the checkpoints. Not "out of the woods" yet, so to pseak, as it had begun failing after about 36 hours one time...but...I am hopeful.
cruggeri
Novice
Posts: 8
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri »

and this morning I came in and see the AVHDX did not merge properly. So, still stuck.
pwm001
Lurker
Posts: 1
Liked: never
Joined: May 29, 2018 7:12 pm
Full Name: Philip W Michael
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by pwm001 »

Have you found a solution yet? I am having the same issue. I wonder if it's some sort of rights/permissions issue. I am using the free version of Veeam for my home lab and have a script that runs from Task Scheduler each night. I have been using this setup for about a year now and never had any problems. Lately I changed some admin passwords and created a new domain admin account so I could disable the default domain admin account. I use the new account to run the scripts and sometimes it works and sometimes I have the same issue as you. I can run the backups just fine one at a time from the Veeam B&S, but my script fails most of the time. I haven't tried restarting the VMMS to see if it will merge all the VMs, but I will try that the next time it happens. Like you, some of the VMs merge after shutting them down and some fail and I have to fix them manually if I can. Powershell commands don't fix them. You are the only other person I have found with the exact same issue. Just an FYI, my script runs at 4 AM on one host to backup 6 VMs on two hosts. At this time, there is no one logged in to the server.
cruggeri
Novice
Posts: 8
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri »

I have been out of office. It is still failing; however, we have a little more information.

While out, I had someone fill in for me. We had resolved to shutting down the machine, backing it up, then immediately restoring it. When my "fill in" attempted this for the 1st time, he was unable to restore and we discovered the Hyper-V virtual machine management service needed to be restarted (I had been rebooting the host as well).

Upon my return, I was attempting to restart the VMMS service to see if the backups would succeed, which they didn't; however, I did see that when the machine was shut down to run the backup, the snapshots immediately began to merge, actually, causing the backup to fail; but, everything was then merged.

So, starting last night, I watched the backup run and create the checkpoint. Once the checkpoint creation was completed and the backup began, I restarted the VMMS service. The backup was successful and the Avhdx files merged as expected. I repeated this again this morning, with the same result. It has only been twice now, and we have had issues after successfully running a day, so I don't know if this would continue to work, but it has so far.
gorka
Enthusiast
Posts: 43
Liked: 5 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka »

Hi,
the problem is hyper-v 2016, I am also suffering these last 2 weeks without finding a solution. Every day I have to combine vhdx manually, restart the administrator of virtual machines of hyper-v etc etc, I do not know what else to try, I'm dying
greg.haberek
Lurker
Posts: 2
Liked: never
Joined: Aug 31, 2017 1:15 pm
Full Name: Greg Haberek
Location: Sterling Heights, MI
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by greg.haberek »

I was having the same problem with these .avhdx files remaining after the backup checkpoints were removed. This started happening after running Windows Updates on the hosts on May 25, 2018 (bringing them up to kb4103723). Same symptoms as described previously: there are no checkpoints listed but there are .avhdx files sitting in the Virtual Hard Disks folder for the machine.

I would have to shutdown the virtual machine, manually merge each of the .avhdx file into its parent, and then re-attach the original .vhdx and restart the machine. The guests affected were running any of CentOS 6, CentOS 7, Server 2008 R2, and Server 2016. One thing that was strange, is that it always seemed to be the same guests with the leftover .avhdx files.

I finally stumbled upon kb4103720 which is not coming down through WSUS. I downloaded it manually from the Update Catalog as described in the article and installed it manually on each of my Hyper-V hosts yesterday, before my backups started for the night. This SEEMS to have resolved the issue as of this morning. There are no .avhdx files to be found on any of my machines.

What's strange, is that KB article doesn't specifically mention this issue in its release notes. The closest it comes is this:
kb4103720 wrote:
  • Addresses an issue that makes it impossible to revert to a virtual machine checkpoint. Reapplying the checkpoint fails with an error.
We don't use Veeam Backup for Hyper-V so I can confirm that is not what's causing the issue. (We do use Veeam Backup for Office 365.) We are (unfortunately) using NetVault Backup for a while longer until we can get off some legacy systems that Veeam does not support. For me, this issue seems to be related specifically to the update kb4103723 for Windows Server 2016. However, others seem to have been having this issue since before that update was released.

I'll keep a close eye on this and report back if the situation changes. For now I'm considering this resolved on my Hyper-V hosts.
gorka
Enthusiast
Posts: 43
Liked: 5 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka »

Hi
for a moment I thought I had the solution, but that kb has already been applied to my hyper-v server.
I have a temporary solution, I have created a script that restarts the virtual machine management service of hyper-v, 3 minutes before the Veeam Backup task, with this temporary solution, I avoid that the avhdx remain un-merged, and the Backups jobs are done correctly.
cruggeri
Novice
Posts: 8
Liked: never
Joined: May 21, 2018 3:18 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by cruggeri »

Interesting Gorka....has this continued to work? I have been restarting the VMMS service after the job creates the checkpoint for about 2 weeks now, but if i can add a startup script to the job itself that would eliminate any manual effort.
gorka
Enthusiast
Posts: 43
Liked: 5 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka »

hi Cruggeri, I still use the script (.bat) 2 minutes before the backup task, I have it with the windows task scheduler and I have it execute a .bat file with the following command: net stop vmms && net start vmms . I've been using it for days and it never fails. I have configured it with the windows task scheduler, but it could also be configured with the option that Veeam Backup has for pre-script and post-script, I leave this link where it explains where to configure these parameters.
https://aprendiendoavirtualizar.com/vee ... t-timeout/
gorka
Enthusiast
Posts: 43
Liked: 5 times
Joined: Apr 14, 2016 6:45 pm
Full Name: Gorka Izquierdo Bizkarrondo
Location: Pamplona-IRUÑA , Euskalherria
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gorka »

hi people,
I think the problem with the merge of vhdx is solved with this update KB4284833,I tried to do a backup job with Veeam and when finished I made the merge correctly without restarting the virtual machine manager of hyper-v, I will confirm it tomorrow

Regards
Link
Novice
Posts: 3
Liked: never
Joined: Jun 11, 2013 7:49 am
Full Name: Leigh Sleight
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Link »

I dont see anything in KB424833 that indicates its resolved any issues with change block tracking in hyper-v 2016.
I only have this running in a production environment, that I am not able to test in. I cant afford another serious failure on my cluster.
I am about to downgrade to 2012 R2, and rebuild all my virtual machines with the previous VM version. I even have web servers with hundreds of IP addresses that will need re-adding and binding again, and yes I am willing to go through the pain of downgrading back to 2012 R2 to just get stability back.

Can anyone confirm if this issue has been resolved yet please?


Many thanks
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

I just spent 3 days recovering from "orphaned" avhdx files. Last night when I was getting ready to power up the VM's on my HyperV 2016 host, I checked for updates. Lo and behold, the 2018-07 Cumulative Update was available. I ran it.

Here's a link: https://www.catalog.update.microsoft.co ... =KB4338814

I then made a copy of my smallest 2016 Server and configured it as a test vm with no network (so as to not confuse the entire network of course!).

I ran backups every 5 minutes until I had 20 successful backups.

I then disabled that backup and ran tests of the production machine. I ran those all night and have 13 good copies and no .avhdx files hanging out.

I have since enabled the other 2 production backups for the VMs on this 2016 host. They will run today at 1230, 1530, and 1830 local. I will report results here.

This is the furthest I've gotten with this issue, so I am encouraged yet skeptical. I'm waiting for the other shoe to drop and to have more failures.

Tick tock.

My case number is 03103645.

Kuhnster
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

So the backups of the production servers worked normally on the 2 smaller VM's, but the 2.2TB SQL server failed to remove the .avhdx files, but the checkpoints are not shown in HyperV Manager. Same problem. Discussion with Veeam and a look at the event log shows the following:

'SERVERNAME' background disk merge failed to complete: Account restrictions are preventing this user from signing in. For example: blank passwords aren't allowed, sign-in times are limited, or a policy restriction has been enforced. (0x8007052F). (Virtual machine ID AE580C5A-3DCC-4D4B-A395-E9146DF8DF53)

The backup was stuck at 99 percent for a long time, perhaps 1/2 hour prior to completion, which is when I saw the .avhdx files still in the folder. It does appear to be a timeout or credential issue as I have been using a domain admin account for the credentials when adding the domain-joined hypervisor (that's the way the client wants it).

Veeam suggested I edit the properties of the hypervisor and use the local administrator account, then fix the disks and back up the machine again to see if it works this time.

3 nights in a row ...
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

UPDATE - Luckily when I shutdown the SQL server to edit the disks, the checkpoints merged automatically, so I didn't have that nightmare to haunt me.

What I changed: I edited the hypervisor in Veeam and added the .\administrator credentials. I also set the SQL job to run every two hours, round the clock and increased my retention points to 360 so I have a month locally. I'm hoping doing s so will alleviate the timeout issue and the credential issue all in one fell swoop. My backups will also be much faster as the amount of change should be less for each job run.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by jazzoberoi »

Jeez, 2 years on and the issues are still going on !

Sometimes with REFS and sometimes with Hyper-V itself !!

Can anyone please confirm if the update (July 24, 2018—KB4338822 (OS Build 14393.2395)) fixes all the merge problems with Hyper-V 2016 cluster ?

So i can finally upgrade my 2012R2 cluster to 2016 :)
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

All in all, I think the problems with VMware have been much worse than the Hyper-V problems.
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

I have found out more in the last several days.

You may already know this, but turn off UAC on the hypervisor.

I also have a script running GPUPDATE /Force hourly on the Veeam Servers and all 3 hypervisors. We had previously experienced issues where we were unable to start a VM unless we ran GPUPDATE /Force on the hypervisor - this may explain the timeout issue indicated in the merge errors I have experienced.

I can't tell which update helped me get this working, but I'm running a backup of a test VM and will let it run for a week. If it succeeds, and only if it does, I will then add my production servers to VBR and stop using the trial version of VAFW.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by jazzoberoi »

Hi Guys,

can some try the July 24, 2018—KB4338822 (OS Build 14393.2395) - https://support.microsoft.com/en-au/help/4338822 update and see if it resolves all your Hyper-V / ReFS / Windows issues ?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Hi All,

Most likely it won't. We received word that there might be a fix for the issue with AVHDX and that it is being tested by various parties (including us, but I haven't received the results... Holidays you know ;-)). However, I'm pretty sure that it was too early for MSFT to include it in the July update. Unless they decided that it was tested enough
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

I have KB4338822 installed as of 7/25. With all the other changes I've made to my Veeam environment and to the 2016 Hyper-V host itself, it's hard to tell if the KB did the trick. I have had 2 test backups running every half-hour since 7/31 without failure.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Jeff,
Are you now talking about ReFS or the Shared Disk issue?
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

I'm talking about .avhd and .avhdx files and the inability to merge them after the backups, Mike.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler » 1 person likes this post

Jeff, hot from the press! https://support.microsoft.com/en-us/help/4230569/

This week (depending on resources) we will also launch our own KB with more explanation
Kuhnster
Influencer
Posts: 20
Liked: never
Joined: Dec 19, 2014 3:34 pm
Full Name: Jeff Kuhn
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Kuhnster »

Thanks Mike. I'm not using a cluster or shared disks, but it's nice to see MSFT acknowledging there's an issue.

All 3 of my VMs are backing up successfully now due to changes I listed previously.

Jeff
mmondor
Novice
Posts: 3
Liked: never
Joined: Oct 26, 2018 2:08 pm
Full Name: Mark Mondor
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mmondor »

Has anyone ever solved the problem of doing an AAP backup of Server 2012 (not R2) on Server 2016 hyper-v host failing with "Failed to create checkpoint Error: 32768"? This forum and my testing says that KB4072650 which updates the vss integration services to 6.3.9600.18907 does solve this, but is only available for R2. I cannot find it on Microsoft Update Catalog at all and I can't upgrade this server to R2 due to app requirement. My support case is: 03234335.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

When it fails, did you check inside the 2012 server VM what the state of the VSS writers is? Mostly you can see the problem there
aleeri
Novice
Posts: 6
Liked: never
Joined: Dec 14, 2017 1:47 pm
Full Name: Alexander Eriksson
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by aleeri »

Hi,

What is the current status of Hyper-V 2016 and Veeam backups?
We have a rather large Environment with about 1800 Vms running different Windows Server versions. 2008R2-2016
many with 24/7 SLA.

All Infrastructure clusters today are Hyper-V 2012R2 with Storage Spaces 2012R2.

I am in the process of installing two new Hyper-V 2016 clusters with Storage Spaces 2016 (not direct) and I do feel a bit concerned here with Hyper-V 2016.
We only do VM (snapshot) backups from Veeam with Hyper-V guest quiescence, no AAP enabled.

Is it safe to go for Windows Server 2016 Hyper-V for our new clusters?
If you build a brand new Hyper-V 2016 cluster what tweaks do you recommend? Disable UAC?


Best Regards
Alexander
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Alexander,

There is a major change in how backup is done in Hyper-V 2016. Instead of the old method, a complete new one is introduced (with their own CBT, called RCT mechanism).
If you deploy two new clusters, with all VM versions up-to-date then the new system will be used and it works rather well.

So I wouldn't start with any tweaks at this moment, especially since they are brand new and not upgrades. Is there a reason why you don't want to use AAP?
aleeri
Novice
Posts: 6
Liked: never
Joined: Dec 14, 2017 1:47 pm
Full Name: Alexander Eriksson
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by aleeri »

Hi Mike,

Still, "rather well" concerns me. Today we run backups every night on about 1400 VMs. We never had any concerns with disk merges failing.
Out of about 1400 vms we usually have about 5-7 different backups that might fail during the and for that we have a powershell script running in the morning that
fetches all failed vm-backups restarts the hyper-v-VM-backupagent and retry the backup again with 100% success rate.

We really don´t have the need for AAP backups, the Hyper-V guest quiescence is good enough for us.
Also we have different AD domains and physical network Infrastructure for tenant network and the datacenter/veeam network no
communication allowed between the networks. Also we run on-host proxies.

With 2012R2 mainstream support expired, installing two brand new clusters with 2012R2 seems wrong, but still it works!

Can anyone confirm that the avhdx merge issues with 2016 guest is resolved?
mmondor
Novice
Posts: 3
Liked: never
Joined: Oct 26, 2018 2:08 pm
Full Name: Mark Mondor
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mmondor »

Mike Resseler wrote: Oct 29, 2018 7:01 am When it fails, did you check inside the 2012 server VM what the state of the VSS writers is? Mostly you can see the problem there
Yes, the NTDS writer fails with a Nonretryable error, Status 11. It must be rebooted to resolve but fails every time a Veeam AAP job or a Hyper-V Production snapshot is taken. This is the problem with 2012 Domain Controllers on Hyper-v host that can be fixed in R2 using KB4072560 but this update is not available for 2012 non-R2 as far as I can tell. Can't even find the update via Microsoft Update Catalog.
Post Reply

Who is online

Users browsing this forum: No registered users and 31 guests