Host-based backup of Microsoft Hyper-V VMs.
Post Reply
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

aleeri wrote: Oct 30, 2018 9:12 am Hi Mike,

Still, "rather well" concerns me. Today we run backups every night on about 1400 VMs. We never had any concerns with disk merges failing.
Out of about 1400 vms we usually have about 5-7 different backups that might fail during the and for that we have a powershell script running in the morning that
fetches all failed vm-backups restarts the hyper-v-VM-backupagent and retry the backup again with 100% success rate.

We really don´t have the need for AAP backups, the Hyper-V guest quiescence is good enough for us.
Also we have different AD domains and physical network Infrastructure for tenant network and the datacenter/veeam network no
communication allowed between the networks. Also we run on-host proxies.

With 2012R2 mainstream support expired, installing two brand new clusters with 2012R2 seems wrong, but still it works!

Can anyone confirm that the avhdx merge issues with 2016 guest is resolved?
Only merge issue I am aware of is if you are using Shared VHD Sets and backing up a guest cluster.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

NMdange is correct. If you are using guest clusters with shared VHD sets then you can have problems. If your clusters and hosts are fully patched, there should not be problems. I also advice you (now, with all patches :-)) to run a REFS repository with Veeam and even run REFS as your choice for your VM's. (The merge is really fast)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

@ mmondor

The latest integration components are https://support.microsoft.com/en-us/hel ... erver-2012
There are no new ones for 2012. So I don't think that the specific fix for 2012 R2 exists I'm afraid. Did you got already further with support?
AlexLeadingEdge
Veteran
Posts: 456
Liked: 58 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

I am still waiting for a fix for the "Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover)" from page one of this thread.

Will Microsoft ever fix this? Or are they waiting for Veeam to do it for them?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Alex,

We won't be able to fix it, mostly some sort of workaround. But keep working with support. It is probably already with the dev's and they might figure out a workaround if MSFT doesn't come with a fix
stevezemlicka
Novice
Posts: 6
Liked: never
Joined: Aug 10, 2018 6:05 am
Full Name: Stephen Zemlicka
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by stevezemlicka »

I am having this avhdx merge issue that just started as of Monday. We diligently keep all servers (hypervisors and VMs) up-to-date (monthly) so there was no new updates installed immediately prior to this issue (but they were up-to-date as of last month). We are setup with CSVs. I have started a case (#03289031). I can confirm that the get-vmsnapshot shows no snapshots but all the VM hard disks are pointed to avhdx files. If I shut down the VMs, nothing merges. If I try to move the VM to another host while shut down, it errors out and will not start. Moving back to the original host results in an error and it continues to not start.

The error that results from the failure to start is 1069. I have tried assigning permissions to the storage using icacls <Folder with VHDS> /grant “NT VIRTUAL MACHINE\<VM GUID>:(OI)F” with no change.

I have also attempted to disable the backup option in the integration with no changes resulting from that as well.

Regardless of the issues starting the VMs that I messed with, I'd like to figure out the Veeam backup issues which seems to be related to the differencing disk merge issue. I have started and continue to review this thread but am hoping that someone following this may be able to help give me a good direction to at least get the backups going again.
Speights
Service Provider
Posts: 2
Liked: never
Joined: Nov 24, 2013 10:44 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Speights »

We have had the same issue going on approx. 2 months now. Raised a ticket with Veeam they pointed us straight at Microsoft. Have had a support case open with M$ for 1 and 1/2 months to no available.
Brand new Dell Hardware firmware and drivers up to date
Freshly built with windows 2016
All Hosts are patched with latest windows updates
All VMs patched with latest updates.
VMs range from Linux to Windows Server 2016 guests happens on all VMs regardless of OS.
No GPOs applied to Hyper-V hosts
No Anti Virus installed on Hyper-V Hosts
Hyper-V role re-installed
Issue occurs on all 3 hosts they are all stand alone not clustered. Managed to replicate issue on host that has never had the Veeam agent installed so have verified it is a M$ issue (In our case).
The only way we can get the snapshots/Checkpoints to merge is by restarting the VMM service with a batch file just prior to the Veeam backup job runs as some others have done in this forum post. This is our only current work around but it is not reliable and we have to manually merge snapshots weekly when jobs over run it has also caused some vhdx corruption.

I will post back here if M$ actually find a permanent resolution for us but we are not holding our breath as they have been very slow to respond.

Dare I say it anyone tried Windows Server 2019?
stevezemlicka
Novice
Posts: 6
Liked: never
Joined: Aug 10, 2018 6:05 am
Full Name: Stephen Zemlicka
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by stevezemlicka »

While not a solution, I did finally work through my issue with some of the info in this forum as well as a bit of digging. I found that the VMs I was having trouble with which were unable to start were due to the differencing disk becoming disassociated with or having some issue with the parent disk attribute. I don't have a link since it's been some time, but eventually i was able to find a command that I could issue to manually set the parent disk for the differencing disk. Once that was done, I could move the VM to another server and then back and the differencing disk was merged into the parent.

After doing this to all VMs, everything seemed to be running OK again.

Rant Warning:
I think Veeam pointing the finger back to Microsoft and shrugging this issue off is handling issues like this very poorly. Sure, the issue might be a result of a bug with Microsoft's technology. But it is a technology that Veeam has chosen to leverage. Veeam clearly has the option to write their own snapshotting technology or use Microsoft's. Since they chose to save the R&D of developing their own, I think they are obligated to leverage their postition to get better answers from Microsoft when these situations arise on behalf of their customers. Though I'm not promoting Backup Exec or Dell, I know when I bought Backup Exec through Dell that Dell collaborated with Symantec (or vice versa) when I had an issue and, in the rare case that the issue had something to do with a specific Microsoft technology, they reached out to Microsoft and got us hot patches to address the issue because they obviously had better leverage than me as an individual. I get Veeam's position...but with this as well as some other issues recently that they seem to just shrug their shoulders at and say "dunno", I'm at a point where my recommendation for Veeam has slid from a emphatic recommendation to something like "well the product is decent but the support is mediocre. if you can find a product with good support, you should go that direction instead"
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Gostev » 4 people like this post

First, to get "own snapshotting technology" possibility out of this discussion right away: this is just not going to fly with Microsoft. Source: I've been building Windows management software my whole life, for almost 20 years now. Basically, going this route would make our product unsupported by Microsoft, which in turn renders the entire environments where it is used likewise unsupported by Microsoft. In fact, this was exactly what happened at my previous company, resulting in Microsoft KB article clearly stating the above, so I know how this is not acceptable for vast majority of customers. Just like with all other of their enterprise applications, Microsoft only supports a very specific way of backing up and restoring Hyper-V using the specific set of APIs they provide. So, it's not like we ever had a choice that you said we did make :D

Moreover, just to add some color to this, remember that we were the only vendor who did build our own changed block tracking technology for Hyper-V 2012. And despite Hyper-V 2012 did not have its own CBT mechanism, Microsoft was very clear that they're not happy with us running drivers in the parent partition. That is despite the fact there was no "official" API for CBT in 2012.

Second, in your post you're assuming that we did not reach out to Microsoft the moment these issues were reported - but this is simply not so? We had multiple support cases opened on them with Microsoft since the beginning of 2018. And while I already had to explain this in details before, when replying to similar "rant" on ReFS issues mega-thread (issues which took Microsoft 1.5 years to resolve, by the way) - but just to repeat here in short: the only way to raise the priority of some bug within Microsoft is to drive lots of customers to open support cases with them and declare major and measurable revenue impact from this particular bug. It's not until certain revenue loss threshold that they will go seriously after the particular bug with all other bugs they have outstanding. And being a product manager myself, I don't necessarily blame them for their methods, knowing they have 100 times more bugs and enhancements to prioritize than I do. Especially in the days when Hyper-V seems to have overall low priority for them as a company, with Azure and Azure Stack occupying their mind as IaaS platforms of the future.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

I think the general assumption is that Veeam has higher quality access to Microsoft support than Veeam's customers do. However, that's probably not true. When I asked Veeam support to take my issue to Microsoft instead of me, the rep described what the experience would be like and it sounded just as hellish as what I go through. It seems like Microsoft treats most everyone equally poorly.

Though I do occasionally read articles on Raymond Chen's blog about customer questions that somehow made it to him, a top engineer. So, I guess there must be some way to get good customer service out of Microsoft.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Gostev »

You're not incorrect, we do have the Premier TAP Support contract with Microsoft, which actually costs us much $$$ every year... however, as we've learnt based on first-hand experience dealing with ReFS stability issues, it's not a consideration for their bugs prioritization approach. But this makes sense, as I assume there are hundreds of other TAP partners like us, and thousands of key enterprise customers, all with the same highest level of support.
AlexLeadingEdge
Veteran
Posts: 456
Liked: 58 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

I have just installed the latest version of the Hyper-V Integration Components from here:
https://support.microsoft.com/en-in/hel ... l-machines

I downloaded it and installed it using these steps:
(1) Download the CAB Files x86 / X64 depending upon the OS architecture.
(2) Rename file to KBnumber.сab (for example KB3176931.Cab)
(3) Copy to folder C:\
(4) Run command-prompt as admin (Right click on Windows icon and choose option). ...
(5) Enter the command "DISM.exe /Online /Add-Package /PackagePath:c:\KBnumber.cab" (inside quotation marks)

I will see how the backup goes tonight.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

That link is to KB3063109, an older version of the integration components. 6.3.9600.18080 from Oct 2015.

The most recent I'm aware of is KB4072650. That's version 6.3.9600.18907 from Jan 2018. But it's only applicable to Windows Server 2012 R2 and windows Server 2008 R2.
karlbackups
Novice
Posts: 3
Liked: never
Joined: Mar 04, 2016 12:52 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by karlbackups »

I was advised to install latest Windows updates by Veeam support. Installed all of them on Backups servers and Hyper-V cluster servers on December 18th, problem still persist.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by wishr »

Hi Karl,

Which one you are talking about?

BR,
Fedor
AlexLeadingEdge
Veteran
Posts: 456
Liked: 58 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

AlexLeadingEdge wrote: Jan 08, 2019 2:33 am I have just installed the latest version of the Hyper-V Integration Components from here:
https://support.microsoft.com/en-in/hel ... l-machines

...

I will see how the backup goes tonight.
Sorry for the late reply. No improvement on the current situation.
Magz
Service Provider
Posts: 20
Liked: 1 time
Joined: Aug 08, 2012 9:06 am
Full Name: Magne Stenwig
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Magz » 1 person likes this post

I installed KB3063109 on one of my Windows 2008 R2 - guests on which backup failed with error 0x80042308 and rebooted.

The backup is now working without any errors:-)
AlexLeadingEdge
Veteran
Posts: 456
Liked: 58 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

I see on the guest machine (Server 2008 R2) that KB4072650 has just popped up in Windows Update (released 13/03/2018!!!), will install it tomorrow night (Friday).
AlexLeadingEdge
Veteran
Posts: 456
Liked: 58 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

The install of KB4072650 on the guest Server 2008 R2, which is on a Server 2016 host, is now backing up with no warning messages :)
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by wishr »

Hi Alex,

Thanks for sharing that with the community! Much appreciated!

BR,
Fedor
nitramz
Novice
Posts: 3
Liked: 1 time
Joined: Feb 04, 2015 2:04 pm
Full Name: Martin Zimmermann
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nitramz »

Does someone else still have the checkpoint merge issue?
We still have it on a non clustered disk. All latest updates applied.
Restarting vmms before the backup helps.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Martin,

Is this with update 4 installed?
nitramz
Novice
Posts: 3
Liked: 1 time
Joined: Feb 04, 2015 2:04 pm
Full Name: Martin Zimmermann
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nitramz » 1 person likes this post

This fixed the issue with vhds not merging after backup: https://support.microsoft.com/en-us/hel ... -with-erro
mlavie
Enthusiast
Posts: 28
Liked: 1 time
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mlavie »

derek wrote: Jul 07, 2017 5:45 pm I ran across this issue today and did some searching.

I found a post https://superuser.com/questions/1070464 ... 0x800423f4 Where IntelXX mentioned that if you set that VM's checkpointing to Standard instead of the new Production type it will work.
I was able to take a checkpoint using the HyperV interface after changing this.
I'm running a Veeam backup job now with the new setting for my 2008R2 DC VM.
I do not know if "me too" type posts are allowed/encouraged/tolerated here, but changing Checkpoint Type on my new Hyper-V 2019 from Production to Standard solved the AAP backup from jamming.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

It's not been my experience that changing the checkpoint type in Hyper-V Manager influences how Veeam does checkpoints during backup. I've spent a lot of time troubleshooting an issue with Veeam specifically linked to production checkpoints and changing the type in Hyper-V Manager didn't have any effect.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

The biggest problem what I see here is that standard is the "old" way of taking snapshots which is crash-consistent. Production is what is actually preferred.
I prefer that all of you who are suffering from this create a support ticket. It might be on our side, but it can also be on the Microsoft side that something is broken in server 2019 (although I cannot remember that from my testing).

Please, as always, post the support case ID's here and the outcome of the research of the engineers
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by rogerdu »

I am getting this on a Sharepoint server...

Environment:
Hyper-V 2016, Shared Storage Volumes

VM details:
Windows Server 2008 R2 Standard 64 bit
4/24/2019 10:51:09 AM :: Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'HOSTNAME' failed. (Virtual machine ID 8BFF7957-2D17-426C-AAC5-6A5CE9387B8E)

'HOSTNAME' could not initiate a checkpoint operation: %%2147754996 (0x800423F4). (Virtual machine ID 8BFF7957-2D17-426C-AAC5-6A5CE9387B8E)'). Error code: '32768'.
Failed to create VM recovery snapshot, VM ID '8bff7957-2d17-426c-aac5-6a5ce9387b8e'.

opened support case: 03528859
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by wishr »

Hi Roger,

Please try to create a production checkpoint manually as suggested by our support engineer. If it does not work - please open a case with Microsoft.

Thanks
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by rogerdu »

It would seem that once the systems were upgraded to VM hardware version 8, everything started working....

Then over the weekend, it occurred again... Guess I'll keep poking

Roger
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by rogerdu » 1 person likes this post

The Production Snapshots are setup and configured. On one of the 3 failing systems, the Backup Integration was not turned on for one of the servers. It is now working :-)

Still working the other 2.
Post Reply

Who is online

Users browsing this forum: No registered users and 32 guests