Host-based backup of Microsoft Hyper-V VMs.
itsdandandan
Novice
Posts: 3
Liked: 1 time
Joined: Mar 28, 2017 2:47 am
Full Name: Daniel Swafford
Contact:

[MERGED] Veeam Replicas with Shared VHDXs

Post by itsdandandan »

Hi All,

Does Veeam 9.5 support replication of VMs with shared vhdxs? I found some forums threads from like a year ago that say no as Hyper-V does not support taking checkpoints of these disks. Has anything changed since then? I believe these are now supported in Server 2016.

Regards,
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by veremin »

Your post has been merged into existing discussion regarding the same matter. Kindly, check answers above.
adruet
Influencer
Posts: 23
Liked: 7 times
Joined: Oct 31, 2012 2:28 pm
Full Name: Alex
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by adruet »

spgsit5upport wrote:@nmdange, say who? You because you tried it & it works or you because you read it somewhere?

Obviously it does NOT work as reported here
I can confirm it does not work.
We have a hyper-V cluster running Windows server 2016 running latest march CU.
On top of which we created a cluster with two VMs running Windows server 2016 with latest march CU with VHD Set (VHDS).

We are unable to backup them in a constistent manner. It is also impossible to take checkpoint from GUI or from CLI (we get an access denied):
[VmmsAsyncSnapshotTask::RunTask()@307] Caught class Vml::VmAccessDeniedException: Access is denied. (0x80070005)

We opened a case with MS Support who confirmed that during Snapshot creation it uses functions to verify it has full and privileged access to the .VHDS file. Those functions have a bug (or more it seeams that the namespaces exist in code but are not really implemented). Anyway, a fix is in progress and might be available in April (no official release date yet).
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

Alex,

This is actually good news! (Well, besides the fact that this bug is existing for so long now...) But it would be great if that bug finally gets fixed and we can take those backups. If I remember correctly, we have implemented the necessary code to support VHD set but because of the bug it never worked. So I am now crossing my fingers and hope it makes the next update. Will they warn you when that fix will be released? (I would like to know ;-))

Thanks
Mike
adruet
Influencer
Posts: 23
Liked: 7 times
Joined: Oct 31, 2012 2:28 pm
Full Name: Alex
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by adruet » 1 person likes this post

The MS support engineer told us we should be warned when the fix is available (our case number with MS is: 117030715417023). If it is the case, I'll keep you posted.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

Great!

Looking forward to the news! Many people on this forum will be happy to hear it also
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by RGijsen »

We also have a MS ticket for this, 117040415552767. In a 'vanilla' 2016 setup with the RTM media, creating snapshots of VM's with VHDS works fine. However, after installing the cumulative updates that fails, and hence Veeam fails. MS are aware of the issue and there are multiple tickets now about this where they promise a fix. To I keep my fingers crossed for a while :)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

Well, same question as for @Alex... If you have more information, there are many people (besides me ;-)) that would like to hear when the fix is out

Thanks
Mike
cotterj
Lurker
Posts: 1
Liked: never
Joined: May 12, 2017 8:58 pm
Full Name: Joe Cotter
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by cotterj »

Any update on this topic? I'm having the same issue trying to backup a guest cluster with Hyper-V 2016 using VHD sets and Veeam 9.5. Are there any known workarounds? What is the KB update that causes the issues?
jamie2497
Novice
Posts: 9
Liked: never
Joined: Feb 11, 2016 9:10 am
Full Name: Jamie
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by jamie2497 »

Also need an update on this, we are unable to backup our file server cluster at the moment as it uses the new VHDS format!
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

Unfortunately we still don't have news from Microsoft :-(
bjackson@agi.com
Novice
Posts: 9
Liked: 1 time
Joined: Nov 23, 2016 7:41 pm
Full Name: Brad Jackson
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by bjackson@agi.com »

@jamie2497 As an alternative, you could use the new v10 features to backup a file share directly. It would at least work as a band-aid for now.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

Brad,

Unfortunately v10 is not out yet, we just announced it :-)
adruet
Influencer
Posts: 23
Liked: 7 times
Joined: Oct 31, 2012 2:28 pm
Full Name: Alex
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by adruet »

Still no news from Microsoft on our side either :/
jjacobs1
Lurker
Posts: 2
Liked: never
Joined: Jun 02, 2017 2:31 pm
Full Name: Joshua Jacobs
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by jjacobs1 »

Having the same issue. Any word on a workaround?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Vitaliy S. »

Hi Joshua, what is the exact issue you're referring to? Did you check existing answers to the question in the topic?
nismoau
Novice
Posts: 8
Liked: never
Joined: Jul 06, 2016 1:29 am
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by nismoau »

Vitaliy S. wrote: Did you check existing answers to the question in the topic?
What answers are you referring to :)

He's probably referring to the inability to back up VMs with shared VHDXes on a Windows Server 2016 Hyper-V cluster (or host). I'd assume this is still an issue for everybody. I think many of us are at this point assuming that the cause of Veeam not being able to do the aforementioned is due to the inability to create VM checkpoints on the VMs through the Microsoft tools. If that's not the case, let me know :)

I opened a Veeam ticket, and basically got told to pursue the MS checkpoint issue first then come back to Veeam.

See this MS forum for details.

https://social.technet.microsoft.com/Fo ... f=required
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by RGijsen » 1 person likes this post

Ah, that's my post on Technet forum. I've been called this morning that finally the bug is found and identified, and a patch will be released most probably June 2017.
nismoau
Novice
Posts: 8
Liked: never
Joined: Jul 06, 2016 1:29 am
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by nismoau »

Yeah, I'm RustyJusty on the MS forums.
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by RGijsen »

Ah, that figures :)
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by RGijsen »

Still no fix in the July cumulative update :(
At least not documented: https://support.microsoft.com/en-us/help/4025339
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

@RGijsen,

I learned already that with the new CU update system of MSFT they are not adding all items in the documentation. That being said, I haven't tested that update yet to see that there is a fix and I haven't received any communication about it either so I fear it is not in this one
nismoau
Novice
Posts: 8
Liked: never
Joined: Jul 06, 2016 1:29 am
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by nismoau »

Our MS rep said the fix is released in August 16, 2017—KB4034661 (OS Build 14393.1613):
https://support.microsoft.com/en-us/help/4034661

which hopefully addresses the issues in forum https://social.technet.microsoft.com/Fo ... Clustering and https://social.technet.microsoft.com/Fo ... rverhyperv

and hopefully Veeam will also be able to leverage checkpoints to complete backups successfully.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by Mike Resseler »

Thanks for the update! I will have a look at it but give me a bit of time ;-)
chris-au
Novice
Posts: 8
Liked: never
Joined: Nov 04, 2016 1:38 am
Full Name: Chris Gay
Location: Tasmania, Australia
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by chris-au »

I'd been having the same issues with Hyper-V 2016 hostinga guest failover cluster (2012 R2) with the file server role using VHDS for the storage and attempting to backup it up using Veeam. I was getting errors about not being able to create snapshots.

I have deployed 4034661 to our Hyper-V hosts this weekend gone and it has partially improved the issue for me. I have VSS disabled in the backup job for the guest cluster nodes (have not tried with it enabled just yet) and have been able to backup one of the guest cluster nodes with Veeam, however the second node it tried to process failed with the same errors as Robert Gijsen posted above. So that update definitely changed something around snapshots of VMs with VHDS attached.

The only thing I have observed at present that may give any indication as to why one worked and the other didn't is that the one that worked is running on the Hyper-V host that is currently the "Current Host Server" for the Hyper-V cluster and the owner of the CSV.
chris-au
Novice
Posts: 8
Liked: never
Joined: Nov 04, 2016 1:38 am
Full Name: Chris Gay
Location: Tasmania, Australia
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by chris-au »

It seems I spoke to soon. When Veeam tried to backup the guest cluster VMs again tonight it failed on both with the same errors as shown in Robert's post.

I've also discovered that the system drive and the shared VHDS drives for the guest cluster VMs all now have snapshots, however Hyper-V management console does not report any for either of the guest cluster VMs and neither does Get-VMCheckpoint or Get-VMSnapshot. However under settings the system drives do state that they can't be edited because of checkpoints and the shared VHDS each have 2 avhdx files with different IDs.

Not sure how to merge the checkpoints if they're not being reported in Hyper-V or PowerShell.

I've also since read that a few people were saying that the guest VMs also need to be 2016 in order for the host-level backup of VHD Sets to work. Is it at all possible to still do crash consistent backups of guest cluster VMs running 2012 R2 with either shared VHDX or VHD Sets with Veeam? It worked really well with shared VHDXs and Hyper-V 2012 R2, however due to various reasons we needed to upgrade to Hyper-V 2016 and Veeam would no longer backup VMs with shared VHDXs and instead said you need to upgrade to VHD Sets.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by nmdange » 1 person likes this post

It is normal for there to be an avhdx file for vhd sets. In that situation it is not due to a snapshot, see https://blog.workinghardinit.work/2015/ ... rver-2016/
chris-au
Novice
Posts: 8
Liked: never
Joined: Nov 04, 2016 1:38 am
Full Name: Chris Gay
Location: Tasmania, Australia
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by chris-au »

I know it is normal for there to be AN avhdx file for each VHD Set, however there are now TWO avhdx files for each VHD Set
nvolkmer
Novice
Posts: 3
Liked: never
Joined: Jul 20, 2017 7:32 am
Full Name: Nils Volkmer
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by nvolkmer »

Hi,

i've tried to implement the latest windows update to fix that issue for our envirnment. That was kb4034661.
In my case that breaks up the cluster config to sync correctly and stops replicating the storage.

I've opened up a topic in the microsoft forums if someone got a similar issue.
https://social.technet.microsoft.com/Fo ... Clustering
acolada
Lurker
Posts: 1
Liked: never
Joined: Sep 05, 2017 6:42 am
Full Name: acolada
Contact:

Re: Backing up a Windows Failover Cluster with Shared vhdx?

Post by acolada »

Thanks to wonderful 2016 hyper-v technology, I now have 6 avdhx files for every vhd set, after repeated Veeam backups. Some of them even succeeded.
The virtual system disk for every virtual node does not merge, unless I shut down the VM and remove the vhd set disks from it. After that it merges correctly the normal vhdx file.
No merging for avhdx generated by vhd set yet.

Also I have to change the node ownership for the physical csv holding the vhd set, to recover from "cannot open attachment" or "general access denied" error, when trying to get the contents of the vm, or trying to inspect the vhd set.
Post Reply

Who is online

Users browsing this forum: No registered users and 16 guests