Backing up a Windows Failover Cluster with Shared vhdx?

[itsdandandan]

Hi All,

Does Veeam 9.5 support replication of VMs with shared vhdxs? I found some forums threads from like a year ago that say no as Hyper-V does not support taking checkpoints of these disks. Has anything changed since then? I believe these are now supported in Server 2016.

Regards,

[veremin]

Your post has been merged into existing discussion regarding the same matter. Kindly, check answers above.

[adruet]

@nmdange, say who? You because you tried it & it works or you because you read it somewhere?

Obviously it does NOT work as reported here

I can confirm it does not work.
We have a hyper-V cluster running Windows server 2016 running latest march CU.
On top of which we created a cluster with two VMs running Windows server 2016 with latest march CU with VHD Set (VHDS).

We are unable to backup them in a constistent manner. It is also impossible to take checkpoint from GUI or from CLI (we get an access denied):
[VmmsAsyncSnapshotTask::RunTask()@307] Caught class Vml::VmAccessDeniedException: Access is denied. (0x80070005)

We opened a case with MS Support who confirmed that during Snapshot creation it uses functions to verify it has full and privileged access to the .VHDS file. Those functions have a bug (or more it seeams that the namespaces exist in code but are not really implemented). Anyway, a fix is in progress and might be available in April (no official release date yet).

[Mike Resseler]

Alex,

This is actually good news! (Well, besides the fact that this bug is existing for so long now...) But it would be great if that bug finally gets fixed and we can take those backups. If I remember correctly, we have implemented the necessary code to support VHD set but because of the bug it never worked. So I am now crossing my fingers and hope it makes the next update. Will they warn you when that fix will be released? (I would like to know )

Thanks
Mike

[adruet]

The MS support engineer told us we should be warned when the fix is available (our case number with MS is: 117030715417023). If it is the case, I'll keep you posted.

[Mike Resseler]

Great!

Looking forward to the news! Many people on this forum will be happy to hear it also

[RGijsen]

We also have a MS ticket for this, 117040415552767. In a 'vanilla' 2016 setup with the RTM media, creating snapshots of VM's with VHDS works fine. However, after installing the cumulative updates that fails, and hence Veeam fails. MS are aware of the issue and there are multiple tickets now about this where they promise a fix. To I keep my fingers crossed for a while

[Mike Resseler]

Well, same question as for @Alex... If you have more information, there are many people (besides me ) that would like to hear when the fix is out

Thanks
Mike

[cotterj]

Any update on this topic? I'm having the same issue trying to backup a guest cluster with Hyper-V 2016 using VHD sets and Veeam 9.5. Are there any known workarounds? What is the KB update that causes the issues?

[jamie2497]

Also need an update on this, we are unable to backup our file server cluster at the moment as it uses the new VHDS format!

[Mike Resseler]

Unfortunately we still don't have news from Microsoft

[bjackson@agi.com]

@jamie2497 As an alternative, you could use the new v10 features to backup a file share directly. It would at least work as a band-aid for now.

[Mike Resseler]

Brad,

Unfortunately v10 is not out yet, we just announced it

[adruet]

Still no news from Microsoft on our side either :/

[jjacobs1]

Having the same issue. Any word on a workaround?

[Vitaliy S.]

Hi Joshua, what is the exact issue you're referring to? Did you check existing answers to the question in the topic?

[nismoau]

Did you check existing answers to the question in the topic?

What answers are you referring to

He's probably referring to the inability to back up VMs with shared VHDXes on a Windows Server 2016 Hyper-V cluster (or host). I'd assume this is still an issue for everybody. I think many of us are at this point assuming that the cause of Veeam not being able to do the aforementioned is due to the inability to create VM checkpoints on the VMs through the Microsoft tools. If that's not the case, let me know

I opened a Veeam ticket, and basically got told to pursue the MS checkpoint issue first then come back to Veeam.

See this MS forum for details.

https://social.technet.microsoft.com/Fo ... f=required

[RGijsen]

Ah, that's my post on Technet forum. I've been called this morning that finally the bug is found and identified, and a patch will be released most probably June 2017.

[nismoau]

Yeah, I'm RustyJusty on the MS forums.

[RGijsen]

Ah, that figures

[RGijsen]

Still no fix in the July cumulative update
At least not documented: https://support.microsoft.com/en-us/help/4025339

[Mike Resseler]

@RGijsen,

I learned already that with the new CU update system of MSFT they are not adding all items in the documentation. That being said, I haven't tested that update yet to see that there is a fix and I haven't received any communication about it either so I fear it is not in this one

[nismoau]

Our MS rep said the fix is released in August 16, 2017—KB4034661 (OS Build 14393.1613):
https://support.microsoft.com/en-us/help/4034661

which hopefully addresses the issues in forum https://social.technet.microsoft.com/Fo ... Clustering and https://social.technet.microsoft.com/Fo ... rverhyperv

and hopefully Veeam will also be able to leverage checkpoints to complete backups successfully.

[Mike Resseler]

Thanks for the update! I will have a look at it but give me a bit of time

[chris-au]

I'd been having the same issues with Hyper-V 2016 hostinga guest failover cluster (2012 R2) with the file server role using VHDS for the storage and attempting to backup it up using Veeam. I was getting errors about not being able to create snapshots.

I have deployed 4034661 to our Hyper-V hosts this weekend gone and it has partially improved the issue for me. I have VSS disabled in the backup job for the guest cluster nodes (have not tried with it enabled just yet) and have been able to backup one of the guest cluster nodes with Veeam, however the second node it tried to process failed with the same errors as Robert Gijsen posted above. So that update definitely changed something around snapshots of VMs with VHDS attached.

The only thing I have observed at present that may give any indication as to why one worked and the other didn't is that the one that worked is running on the Hyper-V host that is currently the "Current Host Server" for the Hyper-V cluster and the owner of the CSV.

[chris-au]

It seems I spoke to soon. When Veeam tried to backup the guest cluster VMs again tonight it failed on both with the same errors as shown in Robert's post.

I've also discovered that the system drive and the shared VHDS drives for the guest cluster VMs all now have snapshots, however Hyper-V management console does not report any for either of the guest cluster VMs and neither does Get-VMCheckpoint or Get-VMSnapshot. However under settings the system drives do state that they can't be edited because of checkpoints and the shared VHDS each have 2 avhdx files with different IDs.

Not sure how to merge the checkpoints if they're not being reported in Hyper-V or PowerShell.

I've also since read that a few people were saying that the guest VMs also need to be 2016 in order for the host-level backup of VHD Sets to work. Is it at all possible to still do crash consistent backups of guest cluster VMs running 2012 R2 with either shared VHDX or VHD Sets with Veeam? It worked really well with shared VHDXs and Hyper-V 2012 R2, however due to various reasons we needed to upgrade to Hyper-V 2016 and Veeam would no longer backup VMs with shared VHDXs and instead said you need to upgrade to VHD Sets.

[nmdange]

It is normal for there to be an avhdx file for vhd sets. In that situation it is not due to a snapshot, see https://blog.workinghardinit.work/2015/ ... rver-2016/

[chris-au]

I know it is normal for there to be AN avhdx file for each VHD Set, however there are now TWO avhdx files for each VHD Set

[nvolkmer]

Hi,

i've tried to implement the latest windows update to fix that issue for our envirnment. That was kb4034661.
In my case that breaks up the cluster config to sync correctly and stops replicating the storage.

I've opened up a topic in the microsoft forums if someone got a similar issue.
https://social.technet.microsoft.com/Fo ... Clustering

[acolada]

Thanks to wonderful 2016 hyper-v technology, I now have 6 avdhx files for every vhd set, after repeated Veeam backups. Some of them even succeeded.
The virtual system disk for every virtual node does not merge, unless I shut down the VM and remove the vhd set disks from it. After that it merges correctly the normal vhdx file.
No merging for avhdx generated by vhd set yet.

Also I have to change the node ownership for the physical csv holding the vhd set, to recover from "cannot open attachment" or "general access denied" error, when trying to get the contents of the vm, or trying to inspect the vhd set.