Backing up a Windows Failover Cluster with Shared vhdx?

[Andreas Neufert]

great... thanks again

[DaveZ]

I have guest type failover cluster based on 2 VMs working on Windows Server 2012 R2 with shared VHDX as a cluster storage.
I wonder how to correctly create Veeam B&R backup job for this cluster?
Should I create two separated jobs, one for NODE1 and another backup job for NODE2?
When I created one backup job for both nodes, then it is throwing error each time for second server:

Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Veeam application-aware processing) Details: Unknown status of async operation
The shadow copy provider had an error. Check the System and Application event logs for more information.

After fail, job is reiterated, and finish with success on second node.

What is the best practice for that case?

I'm using Veeam B&R 9.0.0.1715

[foggy]

Hi Dave, please review this thread for information on backing up Windows failover clusters with shared VHDX. Basically, these disks can be backed up in a crash-consistent state only.

[response3]

Foggy - We're late to the conversation, but what does "crash-consistent" state mean exactly?

We've got a customer with a SQL 2012R2 failover cluster that we're trying to figure out a solution for. My hope was to use Veeam to backup the VM and application, but if that won't work then just use Veeam for the OS and something else for the application-aware backups. Thoughts?

[Andreas Neufert]

Crash consistent means that the filesystem on the shared disk is not in consistent mode. Like you pulling the powerplug from the server.
It is better than nothing but not ideal.

The main problem is that Microsoft do not give us a chance to bring the shared disk in a consistent state.

[bjackson@agi.com]

I know this is an old post, but I'm having a problem with taking a crash-consistent backup with Veeam. I'm still learning and evaluating Veeam(using Free v9.5 B&R) against a Server 2016 Hyper-V cluster and a Server 2012 R2 VM with a shared VHDX.

From reading through this form, it sounds like I only need to check the box to Disable guest quiescence in the VeeamZip window. Is that correct, or do I have something wrong on how to take the crash-consistent backup?

Lines from the Veeam log.
[06.12.2016 16:39:24] <01> Error Processing VMs with shared VHDX is not supported (System.Exception)
[06.12.2016 16:39:24] <01> Error at Veeam.Backup.Core.CHvTasksFactory.CreateTasksForVmEntry(CVmTaskBlank6 taskSpec, CGuestProcessingOptions effectVssOptions, Guid oijId, CHvDbHost currOwnerHost, CHvSharedSnapshotContextHolder snapshotContextHolder)
[06.12.2016 16:39:24] <01> Error at Veeam.Backup.Core.CHvTasksFactory.TryBuildTask(CVmTaskBlank6 taskBlank, CHvDbHost currOwnerHost, CHvSharedSnapshotContextHolder snapshotContextHolder, List`1 tasks, Object lockObj)

[Vitaliy S.]

Hi Brad,

The behavior you see is expected, as Windows Server 2016 has introduced a new framework for doing backups, thus old VHDX files cannot be processed any longer. To resolve this issue you need to follow one of the solutions:

1. You need to start using VHD set disks. See these two posts for more info: Shared VHDX In Windows 2016: VHDS and the backing storage file and What are VHD Sets in 2016?
2. Downgrade to Windows Server 2012 to continue using VHDX files in the backup job.

Thanks!

[bjackson@agi.com]

Hi Vitality,

Thank you for the notes about VHD Set disks. I setup a test set of servers with the same config(Hyper-V Hosts on Server 2016 and VM Guest on Server 2012). However, it still throws an error when I have the shared disk using VHD Set. It's different this time. Maybe I can only use the VHD Set option if both Hyper-V and the guest are Server 2016?

Code: Select all

[13.12.2016 01:43:04] <10> Warning  [server01]  Failed to create VM recovery checkpoint (mode: Hyper-V child partition snapshot)
[13.12.2016 01:43:04] <10> Error    Job with id "5cd555df-a417-483b-b406-8026e9321f4f" was not found (System.Exception)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.CBackupJob.Get(Guid jobId)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.GetSnapshotConsistencyLevel(IList`1 snapDatas, Boolean& failoverOnCrashConsistent)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.CreateRecoveryCheckpointCollectionSnapshot(IList`1 snapDatas, Exception& exception)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.CreateRecoveryCheckpoint(Exception& exception)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointSnapshot.CreatingSnapshot()
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointSnapshot..ctor(CHvSnapshotAttemptCtx snapGroup, CHvVssConnectionCreatorSet vssConnCreators, CBackup backup)
[13.12.2016 01:43:04] <10> Error    Job with id "5cd555df-a417-483b-b406-8026e9321f4f" was not found (System.Exception)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.CBackupJob.Get(Guid jobId)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.GetSnapshotConsistencyLevel(IList`1 snapDatas, Boolean& failoverOnCrashConsistent)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.CreateRecoveryCheckpointCollectionSnapshot(IList`1 snapDatas, Exception& exception)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.CreateRecoveryCheckpoint(Exception& exception)
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointSnapshot.CreatingSnapshot()
[13.12.2016 01:43:04] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointSnapshot..ctor(CHvSnapshotAttemptCtx snapGroup, CHvVssConnectionCreatorSet vssConnCreators, CBackup backup)

[CarstenT]

Hello Brad,

as far as I know the backup feature if VHD Sets only works if the guest is Server 2016, too.
Do you have the resources to install a 2016 Guest Cluster and rerun the backup test?

I am very interested in this topic. At the moment we have 10 physical SQL Clusters which are backed up by Backup Exec. We would like to virtualize them and use the backup feature of Hyper-V 2016 VHD Sets with Veeam 9.5.

[Vitaliy S.]

Brad,

Yes, guest OS of the VM you're trying to back up has to be Windows Server 2016 too, however the issue you observe could be unrelated, as I see a completely different error message. I have forward this info to our QA team for confirmation.

Thanks!

[Vitaliy S.]

Yes, it has been confirmed that there is a bug for VeeamZIP functionality when using VHD Set disks. We are planning to address this in our next updates. Thanks for your help!

[bjackson@agi.com]

Vitality, thank you for your comments! Do you know if the fix will allow for 2012 R2 Guest VM's to be backed up by 2016 Hyper-V hosts that use either the "VHD Set" or the traditional shared VHDs?

Carsten, I do and will probably try it sometime in the next week or two. If I get around to testing it I'll be sure to post. However, all of my VMs that use clustering are running Server 2012 R2, which is what I'm concerned about. Server 2016 is not a support OS at the moment for the apps that I have vitualized and clustered.

Thanks,
Brad

[Vitaliy S.]

Vitaliy, thank you for your comments! Do you know if the fix will allow for 2012 R2 Guest VM's to be backed up by 2016 Hyper-V hosts that use either the "VHD Set" or the traditional shared VHDs?

Unfortunately, no, as this issue is unrelated to VHD Sets. The condition you're talking about is controlled by Microsoft APIs, which we cannot override or make edits to. Our QA team confirmed that VeeamZIP has an issue of backing up VHD sets when using Windows Server 2016 as a guest OS. Regular backup jobs are not affected by this.

[spgsit5upport]

So to use the backup feature of vhds(ets), BOTH host/guest must be ONLY Server 2016, right?

Does that also mean Veeam REPLICATION?

I am sure I have at least one application on file server 2012 R2 that would need to be upgraded for 2016 (if/and when)

sebus

[Vitaliy S.]

Yes, both in-guest OS and Hyper-V should be running Windows Server 2016 in order to back up VHD sets. Replication of these VMs is still not supported, as it is not possible to create checkpoints as restore points for these VMs.

[spgsit5upport]

So if replication is NOT supported then what is the point? Another technology that simply does NOT deliver? (never will or MS just did not manage on time?)

[Vitaliy S.]

Do not know MS plans, but yes, this is not supported. On a side note, if replication was possible in Veeam, would you want to store target VM in the same format or would rather have a VM replica but with a regular disk format?

[nmdange]

So if replication is NOT supported then what is the point? Another technology that simply does NOT deliver? (never will or MS just did not manage on time?)

Veeam Replication may not support it, but you can use native Hyper-V Replica.

[spgsit5upport]

@nmdange, say who? You because you tried it & it works or you because you read it somewhere?

Obviously it does NOT work as reported here

[spgsit5upport]

Do not know MS plans, but yes, this is not supported. On a side note, if replication was possible in Veeam, would you want to store target VM in the same format or would rather have a VM replica but with a regular disk format?

Same format please. No reason (that I can think of) to change it

[Vitaliy S.]

Ok, the reason why I suggested this is that since checkpoints cannot be created for these disks via UI, then restore points for VM replica are also not possible, however if target VM had disks with other format, then we would be one step closer to having replication feature for these VMs (sort of a workaround for MS limitation) .

[spgsit5upport]

But that would make it difficult for DR

For proper (as quick as possible) DR one would want replication to happen (as scheduled if/when)
So at DR site I always have a defined working copy of VMs, that can be switched on & working on as nothing hapened.

At which point format does NOT matter.

But once the conditions return to normal, I want to bring back the changes from DR site back to main site (and carry on working)

If the format is different, then it would be long process (correct me if I am wrong)

Seb

[Vitaliy S.]

But once the conditions return to normal, I want to bring back the changes from DR site back to main site (and carry on working)

If the format is different, then it would be long process (correct me if I am wrong)

Yes, this is the most tricky part, need to think on the solution, as currently I don't have any suggestions yet. The idea above was proposed just for the DR situation, as you do need to bring the production server live again.

[spgsit5upport]

That is true, but to bring it up, I can currently do VM restore from DR hosted local backup

But if replication was possible even with format change, it would be of course better then NOT existing at all!

[Vitaliy S.]

Yes, I agree with that and have passed this to our dev team already.

[ludok]

Hi, little question here..
I want to use 2012R2 Shared-Vhdx feature, but there is all the limitations included in 2012R2… Mostly the Backup concern.
Just for info (don’t think it’ll actually change anything), I have 2 DC:
•Primary one:
o1 cluster with 3 2012R2 nodes
oDell compellent SC9000 storage

•Secondary one (Online next month):
o1 cluster with 2 2012R2 nodes
oSecond SC9000
We are gonna use LiveVolume feature for the synchronous replication of both compellent. (So same LUN both sides).

I understand that I cannot use VSS backup on virtual machines with 2012R2 shared-vhdx, but isn’t there a way around this?
Recently saw that you had a Windows Agent in beta for VEB, is there any chance to integrate this in VB&R to backup this type of vm’s? (Won’t upgrade to 2016 before a while, but would really love to use this feature).

(The whole subject is about SOFS Cluster with a vm on each HV Cluster using Shared-vhdx for storing my UPD (User Profile Disks) for my RDS Farm.)

Any help is welcome

Thanks!

[DGrinev]

Hi Ludovic and welcome to the community!

Veeam Agent for windows doesn't protect cluster metadata, crash-consistent backup seems like the only option to do backups.
Please review this thread for additional information.

Thanks!

[Mike Resseler]

Hi Ludovic,

To add to Dmitry's answer

If I understand it correctly you want to backup the data on that shared VHDX which is a bunch of files (user profiles). In that case, I would indeed test VEB (soon VAW) on top of that server and backup the data on that shared disk. What can happen is that there is a failover and in that case you will be without backups for that moment (but then you still have the crash-consistent ones from the VM backup). But I assume a failover is something that you only want to have temporarily and not for a long period

[ludok]

Hi,

I would agree with you if it were a simple file share with a bunch of files but UPD are vhd's files/user mounted on the RDS), and microsoft told me I need VSS to backup those...
So this is a no go for me.
Seems I've to stick to a single server to backup it correctly.

But thanks for confirmation.

[Mike Resseler]

Ludovic,

Yes correct. You need VSS so VAW server edition would be the way to go here. (in a single server type of setup)

Mike