Advice on backing up CSV with hardware VSS writer

[rskinner]

I have a four node Hyper-V 2012 R2 cluster with two CSVs connected to an EqualLogic SAN. I'm backing this up with Veeam 7 using an off host proxy and the EqualLogic hardware VSS writer.

I have four jobs backing up the VMs on the cluster. If I manually run jobs one at a time they work fine, but if I try to schedule the jobs to run even 30 mins apart I'm having issues as Veeam is trying to take more than one VSS snapshot at a time. This results in the error "..Another shadow copy creation is already in progress. Wait a few moments and try again..." The job does try again later and it runs successfully using the Microsoft CSV VSS Shadow Copy Provider.

My question is, what's the best way to configure the backups so I can always use the hardware VSS writer? Should I chain the jobs together so only one job runs at a time? Or should I use the manage volumes option to configure only one concurrent snapshot? Or is there another option?

Thanks

[Vitaliy S.]

Hello,

In order to always use hardware VSS provider, you can limit the max number of concurrent tasks on the proxy server. This value should be less than the number of max VSS snapshots allowed on the volume.

Thank you!

[rskinner]

Thanks I'll give it a go.

[Mrdoff]

Hi,

This week installed Veeam B&R.

We did run into some problem and I will try to explain it below:

We have 2 clusters. One Hyper-V 2012 Cluster with 4 nodes and one Hyper-V 2012R2 cluster with 3 nodes. We have connected Veeam to the clusters themself and not to our VMM server. We are using the onhost proxy method. We have installed EMC VSS provider on each host and configured our EMC VNX 5300 SAN. We can take snapshots with the EMC VSS provider but we have noticed that several jobs end with a Warning or Errors due to the EMC VSS provider was busy with another VSS job. We had the same problem when only using on Microsoft VSS provider.
We have also seen snapshots not being deleted after backup completes.

I have tried to change the number of “Max concurrent snapshots” for the CSVs but it doesn’t help. I changed it from 4(Default) to 8. See error message below:

Code: Select all

2014-03-14 13:14:45 :: Unable to create snapshot (EMC VSS Hardware Provider) (mode: Veeam application-aware processing). Details: Unknown status of async operation
Another shadow copy creation is already in progress. Wait a few moments and try again.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Have we missed something in our configuration? It feels like this would be a common issue otherwise?

If we change our design and go for offhost-proxy, can we connect a offhost proxy running 2012 R2 to both the 2012 cluster and the 2012R2 cluster?

Any ideas or solutions to our problem?

[veremin]

Have you tried to play with the max number of concurrent tasks of the proxy server in order to get rid of "Another shadow copy creation is already in progress" warnings? Thanks.

[rskinner]

I've configured the off host proxy to run 8 concurrent jobs based on two CSVs and a maximum of 4 concurrent snapshots per CSV as shown in the "manage volumes" dialog.

However, looking at the job history and the time at which the snapshots of the VMs were taken, I think the problem is caused by Veeam triggering a VSS snapshot of a VM in one job, and triggering a snapshot of a VM in another job before the first snapshot has finished being created. The problem doesn't occur if you only run one job at a time as the snapshots of the VMs are taken sequentially. The problem only seems to occur when there's more than one job backing up VMs on the same CSV and the jobs are running at the same time.

At the moment I have chained my jobs together and this as stopped the problem as the snapshots are taken sequentially in each job.

[Mrdoff]

I have tried to change the max concurrent task on both the proxy and the CSV to 1 but I still have the same problem. The only way for me to get a 100% successful job is when i disable parallel processing.
This is not a option for me because I need to backup over 200 VMs.

My test job with 5 VMs take over 30 minutes when parallel processing is disabled. And it's only backing up changes since last backup. Last backup was 1,8 GB data transferred.

[foggy]

There's a registry key (HyperVRestrictConcurrentSnapshotCreation) that, being set to 1, locks the cluster at the time of snapshot creation thus not allowing more than one CSV snapshot to be created within the cluster at any given moment in time. Should allow to avoid mentioned issues.

If we change our design and go for offhost-proxy, can we connect a offhost proxy running 2012 R2 to both the 2012 cluster and the 2012R2 cluster?

Versions of the Hyper-V host and offhost backup proxy should coincide.

[rskinner]

Please can you point me in the direction of some information on the HyperVRestrictConcurrentSnapshotCreation key and I'll give it a go.

Thanks

[veremin]

This key, as well as, all other VB&R registry keys should go to the standard HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication hive. Thanks.

[rskinner]

So to confirm, I create a REG_DWORD key called HyperVRestrictConcurrentSnapshotCreation in the [HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication] hive on the Veeam server and set it to a value of 1 . Once I've created the key do I need to restart any services?

Thanks

[veremin]

The service restart isn't required. Though, you can do it in order to be on the safe side. Thanks.

[rskinner]

The registry key has fixed the problem whereby more than one snapshot was created at a time. However, I now get errors from the EqualLogic SAN because the Veeam off host proxy tries to connect to the snapshot before the Hyper-V host has closed its connection to the snapshot. This results in the error:

"iSCSI login to target '...' from initiator '...' failed for the following reason: Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target."

This doesn't happen for every VM backup within the jobs, it only happened for one VM out of 13, but I never got this error when chaining the jobs one after the other.

[foggy]

This should not be the case, since all operations are asynchronous and Veeam B&R should wait for the snapshot creation to be completed and snapshot to be released. I recommend contacting support directly to start logs investigation.

[Mrdoff]

The registry key has fixed the problem whereby more than one snapshot was created at a time. However, I now get errors from the EqualLogic SAN because the Veeam off host proxy tries to connect to the snapshot before the Hyper-V host has closed its connection to the snapshot. This results in the error:

"iSCSI login to target '...' from initiator '...' failed for the following reason: Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target."

This doesn't happen for every VM backup within the jobs, it only happened for one VM out of 13, but I never got this error when chaining the jobs one after the other.

Hi rskinner,

May I ask you how long time it takes for your HW VSS provider to create the snapshot? I'm using EMCs VSS provider and it takes often more then 3 minutes to create the snapshot.

Best Regards
Ola

[rskinner]

To answer your question it takes about 3-4 mins to create the snapshot. However, this is getting more interesting as I investigate further. Over the last few days I've run backups with the registry key set to only create on snapshot at a time. Today I've had instability issues with my VMs exactly as described in this post here http://forums.veeam.com/microsoft-hyper ... 19577.html

I'm getting the same error

Code: Select all

 "[i]A component on the server did not respond in a timely fashion. This caused the cluster resource 'VM NAME' (resource type 'Virtual Machine', DLL 'vmclusres.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.[/i]"  

When I look through my cluster logs I can see that this error has been occurring for the last few days, but I hadn't realized. I hadn't realized that the VMs were failing because this is a non-production cluster and I don't have monitoring enable yet to tell me that the VMs have failed and the errors occur in the evening, after the backups, when I'm not working with the system. When I look through my cluster logs I can see that the error was happening for a few days when I first installed Veeam before I chained my backups together.

Having read the post I've linked to above I'm wondering if running backups in parallel with ODX causes the instability. I've now set my backups back to how they were, i.e. chained together and I've deleted the snapshot registry key. It'll be interesting to see if the instability goes away. I'll monitor the logs over the next few days and post back my findings.

[collinp]

With 2012 R2 I have found it much more reliable to not use Hardware VSS and just use the Microsoft VSS provider. I limit all CSV's to 1 concurrent snapshot in the Veeam software and each host to 1 backup at a time. The backups don't take 3 minutes to coordinate the snapshot like they do with Hardware VSS so it ends up being faster than using offhost. And you don't have to manually cleanup snapshots this way. With 2012 and below, I still do offhost due to cluster stability issues.

[rskinner]

Update.

Having disabled ODX the cluster instability issues have stopped. I'll continue to monitor the backups over the next few days and report back again. I'll list my configuration as it may help other EqualLogic and hardware VSS users.

[Gostev]

Having disabled ODX the cluster instability issues have stopped.

As always very sad for ODX: great technology with very poor stability at this time.

[Justin]

Case # 01810951


I am taking to the forums to ask for help as I am getting one or two email suggestions trickling in from support a day, with no luck yet after a full business week.

I have a 3par storage array connected via FC, a dedicated off host proxy, a 5 node hyper-v cluster, and a StoreOnce backup appliance.

The 5 hyper-v nodes as well as the proxy have the 3par CLI and VSS Hardware provider installed.

When I select multiple VMs for backup at once, I get an error - screen shot located here - http://i.imgur.com/Bml6UWw.jpg

If I run the jobs individually they're all successful which indicates a proper configuration. If I run more than once at once they will typically wait for a retry then complete successfully which also makes me think the configuration is in good shape. I am unsure as to where the hold up is at this point but I find it incredibly unlikely that I cannot backup multiple VMs at once. If that's the case how are the folks with 100s of vms completing their backups????