Host-based backup of Microsoft Hyper-V VMs.
Post Reply
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Advice on backing up CSV with hardware VSS writer

Post by rskinner »

I have a four node Hyper-V 2012 R2 cluster with two CSVs connected to an EqualLogic SAN. I'm backing this up with Veeam 7 using an off host proxy and the EqualLogic hardware VSS writer.

I have four jobs backing up the VMs on the cluster. If I manually run jobs one at a time they work fine, but if I try to schedule the jobs to run even 30 mins apart I'm having issues as Veeam is trying to take more than one VSS snapshot at a time. This results in the error "..Another shadow copy creation is already in progress. Wait a few moments and try again..." The job does try again later and it runs successfully using the Microsoft CSV VSS Shadow Copy Provider.

My question is, what's the best way to configure the backups so I can always use the hardware VSS writer? Should I chain the jobs together so only one job runs at a time? Or should I use the manage volumes option to configure only one concurrent snapshot? Or is there another option?

Thanks
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by Vitaliy S. »

Hello,

In order to always use hardware VSS provider, you can limit the max number of concurrent tasks on the proxy server. This value should be less than the number of max VSS snapshots allowed on the volume.

Thank you!
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner »

Thanks I'll give it a go.
Mrdoff
Novice
Posts: 5
Liked: never
Joined: Mar 16, 2014 6:12 pm
Full Name: Ola Haraldsson
Contact:

[MERGED] : Another shadow copy creation is already in progre

Post by Mrdoff »

Hi,

This week installed Veeam B&R.

We did run into some problem and I will try to explain it below:

We have 2 clusters. One Hyper-V 2012 Cluster with 4 nodes and one Hyper-V 2012R2 cluster with 3 nodes. We have connected Veeam to the clusters themself and not to our VMM server. We are using the onhost proxy method. We have installed EMC VSS provider on each host and configured our EMC VNX 5300 SAN. We can take snapshots with the EMC VSS provider but we have noticed that several jobs end with a Warning or Errors due to the EMC VSS provider was busy with another VSS job. We had the same problem when only using on Microsoft VSS provider.
We have also seen snapshots not being deleted after backup completes.

I have tried to change the number of “Max concurrent snapshots” for the CSVs but it doesn’t help. I changed it from 4(Default) to 8. See error message below:

Code: Select all

2014-03-14 13:14:45 :: Unable to create snapshot (EMC VSS Hardware Provider) (mode: Veeam application-aware processing). Details: Unknown status of async operation
Another shadow copy creation is already in progress. Wait a few moments and try again.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.
Have we missed something in our configuration? It feels like this would be a common issue otherwise?

If we change our design and go for offhost-proxy, can we connect a offhost proxy running 2012 R2 to both the 2012 cluster and the 2012R2 cluster?

Any ideas or solutions to our problem?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by veremin »

Have you tried to play with the max number of concurrent tasks of the proxy server in order to get rid of "Another shadow copy creation is already in progress" warnings? Thanks.
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner »

I've configured the off host proxy to run 8 concurrent jobs based on two CSVs and a maximum of 4 concurrent snapshots per CSV as shown in the "manage volumes" dialog.

However, looking at the job history and the time at which the snapshots of the VMs were taken, I think the problem is caused by Veeam triggering a VSS snapshot of a VM in one job, and triggering a snapshot of a VM in another job before the first snapshot has finished being created. The problem doesn't occur if you only run one job at a time as the snapshots of the VMs are taken sequentially. The problem only seems to occur when there's more than one job backing up VMs on the same CSV and the jobs are running at the same time.

At the moment I have chained my jobs together and this as stopped the problem as the snapshots are taken sequentially in each job.
Mrdoff
Novice
Posts: 5
Liked: never
Joined: Mar 16, 2014 6:12 pm
Full Name: Ola Haraldsson
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by Mrdoff »

I have tried to change the max concurrent task on both the proxy and the CSV to 1 but I still have the same problem. The only way for me to get a 100% successful job is when i disable parallel processing.
This is not a option for me because I need to backup over 200 VMs.

My test job with 5 VMs take over 30 minutes when parallel processing is disabled. And it's only backing up changes since last backup. Last backup was 1,8 GB data transferred.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] : Another shadow copy creation is already in pr

Post by foggy »

There's a registry key (HyperVRestrictConcurrentSnapshotCreation) that, being set to 1, locks the cluster at the time of snapshot creation thus not allowing more than one CSV snapshot to be created within the cluster at any given moment in time. Should allow to avoid mentioned issues.
Mrdoff wrote:If we change our design and go for offhost-proxy, can we connect a offhost proxy running 2012 R2 to both the 2012 cluster and the 2012R2 cluster?
Versions of the Hyper-V host and offhost backup proxy should coincide.
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner »

Please can you point me in the direction of some information on the HyperVRestrictConcurrentSnapshotCreation key and I'll give it a go.

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by veremin »

This key, as well as, all other VB&R registry keys should go to the standard HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication hive. Thanks.
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner »

So to confirm, I create a REG_DWORD key called HyperVRestrictConcurrentSnapshotCreation in the [HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication] hive on the Veeam server and set it to a value of 1 . Once I've created the key do I need to restart any services?

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by veremin »

The service restart isn't required. Though, you can do it in order to be on the safe side. Thanks.
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner »

The registry key has fixed the problem whereby more than one snapshot was created at a time. However, I now get errors from the EqualLogic SAN because the Veeam off host proxy tries to connect to the snapshot before the Hyper-V host has closed its connection to the snapshot. This results in the error:

"iSCSI login to target '...' from initiator '...' failed for the following reason: Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target."

This doesn't happen for every VM backup within the jobs, it only happened for one VM out of 13, but I never got this error when chaining the jobs one after the other.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by foggy »

This should not be the case, since all operations are asynchronous and Veeam B&R should wait for the snapshot creation to be completed and snapshot to be released. I recommend contacting support directly to start logs investigation.
Mrdoff
Novice
Posts: 5
Liked: never
Joined: Mar 16, 2014 6:12 pm
Full Name: Ola Haraldsson
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by Mrdoff »

rskinner wrote:The registry key has fixed the problem whereby more than one snapshot was created at a time. However, I now get errors from the EqualLogic SAN because the Veeam off host proxy tries to connect to the snapshot before the Hyper-V host has closed its connection to the snapshot. This results in the error:

"iSCSI login to target '...' from initiator '...' failed for the following reason: Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target."

This doesn't happen for every VM backup within the jobs, it only happened for one VM out of 13, but I never got this error when chaining the jobs one after the other.
Hi rskinner,

May I ask you how long time it takes for your HW VSS provider to create the snapshot? I'm using EMCs VSS provider and it takes often more then 3 minutes to create the snapshot.

Best Regards
Ola
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner »

To answer your question it takes about 3-4 mins to create the snapshot. However, this is getting more interesting as I investigate further. Over the last few days I've run backups with the registry key set to only create on snapshot at a time. Today I've had instability issues with my VMs exactly as described in this post here http://forums.veeam.com/microsoft-hyper ... 19577.html

I'm getting the same error

Code: Select all

 "[i]A component on the server did not respond in a timely fashion. This caused the cluster resource 'VM NAME' (resource type 'Virtual Machine', DLL 'vmclusres.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.[/i]"  
When I look through my cluster logs I can see that this error has been occurring for the last few days, but I hadn't realized. I hadn't realized that the VMs were failing because this is a non-production cluster and I don't have monitoring enable yet to tell me that the VMs have failed and the errors occur in the evening, after the backups, when I'm not working with the system. When I look through my cluster logs I can see that the error was happening for a few days when I first installed Veeam before I chained my backups together.

Having read the post I've linked to above I'm wondering if running backups in parallel with ODX causes the instability. I've now set my backups back to how they were, i.e. chained together and I've deleted the snapshot registry key. It'll be interesting to see if the instability goes away. I'll monitor the logs over the next few days and post back my findings.
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by collinp »

With 2012 R2 I have found it much more reliable to not use Hardware VSS and just use the Microsoft VSS provider. I limit all CSV's to 1 concurrent snapshot in the Veeam software and each host to 1 backup at a time. The backups don't take 3 minutes to coordinate the snapshot like they do with Hardware VSS so it ends up being faster than using offhost. And you don't have to manually cleanup snapshots this way. With 2012 and below, I still do offhost due to cluster stability issues.
rskinner
Influencer
Posts: 19
Liked: 10 times
Joined: Feb 11, 2014 3:05 pm
Full Name: Richard Skinner
Location: London, UK
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by rskinner » 1 person likes this post

Update.

Having disabled ODX the cluster instability issues have stopped. I'll continue to monitor the backups over the next few days and report back again. I'll list my configuration as it may help other EqualLogic and hardware VSS users.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Advice on backing up CSV with hardware VSS writer

Post by Gostev »

rskinner wrote:Having disabled ODX the cluster instability issues have stopped.
As always :( very sad for ODX: great technology with very poor stability at this time.
Justin
Novice
Posts: 7
Liked: never
Joined: Apr 22, 2015 7:27 pm
Contact:

[MERGED] Backup fail - Another shadow copy creation in progr

Post by Justin »

Case # 01810951


I am taking to the forums to ask for help as I am getting one or two email suggestions trickling in from support a day, with no luck yet after a full business week.

I have a 3par storage array connected via FC, a dedicated off host proxy, a 5 node hyper-v cluster, and a StoreOnce backup appliance.

The 5 hyper-v nodes as well as the proxy have the 3par CLI and VSS Hardware provider installed.

When I select multiple VMs for backup at once, I get an error - screen shot located here - http://i.imgur.com/Bml6UWw.jpg

If I run the jobs individually they're all successful which indicates a proper configuration. If I run more than once at once they will typically wait for a retry then complete successfully which also makes me think the configuration is in good shape. I am unsure as to where the hold up is at this point but I find it incredibly unlikely that I cannot backup multiple VMs at once. If that's the case how are the folks with 100s of vms completing their backups????
Post Reply

Who is online

Users browsing this forum: No registered users and 28 guests