Advice on backing up CSV with hardware VSS writer

Hyper-V specific discussions

Advice on backing up CSV with hardware VSS writer

Veeam Logoby rskinner » Fri Mar 14, 2014 9:23 am

I have a four node Hyper-V 2012 R2 cluster with two CSVs connected to an EqualLogic SAN. I'm backing this up with Veeam 7 using an off host proxy and the EqualLogic hardware VSS writer.

I have four jobs backing up the VMs on the cluster. If I manually run jobs one at a time they work fine, but if I try to schedule the jobs to run even 30 mins apart I'm having issues as Veeam is trying to take more than one VSS snapshot at a time. This results in the error "..Another shadow copy creation is already in progress. Wait a few moments and try again..." The job does try again later and it runs successfully using the Microsoft CSV VSS Shadow Copy Provider.

My question is, what's the best way to configure the backups so I can always use the hardware VSS writer? Should I chain the jobs together so only one job runs at a time? Or should I use the manage volumes option to configure only one concurrent snapshot? Or is there another option?

Thanks
rskinner
Influencer
 
Posts: 19
Liked: 10 times
Joined: Tue Feb 11, 2014 3:05 pm
Location: London, UK
Full Name: Richard Skinner

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby Vitaliy S. » Fri Mar 14, 2014 2:43 pm

Hello,

In order to always use hardware VSS provider, you can limit the max number of concurrent tasks on the proxy server. This value should be less than the number of max VSS snapshots allowed on the volume.

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby rskinner » Fri Mar 14, 2014 4:17 pm

Thanks I'll give it a go.
rskinner
Influencer
 
Posts: 19
Liked: 10 times
Joined: Tue Feb 11, 2014 3:05 pm
Location: London, UK
Full Name: Richard Skinner

[MERGED] : Another shadow copy creation is already in progre

Veeam Logoby Mrdoff » Sun Mar 16, 2014 6:27 pm

Hi,

This week installed Veeam B&R.

We did run into some problem and I will try to explain it below:

We have 2 clusters. One Hyper-V 2012 Cluster with 4 nodes and one Hyper-V 2012R2 cluster with 3 nodes. We have connected Veeam to the clusters themself and not to our VMM server. We are using the onhost proxy method. We have installed EMC VSS provider on each host and configured our EMC VNX 5300 SAN. We can take snapshots with the EMC VSS provider but we have noticed that several jobs end with a Warning or Errors due to the EMC VSS provider was busy with another VSS job. We had the same problem when only using on Microsoft VSS provider.
We have also seen snapshots not being deleted after backup completes.

I have tried to change the number of “Max concurrent snapshots” for the CSVs but it doesn’t help. I changed it from 4(Default) to 8. See error message below:

Code: Select all
2014-03-14 13:14:45 :: Unable to create snapshot (EMC VSS Hardware Provider) (mode: Veeam application-aware processing). Details: Unknown status of async operation
Another shadow copy creation is already in progress. Wait a few moments and try again.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.


Have we missed something in our configuration? It feels like this would be a common issue otherwise?

If we change our design and go for offhost-proxy, can we connect a offhost proxy running 2012 R2 to both the 2012 cluster and the 2012R2 cluster?

Any ideas or solutions to our problem?
Mrdoff
Novice
 
Posts: 5
Liked: never
Joined: Sun Mar 16, 2014 6:12 pm
Full Name: Ola Haraldsson

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby v.Eremin » Mon Mar 17, 2014 8:45 am

Have you tried to play with the max number of concurrent tasks of the proxy server in order to get rid of "Another shadow copy creation is already in progress" warnings? Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby rskinner » Mon Mar 17, 2014 9:20 am

I've configured the off host proxy to run 8 concurrent jobs based on two CSVs and a maximum of 4 concurrent snapshots per CSV as shown in the "manage volumes" dialog.

However, looking at the job history and the time at which the snapshots of the VMs were taken, I think the problem is caused by Veeam triggering a VSS snapshot of a VM in one job, and triggering a snapshot of a VM in another job before the first snapshot has finished being created. The problem doesn't occur if you only run one job at a time as the snapshots of the VMs are taken sequentially. The problem only seems to occur when there's more than one job backing up VMs on the same CSV and the jobs are running at the same time.

At the moment I have chained my jobs together and this as stopped the problem as the snapshots are taken sequentially in each job.
rskinner
Influencer
 
Posts: 19
Liked: 10 times
Joined: Tue Feb 11, 2014 3:05 pm
Location: London, UK
Full Name: Richard Skinner

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby Mrdoff » Mon Mar 17, 2014 2:58 pm

I have tried to change the max concurrent task on both the proxy and the CSV to 1 but I still have the same problem. The only way for me to get a 100% successful job is when i disable parallel processing.
This is not a option for me because I need to backup over 200 VMs.

My test job with 5 VMs take over 30 minutes when parallel processing is disabled. And it's only backing up changes since last backup. Last backup was 1,8 GB data transferred.
Mrdoff
Novice
 
Posts: 5
Liked: never
Joined: Sun Mar 16, 2014 6:12 pm
Full Name: Ola Haraldsson

Re: [MERGED] : Another shadow copy creation is already in pr

Veeam Logoby foggy » Tue Mar 18, 2014 4:49 pm

There's a registry key (HyperVRestrictConcurrentSnapshotCreation) that, being set to 1, locks the cluster at the time of snapshot creation thus not allowing more than one CSV snapshot to be created within the cluster at any given moment in time. Should allow to avoid mentioned issues.

Mrdoff wrote:If we change our design and go for offhost-proxy, can we connect a offhost proxy running 2012 R2 to both the 2012 cluster and the 2012R2 cluster?

Versions of the Hyper-V host and offhost backup proxy should coincide.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby rskinner » Tue Mar 18, 2014 9:39 pm

Please can you point me in the direction of some information on the HyperVRestrictConcurrentSnapshotCreation key and I'll give it a go.

Thanks
rskinner
Influencer
 
Posts: 19
Liked: 10 times
Joined: Tue Feb 11, 2014 3:05 pm
Location: London, UK
Full Name: Richard Skinner

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby v.Eremin » Wed Mar 19, 2014 8:20 am

This key, as well as, all other VB&R registry keys should go to the standard HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication hive. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby rskinner » Wed Mar 19, 2014 8:46 am

So to confirm, I create a REG_DWORD key called HyperVRestrictConcurrentSnapshotCreation in the [HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication] hive on the Veeam server and set it to a value of 1 . Once I've created the key do I need to restart any services?

Thanks
rskinner
Influencer
 
Posts: 19
Liked: 10 times
Joined: Tue Feb 11, 2014 3:05 pm
Location: London, UK
Full Name: Richard Skinner

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby v.Eremin » Wed Mar 19, 2014 8:59 am

The service restart isn't required. Though, you can do it in order to be on the safe side. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby rskinner » Thu Mar 20, 2014 9:04 am

The registry key has fixed the problem whereby more than one snapshot was created at a time. However, I now get errors from the EqualLogic SAN because the Veeam off host proxy tries to connect to the snapshot before the Hyper-V host has closed its connection to the snapshot. This results in the error:

"iSCSI login to target '...' from initiator '...' failed for the following reason: Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target."

This doesn't happen for every VM backup within the jobs, it only happened for one VM out of 13, but I never got this error when chaining the jobs one after the other.
rskinner
Influencer
 
Posts: 19
Liked: 10 times
Joined: Tue Feb 11, 2014 3:05 pm
Location: London, UK
Full Name: Richard Skinner

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby foggy » Thu Mar 20, 2014 2:17 pm

This should not be the case, since all operations are asynchronous and Veeam B&R should wait for the snapshot creation to be completed and snapshot to be released. I recommend contacting support directly to start logs investigation.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Advice on backing up CSV with hardware VSS writer

Veeam Logoby Mrdoff » Thu Mar 20, 2014 3:32 pm

rskinner wrote:The registry key has fixed the problem whereby more than one snapshot was created at a time. However, I now get errors from the EqualLogic SAN because the Veeam off host proxy tries to connect to the snapshot before the Hyper-V host has closed its connection to the snapshot. This results in the error:

"iSCSI login to target '...' from initiator '...' failed for the following reason: Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target."

This doesn't happen for every VM backup within the jobs, it only happened for one VM out of 13, but I never got this error when chaining the jobs one after the other.


Hi rskinner,

May I ask you how long time it takes for your HW VSS provider to create the snapshot? I'm using EMCs VSS provider and it takes often more then 3 minutes to create the snapshot.

Best Regards
Ola
Mrdoff
Novice
 
Posts: 5
Liked: never
Joined: Sun Mar 16, 2014 6:12 pm
Full Name: Ola Haraldsson

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 7 guests