Comprehensive data protection for all workloads
Post Reply
ameyer
Novice
Posts: 9
Liked: never
Joined: Feb 14, 2012 3:44 pm
Full Name: Andrew Meyer
Contact:

Parallel Processing, VBK file locked error

Post by ameyer » Aug 19, 2013 2:50 pm

After doing an in-place upgrade from v6.5 to v7, everything went smoothly for the first few nights of backups. I then decided to enable Parallel Processing and have started to receive errors about the VBK file being in use: "The process cannot access the file because it is being used by another process". The backup job ultimately fails.

If I retry the job enough times, it will complete all VMs successfully. Nothing else would be accessing the VBK besides Veeam and this would be the only job running at that time.

Any thoughts on what I might be doing wrong?


Thanks,
Andy

Gostev
SVP, Product Management
Posts: 24797
Liked: 3540 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Parallel Processing, VBK file locked error

Post by Gostev » Aug 19, 2013 2:54 pm

You are not doing anything wrong, but some process is locking the backup files (most likely, some external process). Try Process Monitor on Veeam backup files, or open a support case. Thanks!

ameyer
Novice
Posts: 9
Liked: never
Joined: Feb 14, 2012 3:44 pm
Full Name: Andrew Meyer
Contact:

Re: Parallel Processing, VBK file locked error

Post by ameyer » Aug 19, 2013 3:13 pm

Thanks for the quick reply!

marcseitz
Influencer
Posts: 17
Liked: 5 times
Joined: Apr 04, 2012 11:17 am
Full Name: Marc Seitz
Contact:

[MERGED] : Jobs will fail while using parallel Processing

Post by marcseitz » Aug 29, 2013 1:58 pm

Hi Guys,

we've updated to Version 7.0 (of course, without problems).
We want to leverage from the parallel processing of VMs within a Job.

The Problem while using parallel processing is, that a random amount of VMs of a Job is failling with the following error message: (some Jobs are working perfect and without problems)
28.08.2013 23:49:01 :: Error: Client error: End of file
Exception from server: The process cannot access the file because it is being used by another process.
Failed to create or open file [<CIFS-SHARE\FOLDER\BACKUP>.vbk].
Failed to backup text locally. Backup: [veeamfs:0:048c2c67-a416-444d-861d-0793a96d19ed (vm-198919)\summary.xml@<CIFS-SHARE\FOLDER\BACKUP>V
I've seen this problem, when only one Job is running to process 5VMs (2 VMDKs per VM).
I can see, that the VM which is failing doesn't get any Proxy Server assigned. It's Createing the Snapshot, Releasing the Guest and Removing the Snapshot directly after this.

Did anybody of you see this problem too?

(Still working with the Support Guys: 00436068)

Thanks & Regards,
Marc

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 29, 2013 2:12 pm

Hi, Mark, as a first investigation step, kindly, check whether there are some external activities that might lock the backup files. Otherwise, you can just wait and see what support team suggest to you. Thanks.

marcseitz
Influencer
Posts: 17
Liked: 5 times
Joined: Apr 04, 2012 11:17 am
Full Name: Marc Seitz
Contact:

Re: Parallel Processing, VBK file locked error

Post by marcseitz » Aug 29, 2013 2:21 pm

Hi Vladimir,

thanks for Merging the Topic.
Of course I'm waiting, what the Support Guys can figure out - Just want to use every available channel ;-)

The used CIFS-Share is only used by Veeam B&R and no other opened Sessions / Files on this Share from other Applications.
I'll update you when there are any news.

Thanks,
Marc

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 29, 2013 2:54 pm

So far, there are only you two who have reported about this issue. And the op, in contrast to you, has never come back to update the topic with support ticket number or found resolution, so, we can’t be sure what might be the cause of this problem. Thanks.

marcseitz
Influencer
Posts: 17
Liked: 5 times
Joined: Apr 04, 2012 11:17 am
Full Name: Marc Seitz
Contact:

Re: Parallel Processing, VBK file locked error

Post by marcseitz » Aug 29, 2013 3:11 pm

Hi again,

just a question if you can tell me, how the traffic is flowing while parallel processing.
I've backed up 5VMs at the moment with serveral VMDKs, using several Backup Proxies.

On my Repository, I've only seen one connection from one Proxy.

Is it correct, that all Proxies will push the Traffic to the Proxy which is holding the lock on the File?
(So the Proxies which are not holding the Lock, use the "first touch Proxy" as Proxy too?

Thanks,
Marc

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 29, 2013 4:05 pm

Actually, it’s not a proxy server that communicates with .vbk file, but rather a target repository agent. So, it shouldn’t be the case.
On my Repository, I've only seen one connection from one Proxy.
The connection occurs between only pair of agents. That being said, in one time only one proxy server can contact repository. And in case of CIFS share, share, the communication takes place between proxy server and proxying server. Thanks.

mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: Parallel Processing, VBK file locked error

Post by mcwill » Aug 30, 2013 10:50 am

v.Eremin wrote:So far, there are only you two who have reported about this issue. And the op, in contrast to you, has never come back to update the topic with support ticket number or found resolution, so, we can’t be sure what might be the cause of this problem. Thanks.
We also hit this in testing V7 before deploying to Prod. Ticket raised was # 00429947.

Ours seemed to be caused by a slow cifs share, (An Apple Time Capsule of all things) and the problem went away when we changed to a different backup storage unit. As it was a test setup only we didn't push it and allowed the ticket to be closed once a workaround was discovered.

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 30, 2013 11:04 am

As far as I can see, the Marc’s case is also related to the usage of CIFS share. So, Mark, it might be worth changing temporarily target repository and seeing whether the said issue is caused by the network connectivity or by something else. Thanks.

marcseitz
Influencer
Posts: 17
Liked: 5 times
Joined: Apr 04, 2012 11:17 am
Full Name: Marc Seitz
Contact:

Re: Parallel Processing, VBK file locked error

Post by marcseitz » Aug 30, 2013 11:13 am

Hi Vladimir,

changing to another Repository is not possible in our Environment (>60TB Backup-Data) - sorry.
I don't think, it's a network connectivity related problem.
I can see, that (for example) 5 of 7 VMs will be processed correctly. So the Backup-File is locked by the Repository-Agent on one Proxy-Server.
Is it possible, that other Proxy-Servers (which should handle the other 2 VMs) are trying to open the locked file directly instead of using the Repository-Agent?

Regards,
Marc

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 30, 2013 11:25 am

Is it possible, that other Proxy-Servers (which should handle the other 2 VMs) are trying to open the locked file directly instead of using the Repository-Agent?
Nope, only the agent deployed on repository or on proxying server (in case of CIFS) communicates with backup file, and it’s impossible that source agents deployed on proxy servers will try to bypass it. Thanks.

marcseitz
Influencer
Posts: 17
Liked: 5 times
Joined: Apr 04, 2012 11:17 am
Full Name: Marc Seitz
Contact:

Re: Parallel Processing, VBK file locked error

Post by marcseitz » Aug 30, 2013 11:27 am

Sorry for the stupid question - What exactly is the "Repository Agent"? Where is this installed?
We have 6 Proxy-Servers and just updated from 6.5 to 7.0 without installing a Repository Agent manually.

How is the Server elected to be the Repository-Agent?

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 30, 2013 11:51 am

The component (transport service) that is deployed on a server when you assign a repository role to it (or add it to a backup console, in other words). Since SMB share does not support Veeam transport services, data to the SMB share is written from a Windows-based proxying server. Thanks.

marcseitz
Influencer
Posts: 17
Liked: 5 times
Joined: Apr 04, 2012 11:17 am
Full Name: Marc Seitz
Contact:

Re: Parallel Processing, VBK file locked error

Post by marcseitz » Aug 30, 2013 12:48 pm

And which Server is getting elected when running several Proxy-Servers (with, of course, transport Service)? Election per Job? Election once?

veremin
Product Manager
Posts: 16897
Liked: 1437 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel Processing, VBK file locked error

Post by veremin » Aug 30, 2013 1:04 pm

I’m not sure I quite follow you on that. In general, the target component is able to communicate with many proxy servers, though, only with one in the same time. Thanks.

nefes
Veeam Software
Posts: 628
Liked: 156 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: Parallel Processing, VBK file locked error

Post by nefes » Aug 30, 2013 2:24 pm

marcseitz wrote:And which Server is getting elected when running several Proxy-Servers (with, of course, transport Service)? Election per Job? Election once?
You can either specify this server explicitly in Repository options or it will be chosen at the start of each job from the list of proxy servers used by that job.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 18 guests