Parallel Processing, VBK file locked error

[ameyer]

After doing an in-place upgrade from v6.5 to v7, everything went smoothly for the first few nights of backups. I then decided to enable Parallel Processing and have started to receive errors about the VBK file being in use: "The process cannot access the file because it is being used by another process". The backup job ultimately fails.

If I retry the job enough times, it will complete all VMs successfully. Nothing else would be accessing the VBK besides Veeam and this would be the only job running at that time.

Any thoughts on what I might be doing wrong?


Thanks,
Andy

[Gostev]

You are not doing anything wrong, but some process is locking the backup files (most likely, some external process). Try Process Monitor on Veeam backup files, or open a support case. Thanks!

[ameyer]

Thanks for the quick reply!

[marcseitz]

Hi Guys,

we've updated to Version 7.0 (of course, without problems).
We want to leverage from the parallel processing of VMs within a Job.

The Problem while using parallel processing is, that a random amount of VMs of a Job is failling with the following error message: (some Jobs are working perfect and without problems)

28.08.2013 23:49:01 :: Error: Client error: End of file
Exception from server: The process cannot access the file because it is being used by another process.
Failed to create or open file [<CIFS-SHARE\FOLDER\BACKUP>.vbk].
Failed to backup text locally. Backup: [veeamfs:0:048c2c67-a416-444d-861d-0793a96d19ed (vm-198919)\summary.xml@<CIFS-SHARE\FOLDER\BACKUP>V

I've seen this problem, when only one Job is running to process 5VMs (2 VMDKs per VM).
I can see, that the VM which is failing doesn't get any Proxy Server assigned. It's Createing the Snapshot, Releasing the Guest and Removing the Snapshot directly after this.

Did anybody of you see this problem too?

(Still working with the Support Guys: 00436068)

Thanks & Regards,
Marc

[veremin]

Hi, Mark, as a first investigation step, kindly, check whether there are some external activities that might lock the backup files. Otherwise, you can just wait and see what support team suggest to you. Thanks.

[marcseitz]

Hi Vladimir,

thanks for Merging the Topic.
Of course I'm waiting, what the Support Guys can figure out - Just want to use every available channel

The used CIFS-Share is only used by Veeam B&R and no other opened Sessions / Files on this Share from other Applications.
I'll update you when there are any news.

Thanks,
Marc

[veremin]

So far, there are only you two who have reported about this issue. And the op, in contrast to you, has never come back to update the topic with support ticket number or found resolution, so, we can’t be sure what might be the cause of this problem. Thanks.

[marcseitz]

Hi again,

just a question if you can tell me, how the traffic is flowing while parallel processing.
I've backed up 5VMs at the moment with serveral VMDKs, using several Backup Proxies.

On my Repository, I've only seen one connection from one Proxy.

Is it correct, that all Proxies will push the Traffic to the Proxy which is holding the lock on the File?
(So the Proxies which are not holding the Lock, use the "first touch Proxy" as Proxy too?

Thanks,
Marc

[veremin]

Actually, it’s not a proxy server that communicates with .vbk file, but rather a target repository agent. So, it shouldn’t be the case.

On my Repository, I've only seen one connection from one Proxy.

The connection occurs between only pair of agents. That being said, in one time only one proxy server can contact repository. And in case of CIFS share, share, the communication takes place between proxy server and proxying server. Thanks.

[mcwill]

So far, there are only you two who have reported about this issue. And the op, in contrast to you, has never come back to update the topic with support ticket number or found resolution, so, we can’t be sure what might be the cause of this problem. Thanks.

We also hit this in testing V7 before deploying to Prod. Ticket raised was # 00429947.

Ours seemed to be caused by a slow cifs share, (An Apple Time Capsule of all things) and the problem went away when we changed to a different backup storage unit. As it was a test setup only we didn't push it and allowed the ticket to be closed once a workaround was discovered.

[veremin]

As far as I can see, the Marc’s case is also related to the usage of CIFS share. So, Mark, it might be worth changing temporarily target repository and seeing whether the said issue is caused by the network connectivity or by something else. Thanks.

[marcseitz]

Hi Vladimir,

changing to another Repository is not possible in our Environment (>60TB Backup-Data) - sorry.
I don't think, it's a network connectivity related problem.
I can see, that (for example) 5 of 7 VMs will be processed correctly. So the Backup-File is locked by the Repository-Agent on one Proxy-Server.
Is it possible, that other Proxy-Servers (which should handle the other 2 VMs) are trying to open the locked file directly instead of using the Repository-Agent?

Regards,
Marc

[veremin]

Is it possible, that other Proxy-Servers (which should handle the other 2 VMs) are trying to open the locked file directly instead of using the Repository-Agent?

Nope, only the agent deployed on repository or on proxying server (in case of CIFS) communicates with backup file, and it’s impossible that source agents deployed on proxy servers will try to bypass it. Thanks.

[marcseitz]

Sorry for the stupid question - What exactly is the "Repository Agent"? Where is this installed?
We have 6 Proxy-Servers and just updated from 6.5 to 7.0 without installing a Repository Agent manually.

How is the Server elected to be the Repository-Agent?

[veremin]

The component (transport service) that is deployed on a server when you assign a repository role to it (or add it to a backup console, in other words). Since SMB share does not support Veeam transport services, data to the SMB share is written from a Windows-based proxying server. Thanks.

[marcseitz]

And which Server is getting elected when running several Proxy-Servers (with, of course, transport Service)? Election per Job? Election once?

[veremin]

I’m not sure I quite follow you on that. In general, the target component is able to communicate with many proxy servers, though, only with one in the same time. Thanks.

[nefes]

And which Server is getting elected when running several Proxy-Servers (with, of course, transport Service)? Election per Job? Election once?

You can either specify this server explicitly in Repository options or it will be chosen at the start of each job from the list of proxy servers used by that job.