Host-based backup of VMware vSphere VMs.
Post Reply
vmos
Novice
Posts: 4
Liked: never
Joined: Sep 05, 2013 8:26 am
Full Name: vmos
Contact:

backup copy"waiting for another task to complete" what task?

Post by vmos »

Hello, we've got veeam 7 runing a bunch of jobs across two servers. Some of the jobs also have secondary destinations. There's a couple of them have this particular problem.

In these cases, the secondary destination is a linux server and the copy mode is "by job" and direct. It'll run along, do a few VMs and then sit there for ages, on all the pending VMs, the status is either "latest restore point is already copied" or "Waiting for another task to complete" but what task?

There are no other jobs running, the other veeam server may be doing it's thing, but it's not interacting with this one. The concurrency on the linux repository is 6 jobs although increasing it to 20 doesn't seem to make much difference.

Looking at the throughput graph, it seems to wait about ten minutes between VMs. How can I find out what this is waiting on and can I speed it up?
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by Vitaliy S. »

Hello,
vmos wrote:It'll run along, do a few VMs and then sit there for ages, on all the pending VMs, the status is either "latest restore point is already copied" or "Waiting for another task to complete" but what task?
These are two different messages, do you see them at the same time? Each task processes one virtual disk, so the the second message tells you that there are no slots available to continue processing your VMs. As to the latest restore is already copied, then you should see a restore point created on the target repository. Is this the case?
vmos wrote:There are no other jobs running, the other veeam server may be doing it's thing, but it's not interacting with this one.
How many tasks do you have configured on the source repository?

Thank you!
vmos
Novice
Posts: 4
Liked: never
Joined: Sep 05, 2013 8:26 am
Full Name: vmos
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by vmos »

I've got two tasks configured to use this repository, at the time I'm getting these messages, usually one of them is idle. To be sure I've gone and disabled the other task but it doesn't make much difference
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by Vitaliy S. »

vmos wrote:I've got two tasks configured to use this repository, at the time I'm getting these messages, usually one of them is idle.
Unfortunately, it is still not clear whether you have two jobs using this repository as a source or as a target. Since concurrent tasks limit doesn't make any difference I was trying to understand what is the limit for source repository, the one which is used by regular backup jobs.
vmos
Novice
Posts: 4
Liked: never
Joined: Sep 05, 2013 8:26 am
Full Name: vmos
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by vmos »

On each windows veeam server I've got half a dozen jobs, on each of my source hosts there are one or two jobs, on each linux repo (target) there are two jobs.

BUT (big butt) in order to try and get to the root of this issue, I've disabled everything except this one job and it still sits there saying "waiting for task to complete"

So what's it waiting on if all the other tasks are disabled?
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by Vitaliy S. »

Let's ask our support team for assistance with logs review, as it should be easy to find out what resource is busy and why.
JWester
Service Provider
Posts: 63
Liked: 7 times
Joined: Apr 04, 2011 8:56 am
Full Name: Joern Westermann
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by JWester »

Any news on this?
We have the same issue (case 00524953):
Currently only one backup copy job is running. It copies a VM (app. 30-45 secs), then takes a nap of 3-4 minutes, copies the next VM, nap, ...
No load on cpu, network, sql, disk...
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by foggy »

I cannot check the OP's case since he haven't provided the ID here, so please continue working with support on this.

Btw, what are the limitations on the number of concurrent tasks on your repositories?
JWester
Service Provider
Posts: 63
Liked: 7 times
Joined: Apr 04, 2011 8:56 am
Full Name: Joern Westermann
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by JWester »

No limits on tasks and bandwidth.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by foggy »

I was talking about the maximum number of concurrent tasks setting for backup repository (available in the repository wizard). Do you mean everything is at its default values?
JWester
Service Provider
Posts: 63
Liked: 7 times
Joined: Apr 04, 2011 8:56 am
Full Name: Joern Westermann
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by JWester »

Edit Backup Repository / Repository / Load Control
-> "limit maximum concurrent tasks" is not checked
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by foggy »

Got it. And there are no other jobs running involving the same repositories, right?
JWester
Service Provider
Posts: 63
Liked: 7 times
Joined: Apr 04, 2011 8:56 am
Full Name: Joern Westermann
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by JWester »

foggy wrote:Got it. And there are no other jobs running involving the same repositories, right?
Correct.
palw
Novice
Posts: 8
Liked: never
Joined: Apr 16, 2014 10:52 am
Full Name: Paul Wilson
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by palw »

I've been having a similar issue to this. We've been testing our Veeam backup solution for the last couple of months. I've
decided on using reverse incremental to multiple windows server repositories for our initial backups but only want to keep 2
weeks and then keep a second copy with retentions offsite.

I have initially setup 3 backup copy jobs with different GFS retention periods depending on the requirements and placed
backup jobs in the copy job. However when each copy job runs it's only copying 1 VM at a time with the next VM Action of
"Waiting for another task to complete", I have about 200 VM's so this will take too long.

Is this the expected behaviour with backup copy jobs? Or should a single copy job be able to process multiple VM's at the
same time?

The offsite repository it set to handle 6 jobs concurrently, and all the proxies can handle at least 4 concurrent tasks. I
have 1Gb network connection to the offisite repository.

Thanks
Gostev
Chief Product Officer
Posts: 31807
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by Gostev »

Yes, it is expected. What is the bottleneck in your case? Since Backup Copy does nothing but copy data blocks from one storage to another, processing multiple VMs in parallel will not have any impact in most cases (and especially so on 1Gb network connection that can easily be saturated by the single stream - and you effectively have 3 due to 3 backup copy jobs). Thanks!
palw
Novice
Posts: 8
Liked: never
Joined: Apr 16, 2014 10:52 am
Full Name: Paul Wilson
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by palw »

Hi Anton, Thanks for your response, if its just copying data then that makes sense.

Bottleneck is the Target (Source 5% > Proxy 11% > Network 4% > Target 99%) The Target is a dedup appliance, certainly not slow disk but not as fast as the initial backups.

It may have just been the initial seed that seemed to be taking forever, its running OK now, although nowhere near as fast as the backups. What is the best practice when setting up backup copy jobs? If I have 100 VM's that I would like a on a 6 month retention, should I have them all in 1 backup copy job or separate them?
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by Vitaliy S. »

Hi Paul,

Yes, dedupe storage is not a good target for jobs using random I/O compared to sequential writes. Having target as a bottleneck, means that configuring additional backup copy jobs to transfer VM data will not increase the overall performance. Improving target storage performance will allow you running more backup copy jobs at a time.

Thanks!
Wector
Novice
Posts: 4
Liked: 1 time
Joined: Dec 20, 2014 2:37 pm
Full Name: Tim Pfeiffenberger
Contact:

Re: backup copy"waiting for another task to complete" what t

Post by Wector »

Hi,
we also try to use a "backp copy Job" to send the backup to another location. beacause the Job works sequential there ist mutch time gone. please have a look: http://www.directupload.net/file/d/3842 ... xw_jpg.htm
This could be twice as fast when the copy Job uses parallel proccessing to.

regards
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Gostev, masahide.k and 52 guests