Comprehensive data protection for all workloads
Post Reply
Taroking1
Influencer
Posts: 23
Liked: 6 times
Joined: Mar 22, 2021 11:18 pm
Contact:

Need a method for timing out a backup copy job or killing it in the console

Post by Taroking1 »

We need a method for having backup copy jobs timeout (especially WAN accelerator jobs) and also a method for killing stuck backup copy jobs from within the console. If you have a hung job that is just sitting there waiting for resources, setting it to disable will accomplish nothing. The job will now say stopping, but will never stop.

The main issue with this is that if this is a WAN accelerator job, then other jobs back up behind it. So in this scenario, it would be great to have a timeout period for a job that is waiting for something. In addition, we need a method for killing the job from within the console. I am aware of going to the relevant repository servers and killing off processes within task manager, but this is a kludge.

I am also aware that just killing a copy job might result in the file it was working on being corrupted. However a better method to detect this corruption and allow it to be copied again is the correct solution. I would rather have one corrupted incremental that needs me to intervene on, rather than 100's of servers in other jobs not replicating offsite due to them being backed up behind a stuck WAN accelerator job.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by PetrM »

Hello,

I would try to address the issue with a job which is waiting for resources for some unknown reason, have you had a chance to open a support case?

Thanks!
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by Egor Yakovlev »

Hi Taroking1,

- if Jobs hang \ stuck in stopping - it's abnormal and needs investigation with support
- if "Disable" a job does not disables it - it's abnormal and needs to be investigated with support

I would strongly advise to avoid task manager massacre on processes and troubleshoot your case in detail with our support engineers.

/Thanks!
Taroking1
Influencer
Posts: 23
Liked: 6 times
Joined: Mar 22, 2021 11:18 pm
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by Taroking1 »

I have opened a ticket. More to the point, it looks like the patch for V11 has fixed the issue where the job is just sitting there doing nothing. It is now failing the particular machine it was stuck on and progressing to the next item.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by PetrM »

Looks like the only remaining question is why job does not process this specific item which will be failed later. It makes sense to clarify it with our engineers as well.

Thanks!
Taroking1
Influencer
Posts: 23
Liked: 6 times
Joined: Mar 22, 2021 11:18 pm
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by Taroking1 »

That is a good question Petr. A couple of weeks into troubleshooting with an open ticket and no resolution yet.

Most of the functions within Veeam are excellent, but I would say that my experience with the backup copy has been abysmal. One issue after the next and really lets the product down.

Direct copy is the only method that reliably works but will result in me pushing 100's of GB over our WAN network from remote sites with slow links. This is not actually a viable solution for us.

The WAN accelerator appears to be a hot mess of issues. If I use the low bandwidth, the job will start running nicely and then at random times within the backup copy I get "Global cache synchronization completed" which takes 1.5-1hrs and then away it goes again, but then after another 4 or 5 machines it does it again. Overall backup copy job is taking 18-23 hours. I can do direct copy in about 8 hours and that is even with Network traffic rules limiting my bandwidth.

Use high bandwidth and then you run into the job pausing to Create Fingerprints for all of the machines. Even though there has been no changes to the job from yesterday.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by PetrM »

Hello,

Actually, the main problem is that we know just symptoms but we have no idea what exactly triggers these issues. I would continue working with our support team in order to get an explanation. Looks like the issue is not trivial and requires deep examination of debug logs and performing of various tests. Could you please share a support case ID for our reference?

Thanks!
Taroking1
Influencer
Posts: 23
Liked: 6 times
Joined: Mar 22, 2021 11:18 pm
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by Taroking1 » 1 person likes this post

04712150
Taroking1
Influencer
Posts: 23
Liked: 6 times
Joined: Mar 22, 2021 11:18 pm
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by Taroking1 » 1 person likes this post

In case anyone is reading. We got around to a resolution. We were still having issues with WAN accelerator in Low Bandwidth mode so I am not using it until some more V11 patches come out.

What I have done
Put the WAN accelerators into high bandwidth mode
Set the Network Traffic rules to limit my bandwidth to about 50% of the available per WAN link

The job is now running consistently every night with no issues.
The amount of WAN traffic I am generating is more than Low bandwidth mode, but it is still much better than direct copy and it is now rock solid.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Need a method for timing out a backup copy job or killing it in the console

Post by PetrM »

Hello,

Yes, we read! :) Your willingness to share the workaround with our Community is highly appreciated!

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 132 guests