Comprehensive data protection for all workloads
Post Reply
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Backup overlap then hanging

Post by roadking »

So I'm new to Veeam Backups and running version 6 patch 3 and finding some interesting events

-- I have two backup proxies - one on the backup server itself (a vm), one on another VM
-- I'm running backups on the weekends only. These are reverse incremental jobs with a single full
-- I have tried to spread them out based on how many nodes are in the jobs and how long the incremental may take
-- some are spread out 4 - 5 hours apart yet due to unknown circumstances they inevitably end up overlapping
-- after several hours elapse, more jobs start based on the schedule then they overlap the ones that are still running
-- I had configured the backups to alternate the proxy servers so that IF there was an overlap, then the backups 'should' continue to work
-- what ends up happening is that all jobs end up hanging and I have to restart services to get the jobs to stop

I have read on numerous threads here that I should be able to start all the jobs at once and they will wait until they have resources to begin running. I have yet to experience this.

I can however re-run the jobs manually -- one at a time and they always finish without an error. This only occurs when multiple jobs are scheduled then overlap one another. Then they seem to get tripped up on one another and fail.

I am now trying to run the jobs with post job activities and including the next job in the line so that after one job completes - however long that may take - then the others will start in order.

Is there a best practice for job scheduling or could there be something wrong in the setup I have?

Thanks - and bear with me as I'm still new to this.
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Backup overlap then hanging

Post by dellock6 » 1 person likes this post

Sounds like your two involved storages are too slow to allow for two concurrent backup jobs, either the production (source) storage, or the backup (target) storage. what are the bottlenecks indicated in the job logs?
One possible solution, if you are sending all backup to the same repository (maybe the same one in the Veeam Backup server?) you can limit the "ingestion rate" of that repository so it would not fail the jobs.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Backup overlap then hanging

Post by roadking »

Interesting that the storage would be the slow part - they are NetApp FAS2040 going to SAS disks and both filers sit idle for the most part. Looking at the overall performance on them during backups from Veeam the total CPU doesn't reach 40%. Disk I/O seems good too - though I wouldn't know how best to measure that -- no baselines have been taken.

I can try the ingestion rate -- I would think that would cause them to run slower - rather than preventing the jobs from failing. I can see though that when the jobs attempt to run that one of them (as they are on top of one another) removes the hot-add feature after a certain node is done - then removing all mounts from the proxy. If that proxy was in the middle of backing up a node that relied on the mount then the jobs stop -- with no error - just stop and I end up having to reboot the machine to get the job to stop.

I really think that's the problem and I don't really see slow throughput -- or issues with the storage. I get backup rates upwards of 140MB/s on occasion - depending on how busy the target is. My bottle neck always says Target (understood that to mean the repository) and most jobs look like this for BUSY: SOURCE 67% > Proxy 11% > Network 38% > Target 89%

I'm under the impression that this is suppose to be good numbers from Veeam.
Jfmoots
Veeam Software
Posts: 215
Liked: 26 times
Joined: Oct 28, 2011 3:26 pm
Full Name: James Moots
Location: Ohio, United States
Contact:

Re: Backup overlap then hanging

Post by Jfmoots »

Those are pretty good bottleneck stats. Tell us about each proxy.

How many vCPUs?

How many concurrent jobs is each proxy configured to run?
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Backup overlap then hanging

Post by roadking »

proxy stats are

Backup Server Proxy
-- 4 CPU
-- 8GB ram
-- Vmware VM 4.1
-- Windows 2008 Server

Additional Proxy is the same config

I had hoped that each proxy would run just one job as I configured each to run several hours apart from one another. However - there have been unknown issues that slowed each job making them take longer than the timing would allow- thus they end up overlapping one another.

I still think this is the problem. When they overlap, jobs are running to the same proxys that have mounted drives (using hot-add) then when the second job starts and mounts it's drives, then the 1st job finishes with it's node then dismounts the drives from the 1st job AND ALSO the 2nd. Causing the 2nd job to stall as there are no disks to read from.

I am rescheduling these jobs to run using post job activities using the cmd line for the next job - I've seen that called 'daisy chaining' the jobs on this forum -- and I'm hopeful that this will solve the problem.
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Backup overlap then hanging

Post by dellock6 »

Robert, have you checked if proxies are configured to run only one cuncurrent job each? Because if you do so, each of them will run a single job, and other jobs will stay in queue waiting for an available slot in one of the proxies.
If the proxies are running more than one job even if they are configured to do so, I would probably open a ticket to investigate, because is an unexpected behaviour.
Also, I have proxies with two job each in our DC, and I never seen a job dismounting disks from the other, even when overlapping...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Post Reply

Who is online

Users browsing this forum: BackupBytesTim, Bing [Bot], Semrush [Bot] and 272 guests