Comprehensive data protection for all workloads
Post Reply
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Feature Request: Keep "Resource not ready" in job log

Post by nmdange »

When running a backup job for a large number of VMs, many of those VMs will spend a lot of time in a "Resource not ready" state. This could be any number of resources, like the backup repository, the proxy, etc. However, once the VM starts processing, the job details only show the message "Required backup infrastructure resources have been assigned".

This makes it hard to see after the fact what resources were causing the delay. It would be nice if you could see in the UI how long each VM spent waiting for each specific resource. This would help us know what piece of the backup infrastructure is limiting backup parallelism when managing a large environment. So we would know if adding a new backup proxy makes sense, for example. As it is, right now I can only see this information by watching a backup job live while it's running, which is a bit of a problem when the jobs always run in the middle of the night :D
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by wishr »

Hi again Nmdange,

Thank you for the request. It's noted.

Btw, have you managed to identify the root cause?

Thanks
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by wishr »

I would like to double check the following: is there anything else after "Resource not ready" phrase in your case, because usually there is an indication which resource is not ready.

Also, could you please let me know if you, by any chance, use Veeam MP or Veeam ONE? The second part of your post sounds like a good potential feature for some of our monitoring tools ;)

BR,
Fedor
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by Shestakov »

Hello,
what if you could see statistics of proxies busyness? E.g. proxy A had all its resources (parallel tasks) used by job X during particular period of time. So you can understand which proxies are overused and/or which jobs is better to re-schedule.
By the way, do you use Veeam ONE?
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by nmdange »

Since most of my environment is Hyper-V, I see a lot of "Resource not ready: Snapshot", but I have also seen "Resource not ready: Backup Proxy" and "Resource not ready: Backup Repository". I know what the causes are and how to fix them, it's just I can't see what the type of resource is unless I watch the backups live.

I don't have Veeam ONE licensing. I could definitely see Veeam ONE having detailed reports on parallel tasks, but at a minimum, just being able to look at a job in the console after the fact and seeing which resource type was waiting would be enough for me.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by Vitaliy S. »

Yes, this log entry would be a nice to have feature, I agree. Also, I would recommend taking a look at Veeam ONE 9.5 U4 (once available) new features that might also be able to solve your existing issue. Thanks!
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

[MERGED] Feature request: Resource not ready

Post by hyvokar »

Hi,

When job action log states, that "resource not ready" (usually backup repository in my case), could you please add WHICH resource is not ready. I mean I have several backup repositories.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature request: Resource not ready

Post by foggy »

Hi, typically it should state the particular resource name, but seems there are situations when it is not the case. We will look into it, thanks.
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Feature request: Resource not ready

Post by hyvokar »

Thanks for the reply foggy.

Personally I don't recall ever seeing the resource name, but could be my memory.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
crackocain
Service Provider
Posts: 240
Liked: 27 times
Joined: Dec 14, 2015 8:20 pm
Full Name: Mehmet Istanbullu
Location: Türkiye
Contact:

Re: Feature request: Resource not ready

Post by crackocain »

+1
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Feature request: Resource not ready

Post by nmdange »

Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by Zew »

Running Veeam 11. Same issues still exists. Job @ 99% stating "Resource not ready: Backup repository".
Same boat as others, doesn't specify which (although I know which one cause there is only one for these jobs).

Which is super weird cause the jobs process (data transferred) but then cause other jobs of the same nature all started at the same time (source of problem was a long list of "old style" BCJ were paused and then restarted after some physical maintenance work was completed). It's funny cause they will transfer the data and then wait at this state instead of completing:

E.G.

Job 1 - Processing 21% (no resource issues)
Job 2 - Processing 0% (waiting for resources)
Job 4 - Processing 99% (Resource not ready: Backup repository) (waiting to merge)
Job 5 - Processing 99% (Resource not ready: Backup repository) (waiting to merge)
Job 6 - Processing 99% (Resource not ready: Backup repository) (waiting to merge)
Job 7 - Processing 0% (waiting for resources)
Job 8 - Processing 99% (Merging oldest incremental into full backup file) (finally after some of the other processes completed)

What's super weird about this, is you figured the job as a whole gets processed as a "batch" and gets resources to complete all required tasks. Instead what appears is each jobs tasks are queued, so they have to wait to completed the final task to complete their tasks.

Not suggesting a change in design, but interesting to note based on the symptoms.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by foggy »

That basically means your repository doesn't have the required amount of slots to perform all these tasks in parallel. If you feel the repository is speced for all this load - then it's worth opening a case for a closer look. Remember though that each backup chain merge process takes a separate slot and if this repository is per-VM chains enabled, then it should have enough slots to process all VMs in the job since each of them has its own chain and requires a slot.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by Zew »

I totally get the slot "tasks" requirements, and understanding jobs have multiple tasks completing for resources the constraints are pretty obvious actually.

Just noted the repo name was not listed in the state of the job. That's all, the rest makes sense once I printed out the design myself on paper.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by foggy »

Ok, misread your post then. ;) Thanks for the feedback, anyway.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Feature Request: Keep "Resource not ready" in job log

Post by Zew »

No worries, thanks for following up Foggy, always good to hear from you.

Just wondering if the root cause of when the resource is not directly named was determined as you mentioned in previous posts it was to be looked at.
foggy wrote: Dec 29, 2018 11:46 am Hi, typically it should state the particular resource name, but seems there are situations when it is not the case. We will look into it, thanks.
if not, no worries again I knew which one it was based on the deployment. Just a curious George here.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 217 guests