Comprehensive data protection for all workloads
Post Reply
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

How to calculate stats associated to bottleneck ratings?

Post by cparker4486 »

Hello,

I'm not exactly sure how to ask this question but I'll give it a shot.

The job summary and actions list show many statistics about a job as well as calculated bottleneck amounts in percentages. What it doesn't show are the values that make up those bottlenecks. Here's a screenshot of a recently completed job:

Image

My biggest concern is the Target bottleneck which is incredibly high. What does it mean that the Target is slowing the job down by 97%? Does that mean the job is only able to write at about 1MB/s but it's capable of a 97% increase (if the target could support that rate)? If so, how do I determine those two values: actual write speed, target write speed?
-- Chris
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by tsightler »

The bottleneck stats show the "percent busy" of every point in the chain. If the target is 97% busy that means that this portion of the process spent 97% of it's time waiting on the target to write data.

There will ALWAYS be a bottleneck in every situation, and typically, for reverse incremental backups, the target will be the bottleneck. This is simply because the target is typically slower disks (most companies backup to slower disks than their production storage), but also because, for reverse incremental, every block read from the source requires 3 blocks to be moved on the target, so the target is doing 3x as much I/O. Assuming this is reverse incremental, I would expect the target to be 90%+ bottleneck in almost all environments.

If you had disks that were twice as fast the target would probably still be the bottleneck, but the utilization of the other stages would go up (perhaps to 40%, 60%, 4%, 97%). Eventually, if the target disks were fast enough, one of the other stages would become the bottleneck, probably the proxy CPU, but perhaps the source.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by foggy »

Also explained in the sticky FAQ topic. Kindly be patient to read it before asking general questions, please!
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by cparker4486 »

foggy wrote:Also explained in the sticky FAQ topic.
No, it's not. The FAQ does not ask or answer the question as I've asked it.
Kindly be patient to read it before asking general questions, please!
Your first sentence was more than enough on directing me to the FAQ. This instruction here, on the other hand, is unnecessary and a bit rude!

edit: Wait, wait. I may need to apologize to you here, foggy. My initial statement saying the FAQ doesn't answer my question may be wrong as I didn't pay enough attention to the last Q/A. I'll have to wait for the next job to experiment with it.
edit 2: Turns out a job is running right now but, unfortunately, the real-time values mentioned in the FAQ are not what I am looking for.
-- Chris
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by cparker4486 »

tsightler wrote:If you had disks that were twice as fast the target would probably still be the bottleneck, but the utilization of the other stages would go up (perhaps to 40%, 60%, 4%, 97%). Eventually, if the target disks were fast enough, one of the other stages would become the bottleneck, probably the proxy CPU, but perhaps the source.
Yes, it is a RI job and thanks for the explanation. But is there a way to determine the actual values that I'm asking about based on the stats given in the summary? For example, is it possible to determine that Veeam would like to write at xMB/s but my target is only capable of writing at yMB/s? I think I'm failing to visualize a 99% bottleneck wait-time.
-- Chris
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by Vitaliy S. »

Hi Chris, I'm afraid there are too many variables here, so there is no definite answer to your question, as once you resolve one bottleneck another one will arise.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by tsightler »

Also, it's not really about MBs, but about IOPS. Reverse incremental is I/O bound, not throughput bound. It's not really possible to turn the bottleneck number into a required IOPS number as that's not really it's intent, it's only to be an indicator as to what to look at if you want to improve performance.
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by cparker4486 »

Vitaliy S. wrote:Hi Chris, I'm afraid there are too many variables here, so there is no definite answer to your question, as once you resolve one bottleneck another one will arise.
Understood. Thanks.
-- Chris
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by cparker4486 »

tsightler wrote:Also, it's not really about MBs, but about IOPS. Reverse incremental is I/O bound, not throughput bound. It's not really possible to turn the bottleneck number into a required IOPS number as that's not really it's intent, it's only to be an indicator as to what to look at if you want to improve performance.
Understood. Thanks. Fortunately my jobs are not taking longer than I can support (quite a bit less, in fact). Thanks again.
-- Chris
andersonts
Veteran
Posts: 307
Liked: 31 times
Joined: Mar 21, 2012 9:56 pm
Full Name: Tim Anderson
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by andersonts »

From another perspective you could utilize the normal Windows and Storage tools to figure out exactly how hard your source and target disks are being hit. Disk latency is probably the easiest one although you could really get into the weeds if you wanted by looking at overall read and write IOPS. I would say in a year at Veeam the most common I see is source and target for bottleneck. Source is pretty easy to figure out especially as I normally install Veeam ONE (even the free edition will give you real time stats that are much more comprehensive and easier to read than the native tools) but target would normally require at least some degree of detail from the storage itself (although if your proxy is virtual you may be able to gather some helpful stats from ONE). There are tons of posts on things you can do for all of the bottleneck scenarios so you should be in pretty good shape here.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by foggy »

cparker4486 wrote:Your first sentence was more than enough on directing me to the FAQ. This instruction here, on the other hand, is unnecessary and a bit rude!
Chris, no way my intention was to be rude here, but rather point you to the piece of information you were looking for (as FAQ topic does answer the "what does it mean"-like questions perfectly). I apologize if I sound rude.
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: How to calculate stats associated to bottleneck ratings?

Post by cparker4486 »

foggy wrote:
Thank you, foggy. I'm sure I just misunderstood.
-- Chris
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 277 guests