How to calculate stats associated to bottleneck ratings?

[cparker4486]

Hello,

I'm not exactly sure how to ask this question but I'll give it a shot.

The job summary and actions list show many statistics about a job as well as calculated bottleneck amounts in percentages. What it doesn't show are the values that make up those bottlenecks. Here's a screenshot of a recently completed job:



My biggest concern is the Target bottleneck which is incredibly high. What does it mean that the Target is slowing the job down by 97%? Does that mean the job is only able to write at about 1MB/s but it's capable of a 97% increase (if the target could support that rate)? If so, how do I determine those two values: actual write speed, target write speed?

[tsightler]

The bottleneck stats show the "percent busy" of every point in the chain. If the target is 97% busy that means that this portion of the process spent 97% of it's time waiting on the target to write data.

There will ALWAYS be a bottleneck in every situation, and typically, for reverse incremental backups, the target will be the bottleneck. This is simply because the target is typically slower disks (most companies backup to slower disks than their production storage), but also because, for reverse incremental, every block read from the source requires 3 blocks to be moved on the target, so the target is doing 3x as much I/O. Assuming this is reverse incremental, I would expect the target to be 90%+ bottleneck in almost all environments.

If you had disks that were twice as fast the target would probably still be the bottleneck, but the utilization of the other stages would go up (perhaps to 40%, 60%, 4%, 97%). Eventually, if the target disks were fast enough, one of the other stages would become the bottleneck, probably the proxy CPU, but perhaps the source.

[foggy]

Also explained in the sticky FAQ topic. Kindly be patient to read it before asking general questions, please!

[cparker4486]

Also explained in the sticky FAQ topic.

No, it's not. The FAQ does not ask or answer the question as I've asked it.

Kindly be patient to read it before asking general questions, please!

Your first sentence was more than enough on directing me to the FAQ. This instruction here, on the other hand, is unnecessary and a bit rude!

edit: Wait, wait. I may need to apologize to you here, foggy. My initial statement saying the FAQ doesn't answer my question may be wrong as I didn't pay enough attention to the last Q/A. I'll have to wait for the next job to experiment with it.
edit 2: Turns out a job is running right now but, unfortunately, the real-time values mentioned in the FAQ are not what I am looking for.

[cparker4486]

If you had disks that were twice as fast the target would probably still be the bottleneck, but the utilization of the other stages would go up (perhaps to 40%, 60%, 4%, 97%). Eventually, if the target disks were fast enough, one of the other stages would become the bottleneck, probably the proxy CPU, but perhaps the source.

Yes, it is a RI job and thanks for the explanation. But is there a way to determine the actual values that I'm asking about based on the stats given in the summary? For example, is it possible to determine that Veeam would like to write at xMB/s but my target is only capable of writing at yMB/s? I think I'm failing to visualize a 99% bottleneck wait-time.

[Vitaliy S.]

Hi Chris, I'm afraid there are too many variables here, so there is no definite answer to your question, as once you resolve one bottleneck another one will arise.

[tsightler]

Also, it's not really about MBs, but about IOPS. Reverse incremental is I/O bound, not throughput bound. It's not really possible to turn the bottleneck number into a required IOPS number as that's not really it's intent, it's only to be an indicator as to what to look at if you want to improve performance.

[cparker4486]

Hi Chris, I'm afraid there are too many variables here, so there is no definite answer to your question, as once you resolve one bottleneck another one will arise.

Understood. Thanks.

[cparker4486]

Also, it's not really about MBs, but about IOPS. Reverse incremental is I/O bound, not throughput bound. It's not really possible to turn the bottleneck number into a required IOPS number as that's not really it's intent, it's only to be an indicator as to what to look at if you want to improve performance.

Understood. Thanks. Fortunately my jobs are not taking longer than I can support (quite a bit less, in fact). Thanks again.

[andersonts]

From another perspective you could utilize the normal Windows and Storage tools to figure out exactly how hard your source and target disks are being hit. Disk latency is probably the easiest one although you could really get into the weeds if you wanted by looking at overall read and write IOPS. I would say in a year at Veeam the most common I see is source and target for bottleneck. Source is pretty easy to figure out especially as I normally install Veeam ONE (even the free edition will give you real time stats that are much more comprehensive and easier to read than the native tools) but target would normally require at least some degree of detail from the storage itself (although if your proxy is virtual you may be able to gather some helpful stats from ONE). There are tons of posts on things you can do for all of the bottleneck scenarios so you should be in pretty good shape here.

[foggy]

Your first sentence was more than enough on directing me to the FAQ. This instruction here, on the other hand, is unnecessary and a bit rude!

Chris, no way my intention was to be rude here, but rather point you to the piece of information you were looking for (as FAQ topic does answer the "what does it mean"-like questions perfectly). I apologize if I sound rude.

[cparker4486]

Thank you, foggy. I'm sure I just misunderstood.