Weird Performance Bottlenecks statistics

[filippoAdb]

I'm making some tests comparing the following backup systems:

System 1
-local B&R server on subnet 12 with proxy and gateway roles
-esxi hosts and vcenter on subnet 12
-nas repository on subnet 12

System 2
-remote B&R server on another site
-local proxy and gateway on local windows server on subnet 20
-esxi hosts and vcenter on subnet 12 (the same of system 1)
-nas repository on subnet 12 (the same of system 1, a dedicated shared folder is used)

I execute the backup of the same VMs with the same configuration at a different not overlapping time.

Here are the typical results after an incremental backup:
system 1: Load: Source 67% > Proxy 0% > Network 75% > Target 77%
system 2: Load: Source 0% > Proxy 40% > Network 0% > Target 55%

Cannot explain the difference in Source and Network statistics.
Please note that for System 2 proxy and gateway server is not in the same subnet of esxi hosts and vcenter.

[PetrM]

Hello,

I wouldn't say that this is weird. The different systems have different data flow paths and report different "bottleneck" statistics: it's expected.

The Source load percentage ratio shows the time spent on fetching data. For instance, the task total time is 10 minutes, if the Data Mover was working 6,7 minutes and 3,2 minutes it was idling: the bottleneck is 67 %. If the Data Mover was reading data for 2 seconds and remained in idle state for 20 minutes, most probably it would be 0 %. The same logic applies to the Network stage, I suppose that the shared memory connection (used when Source and Target Data Movers are running on the same server) for some reason is not fast enough at the OS level of the proxy server in System 1.

Also, I'd never compare the results of the incremental runs, I recommend running full backups to get more precise data.

Thanks!

[filippoAdb]

>>The different systems have different data flow paths and report different "bottleneck" statistics

The data flow paths are not so different. The Esxi hosts are the same, so source is the same, the target Nas is the same, but dedicated shared folders are used.
In the first case proxy and gateway are on local B&R server and they are in the same subnet of source and target
In the second case proxy and gateway are on a different server in another subnet, so traffic flows through a firewall.
Another difference I overlooked is the server OS, win 2019 standard for the first, win 2016 standard for the second.
Source load varies from 67% to 0%, which sounds strange because source is the same.
My suspect is that part of the load moves from Source to Proxy in the second case, but I lack a clear understandings of what is going on.

[PetrM]

Hello,

I agree that both systems are close to each other from a data flow path perspective, but still, these are different systems: we have a firewall, another proxy with another OS in the second system. The fluctuations of percentage load ratios are absolutely expected: any system represents a conveyor where all components are mutually dependent.

The observation about Windows version is interesting because it may affect the data transmission speed over the shared memory when Data Movers are running on the same machine. Anyway, I recommend analyzing performance results based on full backup runs, not on incremental ones.

What is the ultimate goal of your research? Are there any jobs that do not fit the backup window?

Thanks!

[filippoAdb]

>>What is the ultimate goal of your research?

I have to backup 2 geographically distant sites connected via vpn. I can have a B&R server on each site or a single B&R with proxy and gateway on remote site. According to documentation both architecture are supported and there is no indication on how to choose between them. Doing the test I was ready to find different backup times (they are similar), but I was impressed by the different load statistics, which may be due to other overlooked factors (OS and subnet).

[PetrM]

Hello,

Our recommended practice is to keep a single Veeam B&R instance on the DR site to ensure a seamless DRP in case of the primary site failure.

I wouldn't be focused too much on the load statistics if the jobs performance is acceptable. From my point of view, your understanding of the product architecture is already good enough and what's the point to look for a precise "low-level" explanation of the difference in "bottleneck" ratios if there is no problem to solve?

Thanks!