Source bottleneck

[AlexWhit]

I wonder if anyone can help here. i have 3 hosts that i am replicating in 2 replication jobs 2 hosts at the headoffice 1 remote host at a remote site connected via a 1gb link

Job 1 Headoffice to Remote Office
Local host 1, 2 guest os (1 backup proxy)
local host 2, 1 guest os (1 backup proxy)
to
Remote host 1 (1 backup proxy)

64MB/s
source:99%
Proxy:70%
Network:1%
target:15%

job 2 Remote Office to Head office
Remote host 1, 2 guest os(1 backup proxy)
to
local host 2, 1 guest os (1 backup proxy)

62MB/s
source:99%
Proxy:68%
Network:1%
target:18%


is there way to find out why the source is the bottleneck or how to speed it up. or is this the best speed i can look at getting?

Thanks

[foggy]

Alex, what bottleneck stats do these jobs report and what transport mode is being used to retrieve VM data from source?

[AlexWhit]

transport mode is [Hotadd]

i am unsure where to get the stats you require. the proxy's are set up as Virtual appliance and i have forced them to see the "local" data store

thanks

[AlexWhit]

remote to head office
29/12/2015 13:00:55 :: Load: Source 99% > Proxy 68% > Network 1% > Target 18%

head Office to remote
29/12/2015 12:55:19 :: Load: Source 99% > Proxy 70% > Network 1% > Target 15%

[foggy]

Bottleneck source means that your production VM storage cannot provide data any faster. That said, there always will be a bottleneck, in any setup and, as far as I can get, you're also backing up proxy server in these jobs, however, proxy cannot hotadd itself (so is backed up using network) and CBT cannot be used for backing it up, so this affects the overall job performance as well.

[AlexWhit]

Hi foggy

i have excluded the backup proxys in the replication and only replicate the live servers.

thanks for the info ill take a look to see if there is anything i can do to speed up the datastores on the hosts.

[Delo123]

Maybe you can try to add an additional vcpu to the proxy as load on that is also "a bit" high to see if it changes anything....

Ps. 60MB/s isn't that bad for a gigabit link...

[bgalante]

Just a thought...

You are only backing up / replicating very few boxes. These are important boxes I assume.
Is there any other type of backup running on these boxes at the same time as your replication job?
i.e. DB Dump. Tape backup..

What is the total size of the backup /repl job, and what is the duration of the job?

Also, Are you doing replication job from the VM environment itself? I have had issues with that. How about doing a backup job to some cifs, iscsi, other target, then replicating that.

All my problems went away when I stopped repliating from the live vm environment and did a backup first, then replicated from backup.

[AlexWhit]

Hi Bglante
the hosts that i am replicating are mission critical boxes, the guests on them are our ERP,wearhouse mamagement, Exchange, SQl and accounts servers.

Veeam is installed on a physical Box but the proxys are virtual.

no backup is running the main backup is at night to the physical veeam box and that runs at 11pm

i need to replicate the backup as they are at different sites and would like to keep the "offsite" replications as current as possible

thanks

[ober72]

Hi Folks,

I guess if my bottleneck on all of my jobs is 99% then switching transport modes from network to virtual appliance won't make a lot of difference? Or am I missing something here.

Is there a guideline of how to investigate source bottlenecks, I will search the forums of course but if someone has it or a link handy that would be greatly appreciated.

cheers

[Shestakov]

I guess if my bottleneck on all of my jobs is 99% then switching transport modes from network to virtual appliance won't make a lot of difference? Or am I missing something here.

Сonversely, changing transport mode is the best way to improve the "source" statistics. Here is a link to the corresponding user guide section.

[folerx]

hi
Now I run test backup job for first time. Backup over 1gbps Lan to shared folder on external 6tb usb 3.0 hdd. vcenter to hyperv guest.
Summary state that source is bottleneck.
Percent busy for processing stages:
Source 15%
Proxy 2%
Network 0%
Target 0%

How to identify which component on source is bottleneck?

[foggy]

It is the source disk reader, however, overall bottleneck stats look strange to me (numbers are typically larger). Is it a regular backup (not backup copy) job?

[Shestakov]

I`m also curious if you have these results all the time for that job and what is the size of backup.
Thanks!

[folerx]

hi
I read this page:
http://bp.veeam.expert/resource_plannin ... _mode.html

It can be rather slow on 1 Gb Ethernet (about 10-20 MB/s) due to throttling mechanisms so restores via the Network mode can take quite a long time with 1GbE.+

backups run with 40MBs from different datastores. 4 core i7, two job parallel.
all jobs have same perfomance bottleneck: Source
Busy: Source 99% > Proxy 50% > Network 4% > Target 0%
proxy 50% because i use extreme compression. but network is only 4%.

any way to speed up this? registry hack, 4gbps network adapter....?

[Shestakov]

Network transport mode doesn`t mean that "Network" is going to be a bottleneck.
Transport mode is a method that is used by the Veeam Data Mover Service to retrieve VM data from the source and write VM data to the target, while Network in a bottleneck analysis is percent of time while processed data is being taken from the proxy component, and sends it over the network to the target component.
The way to speed up the process is to use faster transport mode, i.e. hot-add or direct SAN.
Thanks!