Discussions specific to the VMware vSphere hypervisor
Post Reply
AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Source bottleneck

Post by AlexWhit » Dec 29, 2015 1:14 pm

I wonder if anyone can help here. i have 3 hosts that i am replicating in 2 replication jobs 2 hosts at the headoffice 1 remote host at a remote site connected via a 1gb link

Job 1 Headoffice to Remote Office
Local host 1, 2 guest os (1 backup proxy)
local host 2, 1 guest os (1 backup proxy)
to
Remote host 1 (1 backup proxy)

64MB/s
source:99%
Proxy:70%
Network:1%
target:15%

job 2 Remote Office to Head office
Remote host 1, 2 guest os(1 backup proxy)
to
local host 2, 1 guest os (1 backup proxy)

62MB/s
source:99%
Proxy:68%
Network:1%
target:18%


is there way to find out why the source is the bottleneck or how to speed it up. or is this the best speed i can look at getting?

Thanks

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Source bottleneck

Post by foggy » Dec 29, 2015 1:28 pm

Alex, what bottleneck stats do these jobs report and what transport mode is being used to retrieve VM data from source?

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Source bottleneck

Post by AlexWhit » Dec 29, 2015 1:39 pm

transport mode is [Hotadd]

i am unsure where to get the stats you require. the proxy's are set up as Virtual appliance and i have forced them to see the "local" data store

thanks

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Source bottleneck

Post by AlexWhit » Dec 29, 2015 1:41 pm

remote to head office
29/12/2015 13:00:55 :: Load: Source 99% > Proxy 68% > Network 1% > Target 18%

head Office to remote
29/12/2015 12:55:19 :: Load: Source 99% > Proxy 70% > Network 1% > Target 15%

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Source bottleneck

Post by foggy » Dec 29, 2015 2:07 pm

Bottleneck source means that your production VM storage cannot provide data any faster. That said, there always will be a bottleneck, in any setup and, as far as I can get, you're also backing up proxy server in these jobs, however, proxy cannot hotadd itself (so is backed up using network) and CBT cannot be used for backing it up, so this affects the overall job performance as well.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Source bottleneck

Post by AlexWhit » Dec 29, 2015 4:00 pm

Hi foggy

i have excluded the backup proxys in the replication and only replicate the live servers.

thanks for the info ill take a look to see if there is anything i can do to speed up the datastores on the hosts.

Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Source bottleneck

Post by Delo123 » Dec 29, 2015 7:00 pm

Maybe you can try to add an additional vcpu to the proxy as load on that is also "a bit" high to see if it changes anything....

Ps. 60MB/s isn't that bad for a gigabit link...

bgalante
Enthusiast
Posts: 43
Liked: 2 times
Joined: Jul 27, 2015 5:14 pm
Full Name: Brian Galante
Contact:

Re: Source bottleneck

Post by bgalante » Jan 03, 2016 11:39 pm

Just a thought...

You are only backing up / replicating very few boxes. These are important boxes I assume.
Is there any other type of backup running on these boxes at the same time as your replication job?
i.e. DB Dump. Tape backup..

What is the total size of the backup /repl job, and what is the duration of the job?

Also, Are you doing replication job from the VM environment itself? I have had issues with that. How about doing a backup job to some cifs, iscsi, other target, then replicating that.

All my problems went away when I stopped repliating from the live vm environment and did a backup first, then replicated from backup.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Source bottleneck

Post by AlexWhit » Jan 04, 2016 10:12 am

Hi Bglante
the hosts that i am replicating are mission critical boxes, the guests on them are our ERP,wearhouse mamagement, Exchange, SQl and accounts servers.

Veeam is installed on a physical Box but the proxys are virtual.

no backup is running the main backup is at night to the physical veeam box and that runs at 11pm

i need to replicate the backup as they are at different sites and would like to keep the "offsite" replications as current as possible

thanks

ober72
Service Provider
Posts: 522
Liked: 74 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Location: CANADA
Contact:

[MERGED] Bottlenect source 99%

Post by ober72 » May 27, 2016 4:23 pm

Hi Folks,

I guess if my bottleneck on all of my jobs is 99% then switching transport modes from network to virtual appliance won't make a lot of difference? Or am I missing something here.

Is there a guideline of how to investigate source bottlenecks, I will search the forums of course :) but if someone has it or a link handy that would be greatly appreciated.

cheers
Geoff Burke
Veeam Certified Architect

Shestakov
Veeam Software
Posts: 6974
Liked: 714 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Source bottleneck

Post by Shestakov » May 27, 2016 5:47 pm

ober72 wrote:I guess if my bottleneck on all of my jobs is 99% then switching transport modes from network to virtual appliance won't make a lot of difference? Or am I missing something here.
Сonversely, changing transport mode is the best way to improve the "source" statistics. Here is a link to the corresponding user guide section.

folerx
Expert
Posts: 105
Liked: 8 times
Joined: Jun 22, 2016 9:47 pm
Full Name: Daniel Kaiser
Contact:

Re: Source bottleneck

Post by folerx » Aug 03, 2016 8:37 am

hi
Now I run test backup job for first time. Backup over 1gbps Lan to shared folder on external 6tb usb 3.0 hdd. vcenter to hyperv guest.
Summary state that source is bottleneck.
Percent busy for processing stages:
Source 15%
Proxy 2%
Network 0%
Target 0%

How to identify which component on source is bottleneck?

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Source bottleneck

Post by foggy » Aug 03, 2016 10:51 am

It is the source disk reader, however, overall bottleneck stats look strange to me (numbers are typically larger). Is it a regular backup (not backup copy) job?

Shestakov
Veeam Software
Posts: 6974
Liked: 714 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Source bottleneck

Post by Shestakov » Aug 04, 2016 11:49 am

I`m also curious if you have these results all the time for that job and what is the size of backup.
Thanks!

folerx
Expert
Posts: 105
Liked: 8 times
Joined: Jun 22, 2016 9:47 pm
Full Name: Daniel Kaiser
Contact:

[MERGED] Very slow over network...

Post by folerx » Aug 28, 2016 8:08 pm

hi
I read this page:
http://bp.veeam.expert/resource_plannin ... _mode.html

It can be rather slow on 1 Gb Ethernet (about 10-20 MB/s) due to throttling mechanisms so restores via the Network mode can take quite a long time with 1GbE.+

backups run with 40MBs from different datastores. 4 core i7, two job parallel.
all jobs have same perfomance bottleneck: Source
Busy: Source 99% > Proxy 50% > Network 4% > Target 0%
proxy 50% because i use extreme compression. but network is only 4%. :?:

any way to speed up this? registry hack, 4gbps network adapter....?

Shestakov
Veeam Software
Posts: 6974
Liked: 714 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Source bottleneck

Post by Shestakov » Aug 28, 2016 10:03 pm

Network transport mode doesn`t mean that "Network" is going to be a bottleneck.
Transport mode is a method that is used by the Veeam Data Mover Service to retrieve VM data from the source and write VM data to the target, while Network in a bottleneck analysis is percent of time while processed data is being taken from the proxy component, and sends it over the network to the target component.
The way to speed up the process is to use faster transport mode, i.e. hot-add or direct SAN.
Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests