Source bottleneck

VMware specific discussions

Source bottleneck

Veeam Logoby AlexWhit » Tue Dec 29, 2015 1:14 pm

I wonder if anyone can help here. i have 3 hosts that i am replicating in 2 replication jobs 2 hosts at the headoffice 1 remote host at a remote site connected via a 1gb link

Job 1 Headoffice to Remote Office
Local host 1, 2 guest os (1 backup proxy)
local host 2, 1 guest os (1 backup proxy)
to
Remote host 1 (1 backup proxy)

64MB/s
source:99%
Proxy:70%
Network:1%
target:15%

job 2 Remote Office to Head office
Remote host 1, 2 guest os(1 backup proxy)
to
local host 2, 1 guest os (1 backup proxy)

62MB/s
source:99%
Proxy:68%
Network:1%
target:18%


is there way to find out why the source is the bottleneck or how to speed it up. or is this the best speed i can look at getting?

Thanks
AlexWhit
Enthusiast
 
Posts: 34
Liked: never
Joined: Thu Jul 21, 2011 10:10 am
Full Name: Alex Whittaker

Re: Source bottleneck

Veeam Logoby foggy » Tue Dec 29, 2015 1:28 pm

Alex, what bottleneck stats do these jobs report and what transport mode is being used to retrieve VM data from source?
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Source bottleneck

Veeam Logoby AlexWhit » Tue Dec 29, 2015 1:39 pm

transport mode is [Hotadd]

i am unsure where to get the stats you require. the proxy's are set up as Virtual appliance and i have forced them to see the "local" data store

thanks
AlexWhit
Enthusiast
 
Posts: 34
Liked: never
Joined: Thu Jul 21, 2011 10:10 am
Full Name: Alex Whittaker

Re: Source bottleneck

Veeam Logoby AlexWhit » Tue Dec 29, 2015 1:41 pm

remote to head office
29/12/2015 13:00:55 :: Load: Source 99% > Proxy 68% > Network 1% > Target 18%

head Office to remote
29/12/2015 12:55:19 :: Load: Source 99% > Proxy 70% > Network 1% > Target 15%
AlexWhit
Enthusiast
 
Posts: 34
Liked: never
Joined: Thu Jul 21, 2011 10:10 am
Full Name: Alex Whittaker

Re: Source bottleneck

Veeam Logoby foggy » Tue Dec 29, 2015 2:07 pm

Bottleneck source means that your production VM storage cannot provide data any faster. That said, there always will be a bottleneck, in any setup and, as far as I can get, you're also backing up proxy server in these jobs, however, proxy cannot hotadd itself (so is backed up using network) and CBT cannot be used for backing it up, so this affects the overall job performance as well.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Source bottleneck

Veeam Logoby AlexWhit » Tue Dec 29, 2015 4:00 pm

Hi foggy

i have excluded the backup proxys in the replication and only replicate the live servers.

thanks for the info ill take a look to see if there is anything i can do to speed up the datastores on the hosts.
AlexWhit
Enthusiast
 
Posts: 34
Liked: never
Joined: Thu Jul 21, 2011 10:10 am
Full Name: Alex Whittaker

Re: Source bottleneck

Veeam Logoby Delo123 » Tue Dec 29, 2015 7:00 pm

Maybe you can try to add an additional vcpu to the proxy as load on that is also "a bit" high to see if it changes anything....

Ps. 60MB/s isn't that bad for a gigabit link...
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Source bottleneck

Veeam Logoby bgalante » Sun Jan 03, 2016 11:39 pm

Just a thought...

You are only backing up / replicating very few boxes. These are important boxes I assume.
Is there any other type of backup running on these boxes at the same time as your replication job?
i.e. DB Dump. Tape backup..

What is the total size of the backup /repl job, and what is the duration of the job?

Also, Are you doing replication job from the VM environment itself? I have had issues with that. How about doing a backup job to some cifs, iscsi, other target, then replicating that.

All my problems went away when I stopped repliating from the live vm environment and did a backup first, then replicated from backup.
bgalante
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Mon Jul 27, 2015 5:14 pm
Full Name: Brian Galante

Re: Source bottleneck

Veeam Logoby AlexWhit » Mon Jan 04, 2016 10:12 am

Hi Bglante
the hosts that i am replicating are mission critical boxes, the guests on them are our ERP,wearhouse mamagement, Exchange, SQl and accounts servers.

Veeam is installed on a physical Box but the proxys are virtual.

no backup is running the main backup is at night to the physical veeam box and that runs at 11pm

i need to replicate the backup as they are at different sites and would like to keep the "offsite" replications as current as possible

thanks
AlexWhit
Enthusiast
 
Posts: 34
Liked: never
Joined: Thu Jul 21, 2011 10:10 am
Full Name: Alex Whittaker

[MERGED] Bottlenect source 99%

Veeam Logoby ober72 » Fri May 27, 2016 4:23 pm

Hi Folks,

I guess if my bottleneck on all of my jobs is 99% then switching transport modes from network to virtual appliance won't make a lot of difference? Or am I missing something here.

Is there a guideline of how to investigate source bottlenecks, I will search the forums of course :) but if someone has it or a link handy that would be greatly appreciated.

cheers
Geoff Burke
VMCE V9
ober72
Service Provider
 
Posts: 263
Liked: 38 times
Joined: Fri Jan 24, 2014 4:10 pm
Location: CANADA
Full Name: Geoff Burke

Re: Source bottleneck

Veeam Logoby Shestakov » Fri May 27, 2016 5:47 pm

ober72 wrote:I guess if my bottleneck on all of my jobs is 99% then switching transport modes from network to virtual appliance won't make a lot of difference? Or am I missing something here.

Сonversely, changing transport mode is the best way to improve the "source" statistics. Here is a link to the corresponding user guide section.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Source bottleneck

Veeam Logoby folerx » Wed Aug 03, 2016 8:37 am

hi
Now I run test backup job for first time. Backup over 1gbps Lan to shared folder on external 6tb usb 3.0 hdd. vcenter to hyperv guest.
Summary state that source is bottleneck.
Percent busy for processing stages:
Source 15%
Proxy 2%
Network 0%
Target 0%

How to identify which component on source is bottleneck?
folerx
Expert
 
Posts: 105
Liked: 8 times
Joined: Wed Jun 22, 2016 9:47 pm
Full Name: Daniel Kaiser

Re: Source bottleneck

Veeam Logoby foggy » Wed Aug 03, 2016 10:51 am

It is the source disk reader, however, overall bottleneck stats look strange to me (numbers are typically larger). Is it a regular backup (not backup copy) job?
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Source bottleneck

Veeam Logoby Shestakov » Thu Aug 04, 2016 11:49 am

I`m also curious if you have these results all the time for that job and what is the size of backup.
Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

[MERGED] Very slow over network...

Veeam Logoby folerx » Sun Aug 28, 2016 8:08 pm

hi
I read this page:
http://bp.veeam.expert/resource_plannin ... _mode.html

It can be rather slow on 1 Gb Ethernet (about 10-20 MB/s) due to throttling mechanisms so restores via the Network mode can take quite a long time with 1GbE.+

backups run with 40MBs from different datastores. 4 core i7, two job parallel.
all jobs have same perfomance bottleneck: Source
Busy: Source 99% > Proxy 50% > Network 4% > Target 0%
proxy 50% because i use extreme compression. but network is only 4%. :?:

any way to speed up this? registry hack, 4gbps network adapter....?
folerx
Expert
 
Posts: 105
Liked: 8 times
Joined: Wed Jun 22, 2016 9:47 pm
Full Name: Daniel Kaiser

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 13 guests