How to Improve Source Bottlenecks

Availability for the Always-On Enterprise

How to Improve Source Bottlenecks

Veeam Logoby sdelacruz » Tue Apr 04, 2017 3:43 pm

Hello everyone.
I am trying to get some ideas on how to improve my backup jobs speeds.
My problem is a source bottleneck.
This is my current setup/
3 Dell Hosts Connected to 2 Dell Equallogic Arrays. 1 GB

Veeam Server is a physical servers with 16 GB RAM Dual Xeon Processor and 5 TB of Local Storage + 10 TB of QNAP iscsi
The Server also acts as my proxy - The server has direct access to the Dell Storage so all my backups are SAN mode.
Most of my jobs get the following statistics.
Load: Source 98% > Proxy 52% > Network 5% > Target 1%
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [san]

this is another job
Using backup proxy VMware Backup Proxy for disk Hard disk 2 [san]
Busy: Source 99% > Proxy 62% > Network 3% > Target 0%
Primary bottleneck: Source

Average Processing rate is 35-55 MB/s

What worries me is that I have a few servers that are now running at 2 TB each.
We are planning to upgrade our Veeam server to a newer model as well as maybe considering an EMC Datadomain for storage, but do now know if this even going to help me.

My question is how can I improve the source bottleneck.
IF I create a VM inside the cluster and use it as a Proxy, does that server need to have direct access to my Backup Repositories?
I always thought that San mode was going to be the fastest way to retrieve data from source.

Any help is greatly appreciated.

is there way to find out why the source is the bottleneck or how to speed it up. or is this the best speed i can look at getting?

Thanks
sdelacruz
Enthusiast
 
Posts: 48
Liked: never
Joined: Tue Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz

Re: How to Improve Source Bottlenecks

Veeam Logoby PTide » Wed Apr 05, 2017 1:43 pm

Hi,

3 Dell Hosts Connected to 2 Dell Equallogic Arrays. 1 GB
Is that a typo or you have 1Gb link between each of the hosts and the storage? What about the connection between proxy and storage?

Thank you
PTide
Veeam Software
 
Posts: 3222
Liked: 269 times
Joined: Tue May 19, 2015 1:46 pm

Re: How to Improve Source Bottlenecks

Veeam Logoby sdelacruz » Wed Apr 05, 2017 2:55 pm

Yes, Host to Storage is 1 GB
Physical Proxy to Storage is also 1GB

The Hosts and Proxy connect to same Storage Switch at 1GB.
sdelacruz
Enthusiast
 
Posts: 48
Liked: never
Joined: Tue Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz

Re: How to Improve Source Bottlenecks

Veeam Logoby PTide » Wed Apr 05, 2017 4:07 pm

I guess that slow network connection might be the reason. 1Gb gives you 120 MB/s in the best case scenario, when nothing else consumes bandwidth.

Thanks
PTide
Veeam Software
 
Posts: 3222
Liked: 269 times
Joined: Tue May 19, 2015 1:46 pm

Re: How to Improve Source Bottlenecks

Veeam Logoby sdelacruz » Wed Apr 05, 2017 4:20 pm

So my only option is to upgrade my SAN network to 10GB?

What about using a vm proxy? Would that improve?
sdelacruz
Enthusiast
 
Posts: 48
Liked: never
Joined: Tue Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz

Re: How to Improve Source Bottlenecks

Veeam Logoby Pat490 » Thu Apr 06, 2017 8:31 am

No, I guess the 1GB connection between Equallogic Storage and Backup Repository will always be a bottleneck
Pat490
Expert
 
Posts: 140
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: How to Improve Source Bottlenecks

Veeam Logoby PTide » Thu Apr 06, 2017 9:22 am

@sdelacruz
So my only option is to upgrade my SAN network to 10GB?
It seems so, however I'd frist double check if it is network to blame, not the disk array.

What about using a vm proxy? Would that improve?
I don't think so as the connection between the proxy and the storage would still be 1Gb.

Where is your backup job pointed to, Local storage or QNAP? How many NICs does your VBR server have?

@Pat490
Do you mean "between Equallogic and the server acting as a proxy?"

Thanks.
PTide
Veeam Software
 
Posts: 3222
Liked: 269 times
Joined: Tue May 19, 2015 1:46 pm

Re: How to Improve Source Bottlenecks

Veeam Logoby Pat490 » Fri Apr 07, 2017 9:13 am

I was thinking about transfer speed between production storage and backup repository in general
Pat490
Expert
 
Posts: 140
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: How to Improve Source Bottlenecks

Veeam Logoby PTide » Fri Apr 07, 2017 11:48 am

Well, if the primary job's target is iSCSI storage then the bandwidth will be consumed by both incoming (production storage --> proxy) and outcoming (proxy --> target storage) datastreams. If that is the case then I'd recommend to point the primary job to the local storage and schedule a backup copy job targeted to the network storage.

Thanks
PTide
Veeam Software
 
Posts: 3222
Liked: 269 times
Joined: Tue May 19, 2015 1:46 pm

Re: How to Improve Source Bottlenecks

Veeam Logoby rekun » Fri Apr 07, 2017 11:58 am

Hi
Your Equallogic most like have more than 1 ethernet port. If i recall correctly even the old models have 4 x 1GB, which when configured correctly should be using iscsi multipathing and therefore split the traffic evenly across all 4 NICs, which should equal a max transferrate close to 500 megabytes pr sec, much more than the 35-55 MBs you have right now.

How many nics are configured on the backup server for use as iscsi multipathing? Could you take a look at the load of the NICs during backup to see if it really is an network issue?

Also have you configured jumbo frames correctly on the Equallogic, all switches involved and the Veeam server?

Is Equallogic Host Integration Toolkit installed in the veeam server?
rekun
Service Provider
 
Posts: 18
Liked: 1 time
Joined: Tue Jun 02, 2015 7:24 pm
Full Name: Rene Knudsen

Re: How to Improve Source Bottlenecks

Veeam Logoby cst.jeremy » Sun Apr 09, 2017 10:17 pm

Here's a good multipathing discussion: viewtopic.php?f=2&t=42142. It specifically mentions EqualLogic.

It could be your 1GbE source array connectivity is the bottleneck. It could also be the disk configuration of the EqualLogic array.

How many disks? What type? 7K, 10K, 15K? How is RAID configured?
cst.jeremy
Lurker
 
Posts: 1
Liked: never
Joined: Mon Jan 30, 2017 9:39 pm
Location: Tulsa, OK
Full Name: Jeremy Johnson

Re: How to Improve Source Bottlenecks

Veeam Logoby daniel.farrelly » Mon Apr 10, 2017 4:08 pm

"source bottleneck" can be somewhat misleading, even if it's true. we get source bottlenecks while averaging ~2.5Gb/s for nightly backups. someone else in a previous post about source bottlenecks talked about using the term bottleneck and its negative connotation, like you're doing something wrong. don't take it personally; the computer is not waving its finger and rolling its eyes while stating your backups are slow 'cause your sources are inferior, puny lil human. it's just saying that something in relation to your source is highest point of congestion whether you're going 50Mb/s or 5Gb/s. in your case, i would look at your backup proxy and increase tasks to eight, twelve or sixteen. multipathing may help somewhat, but i'd recommend switching to a full 10/40G network stack.
daniel.farrelly
Influencer
 
Posts: 10
Liked: 4 times
Joined: Mon Feb 29, 2016 5:16 pm
Full Name: Daniel Farrelly

Re: How to Improve Source Bottlenecks

Veeam Logoby jmmarton » Fri Apr 14, 2017 6:52 pm

You are spot on Daniel. Something will always be the weakest link in the chain and that's what the bottleneck stats report. It doesn't mean that the weakest link is actually a problem. :-)

Joe
jmmarton
Veeam Software
 
Posts: 937
Liked: 100 times
Joined: Tue Nov 17, 2015 2:38 am
Location: Chicago, IL
Full Name: Joe Marton

Re: How to Improve Source Bottlenecks

Veeam Logoby nick.rigas@sjasd.ca » Mon May 01, 2017 7:44 pm 2 people like this post

Hi sdelcruz,

After reading this forum post, I thought to myself that our backend hardware is similar to yours and throughput was identical.

My coluegue searched for best practices for connecting servers to iSCSI SAN's until he improved our results. :D
Long story short and weeks later this is everything we did to improve results:
1) He broke a 2 network cards Team we set up into 2 individual network cards.
Each card on the iSCSI network had it's on IP address.
Thus having 2 x the amount of multipath I/O connections from each server to the SAN.
2) Each IP address has permission to connect to the SAN.
3) He created multiple LUN's
4) He also broke up the single large 20TB virtual disk on the SAN into 4 smaller 5 TB disks. (This was the big one and it really improved throughput).
5) Each HyperV host has access to all 4 smaller 5 TB virtual disks on the SAN. This quadroupled the throughput via Veeam B & R.

A combination of all 5 steps above improved the results. However step 4 & 5 made all Warnings and Errors disappear from the Event Logs. :o
I hope I summarized it all, but after a lot of digging, this has provided a major improvement for us.

Nick
nick.rigas@sjasd.ca
Novice
 
Posts: 7
Liked: 2 times
Joined: Tue Apr 18, 2017 1:58 pm
Full Name: Rigas

[MERGED] How to improve procesing Rate with Dell Equallogic

Veeam Logoby sdelacruz » Thu May 25, 2017 7:32 pm

I have the following issue. My backup processing rates are extremely low at 20-30 MBs/Sec
My setup as follows:

3 Host Cluster
2 iSCSI Equallogic array conected 1gb to Cluster
1 Physical Veeam Server - Direct connection to Equallogics
SAN Transport mode enabled. on all jobs

All my jobs are super slow. Rate avg is 25 Mb/s/. I am worried about the time it would take me to restore in case of a problem.

Is there anything I can do to improve rate? I know Dell Equallogic is not on the SAN mode option.

Any suggestions are greatly appreciated.
sdelacruz
Enthusiast
 
Posts: 48
Liked: never
Joined: Tue Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 14 guests