Comprehensive data protection for all workloads
Post Reply
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

How to Improve Source Bottlenecks

Post by sdelacruz »

Hello everyone.
I am trying to get some ideas on how to improve my backup jobs speeds.
My problem is a source bottleneck.
This is my current setup/
3 Dell Hosts Connected to 2 Dell Equallogic Arrays. 1 GB

Veeam Server is a physical servers with 16 GB RAM Dual Xeon Processor and 5 TB of Local Storage + 10 TB of QNAP iscsi
The Server also acts as my proxy - The server has direct access to the Dell Storage so all my backups are SAN mode.
Most of my jobs get the following statistics.
Load: Source 98% > Proxy 52% > Network 5% > Target 1%
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [san]

this is another job
Using backup proxy VMware Backup Proxy for disk Hard disk 2 [san]
Busy: Source 99% > Proxy 62% > Network 3% > Target 0%
Primary bottleneck: Source

Average Processing rate is 35-55 MB/s

What worries me is that I have a few servers that are now running at 2 TB each.
We are planning to upgrade our Veeam server to a newer model as well as maybe considering an EMC Datadomain for storage, but do now know if this even going to help me.

My question is how can I improve the source bottleneck.
IF I create a VM inside the cluster and use it as a Proxy, does that server need to have direct access to my Backup Repositories?
I always thought that San mode was going to be the fastest way to retrieve data from source.

Any help is greatly appreciated.

is there way to find out why the source is the bottleneck or how to speed it up. or is this the best speed i can look at getting?

Thanks
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: How to Improve Source Bottlenecks

Post by PTide »

Hi,
3 Dell Hosts Connected to 2 Dell Equallogic Arrays. 1 GB
Is that a typo or you have 1Gb link between each of the hosts and the storage? What about the connection between proxy and storage?

Thank you
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

Re: How to Improve Source Bottlenecks

Post by sdelacruz »

Yes, Host to Storage is 1 GB
Physical Proxy to Storage is also 1GB

The Hosts and Proxy connect to same Storage Switch at 1GB.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: How to Improve Source Bottlenecks

Post by PTide »

I guess that slow network connection might be the reason. 1Gb gives you 120 MB/s in the best case scenario, when nothing else consumes bandwidth.

Thanks
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

Re: How to Improve Source Bottlenecks

Post by sdelacruz »

So my only option is to upgrade my SAN network to 10GB?

What about using a vm proxy? Would that improve?
Pat490
Expert
Posts: 170
Liked: 29 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: How to Improve Source Bottlenecks

Post by Pat490 »

No, I guess the 1GB connection between Equallogic Storage and Backup Repository will always be a bottleneck
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: How to Improve Source Bottlenecks

Post by PTide »

@sdelacruz
So my only option is to upgrade my SAN network to 10GB?
It seems so, however I'd frist double check if it is network to blame, not the disk array.
What about using a vm proxy? Would that improve?
I don't think so as the connection between the proxy and the storage would still be 1Gb.

Where is your backup job pointed to, Local storage or QNAP? How many NICs does your VBR server have?

@Pat490
Do you mean "between Equallogic and the server acting as a proxy?"

Thanks.
Pat490
Expert
Posts: 170
Liked: 29 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: How to Improve Source Bottlenecks

Post by Pat490 »

I was thinking about transfer speed between production storage and backup repository in general
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: How to Improve Source Bottlenecks

Post by PTide »

Well, if the primary job's target is iSCSI storage then the bandwidth will be consumed by both incoming (production storage --> proxy) and outcoming (proxy --> target storage) datastreams. If that is the case then I'd recommend to point the primary job to the local storage and schedule a backup copy job targeted to the network storage.

Thanks
rekun
Service Provider
Posts: 18
Liked: 1 time
Joined: Jun 02, 2015 7:24 pm
Full Name: Rene Knudsen
Contact:

Re: How to Improve Source Bottlenecks

Post by rekun »

Hi
Your Equallogic most like have more than 1 ethernet port. If i recall correctly even the old models have 4 x 1GB, which when configured correctly should be using iscsi multipathing and therefore split the traffic evenly across all 4 NICs, which should equal a max transferrate close to 500 megabytes pr sec, much more than the 35-55 MBs you have right now.

How many nics are configured on the backup server for use as iscsi multipathing? Could you take a look at the load of the NICs during backup to see if it really is an network issue?

Also have you configured jumbo frames correctly on the Equallogic, all switches involved and the Veeam server?

Is Equallogic Host Integration Toolkit installed in the veeam server?
cst.jeremy
Lurker
Posts: 1
Liked: never
Joined: Jan 30, 2017 9:39 pm
Full Name: Jeremy Johnson
Location: Tulsa, OK
Contact:

Re: How to Improve Source Bottlenecks

Post by cst.jeremy »

Here's a good multipathing discussion: viewtopic.php?f=2&t=42142. It specifically mentions EqualLogic.

It could be your 1GbE source array connectivity is the bottleneck. It could also be the disk configuration of the EqualLogic array.

How many disks? What type? 7K, 10K, 15K? How is RAID configured?
daniel.farrelly
Influencer
Posts: 15
Liked: 5 times
Joined: Feb 29, 2016 5:16 pm
Full Name: Daniel Farrelly
Contact:

Re: How to Improve Source Bottlenecks

Post by daniel.farrelly »

"source bottleneck" can be somewhat misleading, even if it's true. we get source bottlenecks while averaging ~2.5Gb/s for nightly backups. someone else in a previous post about source bottlenecks talked about using the term bottleneck and its negative connotation, like you're doing something wrong. don't take it personally; the computer is not waving its finger and rolling its eyes while stating your backups are slow 'cause your sources are inferior, puny lil human. it's just saying that something in relation to your source is highest point of congestion whether you're going 50Mb/s or 5Gb/s. in your case, i would look at your backup proxy and increase tasks to eight, twelve or sixteen. multipathing may help somewhat, but i'd recommend switching to a full 10/40G network stack.
jmmarton
Veeam Software
Posts: 2092
Liked: 309 times
Joined: Nov 17, 2015 2:38 am
Full Name: Joe Marton
Location: Chicago, IL
Contact:

Re: How to Improve Source Bottlenecks

Post by jmmarton »

You are spot on Daniel. Something will always be the weakest link in the chain and that's what the bottleneck stats report. It doesn't mean that the weakest link is actually a problem. :-)

Joe
nickforum
Novice
Posts: 7
Liked: 2 times
Joined: Apr 18, 2017 1:58 pm
Full Name: Rigas
Contact:

Re: How to Improve Source Bottlenecks

Post by nickforum » 2 people like this post

Hi sdelcruz,

After reading this forum post, I thought to myself that our backend hardware is similar to yours and throughput was identical.

My coluegue searched for best practices for connecting servers to iSCSI SAN's until he improved our results. :D
Long story short and weeks later this is everything we did to improve results:
1) He broke a 2 network cards Team we set up into 2 individual network cards.
Each card on the iSCSI network had it's on IP address.
Thus having 2 x the amount of multipath I/O connections from each server to the SAN.
2) Each IP address has permission to connect to the SAN.
3) He created multiple LUN's
4) He also broke up the single large 20TB virtual disk on the SAN into 4 smaller 5 TB disks. (This was the big one and it really improved throughput).
5) Each HyperV host has access to all 4 smaller 5 TB virtual disks on the SAN. This quadroupled the throughput via Veeam B & R.

A combination of all 5 steps above improved the results. However step 4 & 5 made all Warnings and Errors disappear from the Event Logs. :o
I hope I summarized it all, but after a lot of digging, this has provided a major improvement for us.

Nick
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

[MERGED] How to improve procesing Rate with Dell Equallogic

Post by sdelacruz »

I have the following issue. My backup processing rates are extremely low at 20-30 MBs/Sec
My setup as follows:

3 Host Cluster
2 iSCSI Equallogic array conected 1gb to Cluster
1 Physical Veeam Server - Direct connection to Equallogics
SAN Transport mode enabled. on all jobs

All my jobs are super slow. Rate avg is 25 Mb/s/. I am worried about the time it would take me to restore in case of a problem.

Is there anything I can do to improve rate? I know Dell Equallogic is not on the SAN mode option.

Any suggestions are greatly appreciated.
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: How to improve procesing Rate with Dell Equallogic Array

Post by DGrinev »

Hi Sam,

Please share the bottleneck stats of the job. Thanks!
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

Re: How to improve procesing Rate with Dell Equallogic Array

Post by sdelacruz »

These are some of the bottleneck on several jobs
Job1
5/25/2017 6:21:36 PM :: Load: Source 99% > Proxy 34% > Network 1% > Target 0% - Procesing Rate 24 MB/s
job2
5/25/2017 10:17:00 PM :: Load: Source 99% > Proxy 36% > Network 2% > Target 0% - Processing rate 19MB/s
Job 3
5/26/2017 4:04:35 AM :: Load: Source 99% > Proxy 52% > Network 3% > Target 0% Processing Rate 21MB/s
-

My storage for hosts are 2 equallogic arrays. Jobs are running as San mode.

Thanks in advance!
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

Re: How to improve procesing Rate with Dell Equallogic Array

Post by sdelacruz »

I just installed a vm on one of the hosts. I am using it as a proxy.
Now the rate has improved to 85/95 MB/s but only for backup jobs of vms that reside on same host as the proxy.

Do i Need to have a proxy on each host so speeds can improve?
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: How to improve procesing Rate with Dell Equallogic Array

Post by DGrinev »

Hi Sam,

Can you check what transport mode has been utilized during the processing? You can see it in the job session log, choose a particular VM on the left right after the proxy server name, there will be a transport mode [tag].
sdelacruz wrote:Do i Need to have a proxy on each host so speeds can improve?
You should deploy backup proxy on each host only if you are utilizing Hotadd mode without shared data storage.
sdelacruz
Enthusiast
Posts: 64
Liked: 1 time
Joined: Feb 01, 2011 8:09 pm
Full Name: Sam De La Cruz
Contact:

Re: How to improve procesing Rate with Dell Equallogic Array

Post by sdelacruz »

Jobs that are using SAN mode. Like I mention, since I am using Dell Equallogic, i am not gettting the full benifits of SAN connection.
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: How to improve procesing Rate with Dell Equallogic Array

Post by DGrinev »

Hi Sam,

Please review existing topic above with considerations on how to improve the processing rate. Thanks!
JohnPassion
Lurker
Posts: 1
Liked: 1 time
Joined: May 19, 2021 1:54 pm
Full Name: João Eduardo Guerra Paixão
Contact:

Re: How to Improve Source Bottlenecks

Post by JohnPassion » 1 person likes this post

Hello,

I have implemented a couple years ago a Veeam B&R with a dedicated VM to serve as proxy even tho it is in the same host and shares disks in my san(not good practice), and what i have noticed from lots of tests that i made is that:

Everytime there is a connection from one component to another you will loose troughtoutput.
1st - Network will always be the bottleneck since basic HDD raids will always outperform a 1Gbit network. Since 10Gbit is becoming cheap, i suggest you start from there.
2nd - Source disks and how they are connected to the source machine vs destination disks layout - Since processing will be done by the agent on the source machine via your proxy machine the faster the disks and their connection type and the raid configurations on both source and destination, the faster you will process the information you are backing up and the faster you will finish your backup.
3rd - Feeding your Veeam proxy repository with ISCSI disks is a lot faster than feeding a shared SMB repository or other methods because of how those different protocols work.

So my advice to you is:
1st switch your network. 1Gbit is outdated and doesnt keep up with todays information output from newer HDDS and SSD.s
2nd - Whatever you do, dont run VM.s from ISCSI disks connected to your HOST.s via a 1Gbit network. That is roadkill.
Either direct connect the disks on your servers raid controler that run the hosts or Fibre Wire a san and map the disks to your host. ISCSI a san via 1 Gbit network for a host is extremely slow for todays standarts and it will be your biggest bottleneck.
2nd check your raid config troughoutput and see what speed your disks are running at when reading and writing on both ends (source and destination) and change accordingly to your needs.(i dont advise running HDD.s on the OS guests since VM.s because extremely slow to respond).
3rd - always configure your repository with ISCSI disks so that it can write and read directly on the destination you like.

My results are in the screenshots below:
https://imagizer.imageshack.com/img924/5223/fXw9XF.jpg - Backups from VMS(shared with proxy) over 10Gbit network to NAS in Raid 5 maped in ISCSI from a source of Raid 50 over fiberwire direct attached storage.

https://imagizer.imageshack.com/img924/2156/1NZvLr.jpg - Backups from our Physical Servers and computers over 10Gbit connections (same backup system) to a NAS in raid 5 also with 10bit connection.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 218 guests