Comprehensive data protection for all workloads
Post Reply
Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Teamed NICs for Backup Copy

Post by Moebius » Nov 20, 2014 10:24 am

This is VB&R v7 on a Windows 2008R2 physical server with 38TB of onboard storage for backups, and doing backup copies to a QNAP NAS (attached via CIFS).
The server has 3 Gigabit NICs and the QNAP has 4. So I teamed a pair on the server and another on the NAS using IEEE 802.3ad Dynamic Link Aggregation and connected the pairs.

I was hoping to be able to fully use the 2Gbps link for my backup copy jobs. However, when running the initial copy the graph showed that the link usage was constantly at 50% or slightly below, with rare spikes up to 55%-60% max (but just spikes).

Trying to use multiple connections I started several backup copy jobs at the same time, but the link usage was still kind of capped at 50%. (In order to test the team I started a plain Windows file copy in the opposite direction at the same time, and this brought the link usage close to 90%, so the NIC team is working for sure).
Some internet digging revealed that link aggregation doesn't increase the bandwidth for a single TCP connection. I was hoping to be able to get a TCP connection for each backup copy job running, but apparently that's not the case.

Would I be better off just splitting the teams, assigning separate IP addresses to the NICs of the NAS, and adding it as a repository multiple times (a repository name for each distinct IP address), then addressing distinct backup copy jobs to "distinct" repositories?

foggy
Veeam Software
Posts: 18114
Liked: 1536 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Teamed NICs for Backup Copy

Post by foggy » Nov 20, 2014 12:40 pm

Lucio, first of all, what were the bottleneck stats for this backup copy job?

Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Teamed NICs for Backup Copy

Post by Moebius » Nov 20, 2014 12:59 pm

Bottleneck was "target" for all three jobs that i ran simultaneously:
#1: Load: Source 44% > Proxy 23% > Network 31% > Target 80%
#2: Load: Source 63% > Proxy 30% > Network 41% > Target 80%
#3: Load: Source 43% > Proxy 29% > Network 28% > Target 80%

emachabert
Veeam Vanguard
Posts: 370
Liked: 164 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Teamed NICs for Backup Copy

Post by emachabert » Nov 20, 2014 1:18 pm

Do not forget that 802.3ad (LACP) won't give you 2Gbit/s per stream. it offers you 1gbit/s max per stream, but allows you to balance the streams over 2 gigabit links.
Moreover, the hash parameters used in the LACP association is deterministic of how the stream will be balanced between the links. For example, if you use ip based (src/dst) hash and you have only one destination (thus one ip dest), you will never use more than one link at a time.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Teamed NICs for Backup Copy

Post by Moebius » Nov 20, 2014 1:34 pm

Right, I was hoping to get more streams by running several backup copy jobs at the same time. At the moment I have only one source ip (the Veeam server + main repository) and one destination ip (the NAS). Can I use a different hashing method and how?

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Teamed NICs for Backup Copy

Post by Shestakov » Nov 20, 2014 2:04 pm

Lucio,
Within one backup session Veeam B&R opens five parallel TCP/IP connections to transfer data from source to target. This happens inside single point-to-point(ip-ip) connection and should be enough to provide some load balancing in the LAG. Source ports will all be different for each of the connection so of the hash algorithm uses src/dst TCP port as part of it's LAG hashing algorithm.
Please read the interesting relevant discussion.
Thanks.

Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Teamed NICs for Backup Copy

Post by Moebius » Nov 20, 2014 2:41 pm

Thank you Nikita, very interesting indeed.
I was going to use LACP for backup COPY jobs only and not for "standard" backup jobs. Does the five parallel TCP connections rule apply in this case also?

However, from what I saw in my test, it looks like either the rule doesn't apply or something along the chain does not support source-port based hashing, as I was not able to use sensibly more than 50% of my 2Gbit/s. My setup is like this:

Veeam Server |=========(LACP)=========|Juniper switches|=========(LACP)=======| QNAP NAS

So probably it's easier to give the NAS two separate ip addresses and to add it twice to the Veeam infrastructure as if they were two different repositories. I haven't tried this yet, and don't know if it is possible.

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Teamed NICs for Backup Copy

Post by Shestakov » Nov 20, 2014 3:44 pm

Lucio,
5 upload streams per job are set by default for every job session.
Please check the link I provided in the previous post, it explains various situations of Veeam-LAG operation.
However, from your job statistics looks like your bottleneck is "Target", improving it can bring more value to the performance.
Thanks.

emachabert
Veeam Vanguard
Posts: 370
Liked: 164 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Teamed NICs for Backup Copy

Post by emachabert » Nov 20, 2014 4:13 pm

Just do what Nikita suggested:
1) ensure the hash algorithm is set to use TCP ports as parameters
2) retry a full backup which will produce a sequential write workload. Your QNAP will certainly be able to handle more iops in sequential write and you should go further than 1Gb/s (120MB/s)
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

Post Reply

Who is online

Users browsing this forum: mcz, natmar25 and 60 guests