Teamed NICs for Backup Copy

[Moebius]

This is VB&R v7 on a Windows 2008R2 physical server with 38TB of onboard storage for backups, and doing backup copies to a QNAP NAS (attached via CIFS).
The server has 3 Gigabit NICs and the QNAP has 4. So I teamed a pair on the server and another on the NAS using IEEE 802.3ad Dynamic Link Aggregation and connected the pairs.

I was hoping to be able to fully use the 2Gbps link for my backup copy jobs. However, when running the initial copy the graph showed that the link usage was constantly at 50% or slightly below, with rare spikes up to 55%-60% max (but just spikes).

Trying to use multiple connections I started several backup copy jobs at the same time, but the link usage was still kind of capped at 50%. (In order to test the team I started a plain Windows file copy in the opposite direction at the same time, and this brought the link usage close to 90%, so the NIC team is working for sure).
Some internet digging revealed that link aggregation doesn't increase the bandwidth for a single TCP connection. I was hoping to be able to get a TCP connection for each backup copy job running, but apparently that's not the case.

Would I be better off just splitting the teams, assigning separate IP addresses to the NICs of the NAS, and adding it as a repository multiple times (a repository name for each distinct IP address), then addressing distinct backup copy jobs to "distinct" repositories?

[foggy]

Lucio, first of all, what were the bottleneck stats for this backup copy job?

[Moebius]

Bottleneck was "target" for all three jobs that i ran simultaneously:
#1: Load: Source 44% > Proxy 23% > Network 31% > Target 80%
#2: Load: Source 63% > Proxy 30% > Network 41% > Target 80%
#3: Load: Source 43% > Proxy 29% > Network 28% > Target 80%

[emachabert]

Do not forget that 802.3ad (LACP) won't give you 2Gbit/s per stream. it offers you 1gbit/s max per stream, but allows you to balance the streams over 2 gigabit links.
Moreover, the hash parameters used in the LACP association is deterministic of how the stream will be balanced between the links. For example, if you use ip based (src/dst) hash and you have only one destination (thus one ip dest), you will never use more than one link at a time.

[Moebius]

Right, I was hoping to get more streams by running several backup copy jobs at the same time. At the moment I have only one source ip (the Veeam server + main repository) and one destination ip (the NAS). Can I use a different hashing method and how?

[Shestakov]

Lucio,
Within one backup session Veeam B&R opens five parallel TCP/IP connections to transfer data from source to target. This happens inside single point-to-point(ip-ip) connection and should be enough to provide some load balancing in the LAG. Source ports will all be different for each of the connection so of the hash algorithm uses src/dst TCP port as part of it's LAG hashing algorithm.
Please read the interesting relevant discussion.
Thanks.

[Moebius]

Thank you Nikita, very interesting indeed.
I was going to use LACP for backup COPY jobs only and not for "standard" backup jobs. Does the five parallel TCP connections rule apply in this case also?

However, from what I saw in my test, it looks like either the rule doesn't apply or something along the chain does not support source-port based hashing, as I was not able to use sensibly more than 50% of my 2Gbit/s. My setup is like this:

Veeam Server |=========(LACP)=========|Juniper switches|=========(LACP)=======| QNAP NAS

So probably it's easier to give the NAS two separate ip addresses and to add it twice to the Veeam infrastructure as if they were two different repositories. I haven't tried this yet, and don't know if it is possible.

[Shestakov]

Lucio,
5 upload streams per job are set by default for every job session.
Please check the link I provided in the previous post, it explains various situations of Veeam-LAG operation.
However, from your job statistics looks like your bottleneck is "Target", improving it can bring more value to the performance.
Thanks.

[emachabert]

Just do what Nikita suggested:
1) ensure the hash algorithm is set to use TCP ports as parameters
2) retry a full backup which will produce a sequential write workload. Your QNAP will certainly be able to handle more iops in sequential write and you should go further than 1Gb/s (120MB/s)