Multi-NIC Backups

[TheGrove]

From what I've seen the only way to use multiple NIC's for backups is to link a repository to a NIC and force the backup to use that repository/NIC combination.

Has anyone seen any documentation, articles, etc. that give the steps to do this and get it working. I've seen some messages about people doing this but i have not seen anyone document what they did to get it to work. I would like to try this but I'm unclear about the actual steps to implement.

Thanks.

[TheGrove]

I meant to add we are using SOBR repositories if it matters.

[veremin]

Hi, Bill, can you shed some light on what you're trying to achieve or solve? Make backups pointed to the same repository use different network to transfer data traffic? Team multiple networks adapters on the repository server? Something else? Thanks!

[TheGrove]

Sure, you can see my original post at veeam-backup-replication-f2/network-per ... 77790.html where i talk about my environment and how i have teamed 4 NiCs together but only one of them is used for data flowing into the repository.

I'm trying to get the system to use multiple NICs in parallel to increase throughput. From what i've seen the only way to do this is to have separate repositories and link a NIC to the repository. Then have each job use a different repository so the multiple NICs can be used.

What I haven't seen is any clear concise instructions on how to do this.

Thanks.

[PetrM]

Hi Bill,

The decision about NIC is taken by OS and not by the application. I assume that you can reproduce the same issue even without Veeam, most probably data flow will go through the same NIC if you load network channel by another traffic. Also, would you be so kind to clarify what's working bad now? Are backup jobs slow or do you notice some errors?

Thanks!

[TheGrove]

We are trying to get our active full backups to run faster. Originally we only had 1 NIC active. While monitoring the server while the full backups were running the network throughput maxed out at 1Gbps or the speed of the single NIC. We noticed not only did the backups compete with each other for throughput and when the SOAR offload ran it just made it worst. All the backups were reporting the network as the bottleneck.

So knowing we had 4 NICs in the server we activated the other 3 and teamed them. This helped the offload as the system balances the throughput across the NICs so I now max out my internet connection. But the data coming into the server is still bottle-necked thru the single NIC.

So what I want to do is get my inbound traffic to use multiple NICs so there is less competition between the backup jobs for network resources.

I know Veeam relies on the underlying OS but I have read threads on this forum where others have configured their systems and assigned each repository its own network/NIC. In this setup when the backup jobs run each one will use the NIC configured for the repository desired. This should allow me to have multiple NICs in use increasing my network bandwidth.

What I'm not sure of is everywhere I need to configure to get this to happen. Let me know what additional information I can provide.

[PetrM]

Hi Bill,

I appreciate the detailed explanation. Do you see this problem only when Veeam jobs are running or with any inbound traffic that comes to the server?

Thanks!

[TheGrove]

The only thing that puts a load on the server is the backups. This is a standalone server that controls the jobs, hosts the SQL database and hosts the repositories. I have 4 other servers that are the my VMWare proxy servers split between my 2 VMWare clusters. So unless a backup or SOBR upload is running there is no load in or out.

[PetrM]

Hi Bill,

Do you have a chance to perform some kind of stress testing and generate some random traffic? Basically, an interesting overview of different tools is presented on this page but I guess even iPerf would be fine.

Thanks!

[EdgarRicharte]

Look into setting up multipathing. We have 2x10GB NIC and 2x1GB NIC. Backups for our cloud connect server are spread amongst 4 NIC's. One of the 10GB NIC is prioritized. But the other NIC's are available on the OS level. So our synology shows multiple connections from the same server. It seemed to have lowered our overall backup times.

[TheGrove]

Hi Bill,

Do you have a chance to perform some kind of stress testing and generate some random traffic? Basically, an interesting overview of different tools is presented on this page but I guess even iPerf would be fine.

Thanks!

I could but I'm not following the purpose. It balances fine outbound but there is no balancing inbound. The all NICs in the team respond with the same MAC address so the switch will route all traffic to a single NIC.

I believe i need to break the team and manually force traffic to a particular NIC. My understanding is the only way to do this is to assign a network to a repository then direct a backup to use that repository/NIC combination.

[TheGrove]

Look into setting up multipathing. We have 2x10GB NIC and 2x1GB NIC. Backups for our cloud connect server are spread amongst 4 NIC's. One of the 10GB NIC is prioritized. But the other NIC's are available on the OS level. So our synology shows multiple connections from the same server. It seemed to have lowered our overall backup times.

How would you suggest doing this?

[einhirn]

Hi TheGroove,

[...] It balances fine outbound but there is no balancing inbound. The all NICs in the team respond with the same MAC address so the switch will route all traffic to a single NIC. [...]

ethernet doesn't support "out of order" packet delivery (unlike TCP), so operating sytems and switches usually stick to using hashes of either MAC, IP or Ports or some combination thereof to prevent some packets of a connection going down a different link than others and getting delivered out of order (due do different loading or lengths or speeds of the links etc.).

Perhaps your switch supports using IP and/or Port hashes for traffic distribution, then you could just add some IP adresses (up to 4 in total, I guess) and distribute traffic that way. Or maybe it even supports a round-robin distribution, though unlikely.

All that trouble is the reason for iSCSI-SAN not using link aggregation but multipathing instead.

bye,
Einhirn

[cblais]

This to me sounds like a possible case for switch aware teaming on the hosts, which would allow for inbound and outbound to have full use of the redundant links. That would make for some downtime and extra complexity.

Before going down that route, have you configured direct storage access from your veeam server to the production storage? I've found that is immensely faster and much less complex to setup. Plus it has the bonus of shifting much of the workload to the storage NICs which are typically much more capable.

[wfturner_merit]

Network aggregation (i.e. channel bonding, LACP, etc) generally can provide load balancing (NLB) using hash algorithms. These are generally computed based on either layer 2 (MAC src-dst) or layer 3 (IP/TCP src-dst and sometimes port). The server and the switch have to be in agreement for such load balancing hashes to work (otherwise packets end up on "unexpected" interfaces). Not all network aggregation configurations support them either.

However, even if your setup is able to NLB, individual TCP streams most always stick to a single source & destination interface (they can't span multiple, due the risks of out-of-order packets and other potential problems). If the hash algo was using src-dst-port, each backup job *should* be able to end up on a different destination interface, but based on your observations I would wager that's not happening. I don't personally have much experience with Windows NLB (we're mostly a Linux shop) but it looks like the following powershell cmdlet might provide some information about the NLB:
Get-NetLbfoTeam

Some additional information is here: https://docs.microsoft.com/en-us/powers ... ver2019-ps

Based on that document, you may want to (in a maintenance window) try configuring the following values for the team:
-LoadBalancingAlgorithm Dynamic -TeamingMode SwitchIndependent

That should (based on my reading) do something similar to the Linux Adaptive LoadBalancing (ALB) system. In this mode, the server doesn't work with the switch, but instead gives each team member an independent virtual MAC. It then uses the hash algo (LoadBalancingAlgorithm in the powershell command) to "force" certain traffic to certain NICs.

Again, I've not used Windows servers for this, so I wish you the best of luck. If none of the above works for you, I would mirror what others have suggested above and configured multiple "backup repositories" on the server (bound to different IP addresses for each), then space out your jobs to use them, without the NLB. This would be annoying from a management perspective, but should accomplish the goal. The different backup repos could all be different folders on the same mount as well (we do this to provide shared storage for multiple backup managers).

Either way, good luck!

Edit to add: I would say the easiest way to do multiple backups repositories (not NLB) on a single server would be:
1) Give however many NICs you want to use different IP addresses
2) Make sure your share is listening on the new IPs (service/host restart usually is the easiest way, may be automatic with Windows)
3) In Veeam, add the each IP address as a new backup repository (instead of using the DNS name. You could also give the IPs each a DNS entry)
4) Now you have "different" backup repos that are backed by a single server, assign/balance jobs as needed

[Grime121]

We use a separate vLAN for our backup traffic. To accomplish this we assigned vmkernel IPs on the backup vLAN to all of our VM Hosts. The Veeam proxies are all on the backup vLAN, too. We then just added host entries on all of the Veeam proxies for all of those VM Host backup vLAN IPs. That way, the backup traffic between the hosts and the Veeam proxies is all kept on the backup vLAN. Guest indexing, etc. is still done across vLANs, as the VMs themselves do not have NICs on the backup vLAN, but that’s minimal traffic.

It sped things up for us quite a bit because cross-vLAN traffic has to go through the firewalls, which currently only have 1Gbit interfaces (hoping to upgrade to 10Gbit interfaces during the next firewall refresh).

Oh, our VBR and other repositories are all located on the backup vLAN, too.

[TheGrove]

This to me sounds like a possible case for switch aware teaming on the hosts, which would allow for inbound and outbound to have full use of the redundant links. That would make for some downtime and extra complexity.

Before going down that route, have you configured direct storage access from your veeam server to the production storage? I've found that is immensely faster and much less complex to setup. Plus it has the bonus of shifting much of the workload to the storage NICs which are typically much more capable.

Unfortunately the backup server doesn't have access directly to the SAN.

[TheGrove]

Network aggregation (i.e. channel bonding, LACP, etc) generally can provide load balancing (NLB) using hash algorithms. These are generally computed based on either layer 2 (MAC src-dst) or layer 3 (IP/TCP src-dst and sometimes port). The server and the switch have to be in agreement for such load balancing hashes to work (otherwise packets end up on "unexpected" interfaces). Not all network aggregation configurations support them either.

However, even if your setup is able to NLB, individual TCP streams most always stick to a single source & destination interface (they can't span multiple, due the risks of out-of-order packets and other potential problems). If the hash algo was using src-dst-port, each backup job *should* be able to end up on a different destination interface, but based on your observations I would wager that's not happening. I don't personally have much experience with Windows NLB (we're mostly a Linux shop) but it looks like the following powershell cmdlet might provide some information about the NLB:
Get-NetLbfoTeam

Some additional information is here: https://docs.microsoft.com/en-us/powers ... ver2019-ps

Based on that document, you may want to (in a maintenance window) try configuring the following values for the team:
-LoadBalancingAlgorithm Dynamic -TeamingMode SwitchIndependent

That should (based on my reading) do something similar to the Linux Adaptive LoadBalancing (ALB) system. In this mode, the server doesn't work with the switch, but instead gives each team member an independent virtual MAC. It then uses the hash algo (LoadBalancingAlgorithm in the powershell command) to "force" certain traffic to certain NICs.

Thanks, this is how it is configured. Balancing works great outbound, just not so good inbound.

Again, I've not used Windows servers for this, so I wish you the best of luck. If none of the above works for you, I would mirror what others have suggested above and configured multiple "backup repositories" on the server (bound to different IP addresses for each), then space out your jobs to use them, without the NLB. This would be annoying from a management perspective, but should accomplish the goal. The different backup repos could all be different folders on the same mount as well (we do this to provide shared storage for multiple backup managers).

This is what i would like to try. But i haven't found any clear concise step-by-step instructions on how to implement this.

Either way, good luck!

Edit to add: I would say the easiest way to do multiple backups repositories (not NLB) on a single server would be:
1) Give however many NICs you want to use different IP addresses
2) Make sure your share is listening on the new IPs (service/host restart usually is the easiest way, may be automatic with Windows)
3) In Veeam, add the each IP address as a new backup repository (instead of using the DNS name. You could also give the IPs each a DNS entry)
4) Now you have "different" backup repos that are backed by a single server, assign/balance jobs as needed

Item 3 is the tricky part that i need to get a handle on.

Thanks for the answer.