Network Performance NIC Teaming

[TheGrove]

My environment, I have 5 HP ProLiant BL460c Gen9 servers in a ESXi, 6.5.0 cluster. These servers are hosting my virtual servers. My backup server is a Dell Poweredge R530. The Dell is the repository and management server. In the VMWare environment I have 2 proxy servers running.

When backups run the proxy servers divide the hosts and drives nicely with each proxy server processing several drives each via hot-adding them. The proxy servers are set to do 8 concurrent tasks each.

We have multiple backup jobs configured with multiple servers in each job. All of the jobs write to a single SOBR repository.

Watching the jobs run multiple jobs will run with multiple servers being processed in each job. The proxy's were sending there data thru a single configured NIC on the backup server. This was saturating the NIC so it is the bottleneck.

There were 3 unused NICs in the backup server. I have connected them and configured teaming in Windows (2012r2) across all 4 NICs.

Watching the backups now the teaming doesn't seem to be working. All traffic is still going the original NIC which is NIC 1 in the team.

How can I get the system to spread the proxy tasks across all of the NICs so i can get 4Gbps instead of 1Gbps?

Hopefully this makes sense. Thanks

[soncscy]

Hey Bill,

Remember, load balancing for the NICs happens at the OS level, an application cannot decide which NIC to use since with NIC Teaming, the entire idea is the application sees just one interface, not 4. The Application shouldn't be able to decide which NIC it uses.

What did you set for the teaming settings? Also are you using Per-VM backups or Per-Backup backups? My guess is that Windows' NIC Teaming will look for processes requesting network access and split the load that way, but reading this [0] it looks like windows splits this off to a separate feature. As I get it, Windows will hash on incoming/destination addresses and likely that's why you're not getting the distribution right as you target the repository. I'd review the below document and see which Windows option will split the incoming connections across the NICs, since as I get it with NIC Teaming Windows decides which path to take based on the source/target destination. It really doesn't seem to focus on load balancing but just uptime.

[0] - https://docs.microsoft.com/en-us/window ... ic-teaming

[TheGrove]

Thanks for the reply. I am set for switch independent dynamic balancing. Thanks for the article, one thing that I noticed is an indication that balancing is more a function of outbound traffic than inbound. Also no matter which NIC sends traffic they all use the same MAC address. I suspect this may be at least part of the issue. My thought is since everything is using a single MAC address the inbound traffic is all being directed to that single NIC.

It looks like I may need to try removing the team and distributing my load a different way.

[soncscy]

Sure, happy to help.

Honestly, you might need to find a non-native Windows solution; reading up on the subject matter, Microsoft pushes NLB which infers you have multiple target servers, so seems the built-in stack can't imagine balancing across multiple target NICs to the same target.

With /etc/hosts edits, undoubtedly you can do the same thing but this is undesirable for obvious reasons (management overhead, it's a "silent" edit, etc)

If you are going to go the DNS or /etc/hosts route, then you can hack this with the Application and make each route a repository path, and then Windows should force each connection over the NIC to a different folder. You'll need to "smartly" balance your jobs, but that actually shouldn't be too hard to write up a script that does the math for you and decides which repository is the most viable target based on the current load.

The pseudo code for adding new jobs would look like (this is not exact, I'm doing this from memory as I'm not connected to my VBR environment at the moment)

$Jobs = Get-VBRJob
$RepoUsage = $Jobs.GetRepository() | Group-Object | Sort-Object -Property ($_.Value.Count) #You'll need some filters here
$RepositoryToUse = $RepoUsage[0] # the least used repository should be on top, and if > 1 repository is "least used", then who cares?

Then you know which repository to set when making new jobs; you can further script it to make the job, or just manually make it. If you find yourself making a lot of jobs frequently, I'd just script this. If it's more of a "once in awhile" situation, then I'd use the same to at least know where to point the new job.