Host-based backup of VMware vSphere VMs.
Post Reply
mephisto
Expert
Posts: 121
Liked: 7 times
Joined: Nov 07, 2012 6:49 pm
Full Name: Mephisto poa
Contact:

network bottleneck, 2~3gbit traffic on 10gbit network

Post by mephisto » 1 person likes this post

Hi guys,

I'm trying to find out the performance bottleneck here with a vcenter 6.7 and hosts on ESXi 6. I've got the backup server using a single 10Gbit nic that is also used for iscsi and network traffic so I'm trying to understand if VMware doesn't allow the traffic to go over 2-3gbit becuase I have iscsi on the same nic for example.

My storage can do 600mb/sec sequential writes, and I have proxy server running on each host with a fair amount of CPU to be able to hotadd disks (reading data from SSDs) and send the traffic compressed I believe over the network to the backup repository where Veeam also runs.

I get a very flat network traffic rate at 200MB/sec, this when backing up stuff from several places at the same time using different proxies, so it becomes evident the network or the backup server itself is the bottleneck. Accoording to Veeam B&R report after the backup finishes, it is a network bottleneck.

Is there anything I could do in this situation to get this running a bit faster? It seems I have everything to make it runs faster, but it doesn't due to some software limit somewhere?
mephisto
Expert
Posts: 121
Liked: 7 times
Joined: Nov 07, 2012 6:49 pm
Full Name: Mephisto poa
Contact:

Re: network bottleneck, 2~3gbit traffic on 10gbit network

Post by mephisto »

image of resource monitor and the backup job itself

https://imgur.com/a/A96CkX7
nitramd
Veteran
Posts: 298
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: network bottleneck, 2~3gbit traffic on 10gbit network

Post by nitramd »

Do you have Throttling enabled?
csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: network bottleneck, 2~3gbit traffic on 10gbit network

Post by csydas »

Hey Mephisto,

Simple test is just iPerf between the two servers; do you see the same speeds there? Since it's a VM, I'd suspect that it's the VMware network stack between the VMs causing you grief. Windows Copy won't be reliable here due to the mischief that Windows and the VMware Tools will do to improve speeds.
jdavidson_waters
Service Provider
Posts: 43
Liked: 15 times
Joined: May 07, 2013 2:50 pm
Full Name: James Davidson
Location: Northeast UK
Contact:

Re: network bottleneck, 2~3gbit traffic on 10gbit network

Post by jdavidson_waters » 1 person likes this post

Sounds like you're using NBD, rather than hot-add or direct SAN.
NBD limited to around 300MB/s when using 10Gb.

More info here:
https://forums.veeam.com/vmware-vsphere ... 39238.html
And
https://kb.vmware.com/s/article/2147768
@jam_davidson
mephisto
Expert
Posts: 121
Liked: 7 times
Joined: Nov 07, 2012 6:49 pm
Full Name: Mephisto poa
Contact:

Re: network bottleneck, 2~3gbit traffic on 10gbit network

Post by mephisto »

nitramd wrote:Do you have Throttling enabled?
no, I've checked, no throttling enabled
csydas wrote:Hey Mephisto,

Simple test is just iPerf between the two servers; do you see the same speeds there? Since it's a VM, I'd suspect that it's the VMware network stack between the VMs causing you grief. Windows Copy won't be reliable here due to the mischief that Windows and the VMware Tools will do to improve speeds.
ipperf between the veeam server and a VM for example?
jdavidson_waters wrote:Sounds like you're using NBD, rather than hot-add or direct SAN.
NBD limited to around 300MB/s when using 10Gb.

More info here:
https://forums.veeam.com/vmware-vsphere ... 39238.html
And
https://kb.vmware.com/s/article/2147768
It says it is hot adding on each proxy server, it is going way faster than before with the veeam server being the proxy alone using NBD. It seems it is using hotadd across all backup jobs. Any suggestions where else I can look?
nitramd
Veteran
Posts: 298
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: network bottleneck, 2~3gbit traffic on 10gbit network

Post by nitramd »

For your repository in Veeam B&R, what number is the "Limit maximum concurrent tasks" set to?
Post Reply

Who is online

Users browsing this forum: No registered users and 53 guests