network bottleneck, 2~3gbit traffic on 10gbit network

[mephisto]

Hi guys,

I'm trying to find out the performance bottleneck here with a vcenter 6.7 and hosts on ESXi 6. I've got the backup server using a single 10Gbit nic that is also used for iscsi and network traffic so I'm trying to understand if VMware doesn't allow the traffic to go over 2-3gbit becuase I have iscsi on the same nic for example.

My storage can do 600mb/sec sequential writes, and I have proxy server running on each host with a fair amount of CPU to be able to hotadd disks (reading data from SSDs) and send the traffic compressed I believe over the network to the backup repository where Veeam also runs.

I get a very flat network traffic rate at 200MB/sec, this when backing up stuff from several places at the same time using different proxies, so it becomes evident the network or the backup server itself is the bottleneck. Accoording to Veeam B&R report after the backup finishes, it is a network bottleneck.

Is there anything I could do in this situation to get this running a bit faster? It seems I have everything to make it runs faster, but it doesn't due to some software limit somewhere?

[mephisto]

image of resource monitor and the backup job itself

https://imgur.com/a/A96CkX7

[nitramd]

Do you have Throttling enabled?

[csydas]

Hey Mephisto,

Simple test is just iPerf between the two servers; do you see the same speeds there? Since it's a VM, I'd suspect that it's the VMware network stack between the VMs causing you grief. Windows Copy won't be reliable here due to the mischief that Windows and the VMware Tools will do to improve speeds.

[jdavidson_waters]

Sounds like you're using NBD, rather than hot-add or direct SAN.
NBD limited to around 300MB/s when using 10Gb.

More info here:
https://forums.veeam.com/vmware-vsphere ... 39238.html
And
https://kb.vmware.com/s/article/2147768

[mephisto]

Do you have Throttling enabled?

no, I've checked, no throttling enabled

Hey Mephisto,

Simple test is just iPerf between the two servers; do you see the same speeds there? Since it's a VM, I'd suspect that it's the VMware network stack between the VMs causing you grief. Windows Copy won't be reliable here due to the mischief that Windows and the VMware Tools will do to improve speeds.

ipperf between the veeam server and a VM for example?

Sounds like you're using NBD, rather than hot-add or direct SAN.
NBD limited to around 300MB/s when using 10Gb.

More info here:
https://forums.veeam.com/vmware-vsphere ... 39238.html
And
https://kb.vmware.com/s/article/2147768

It says it is hot adding on each proxy server, it is going way faster than before with the veeam server being the proxy alone using NBD. It seems it is using hotadd across all backup jobs. Any suggestions where else I can look?

[nitramd]

For your repository in Veeam B&R, what number is the "Limit maximum concurrent tasks" set to?