Dual 10Gbe nics, slow NBD traffic

[itdirector]

We currently run ESXi 5.0 vSphere Essentials & Veeam 7.x essentials. 3 Dell R730 servers. All local DAS, with 7TB of Intel S3500 SSD in each. Dual 10Gbe nics on each server, all connected to a Dell 10Gbe switch. No SAN, no NAS, no vMotion. Simple DAS on each server.

Veeam full backups & full replications between all hosts using hotadd is very fast (proxies on each host): averaging 850MB/s+, almost maxing out the 10Gbe switch. Copying files between two Win2k12R2 Virtual machines on different R730 server hosts is similarly fast : averaging 700MB/s+; so we know the SSD's are not a bottleneck. Again, that is MB, not Mb, & that is between hosts.

However, when using Veeam backup's or replication with NBD on the Proxies instead of Hotadd, speeds drop significantly to around 100MB/s; almost as if it is limiting itself. I am assuming when using NBD, the proxies communicate with the host via the esxi "management network"; so given that when we migrate a VM between hosts, we get the same slow speeds (100MB/s), the management network seems to be the bottle neck.

What doesn't make sense is that we only have one vSwitch configured on each host, with dual 10Gbe nics on the vSwitch. So the proxies above, all of the VM's & the the Management network are all on the same vSwitch, on the same 10Gbe physical switch, using the same 10Gbe nics.

I wanted to see if anyone else has a similar setup & can point me in the right direction to removing the management network bottle neck in our environment so we can switch from hotadd to NBD with veeam, as well as faster migration between hosts with vCenter.

Thank you.

[VladV]

We have seen the same behavior. Every disk processed gets limited to 100 MB/s so when processing multiple disks/jobs in parallel you can reach higher speeds. We managed to get speeds around 300MB/s but when processing a VM with a single large disk we are limited to 100MB/s.

Tried many things but we couldn't find the cause. So we switched to hot add for large VMS that aren't processed in parallel.

I would like to know what is causing this also.

[itdirector]

Anyone else with this setup that can shed some light & point Vlad & I in the right direction?

[id.elizarov]

I guess, limitations in NFC/NBD, such as "...the sum of all NFC connection buffers to an ESXi host cannot exceed 32MB" can cause this behaviour.

https://pubs.vmware.com/vsphere-50/inde ... t.5.5.html

[dellock6]

I'm not sure about this setting being the limit for NFC/Network Mode transfer limits, otherwise it would be hard to explain with Network Mode over 10Gb is so much faster than 1Gb, and definitely can go way above the limit of 1Gbs with a 10Gb cards.