Discussions specific to the VMware vSphere hypervisor
Post Reply
pmaa
Novice
Posts: 7
Liked: never
Joined: Jul 24, 2017 7:44 am
Contact:

Network mode performance on vSphere 6.5

Post by pmaa »

I've learned that vSphere 6.5 now enforces NBDSSL for network mode. From what I've read, on previous versions you could expect at least about 25 MB/s in network mode on gigabit ethernet and maybe more with tuning.

I'm getting 5 MB/s with vSphere 6.5. Is this the new normal or should I expect a bit more?

DGrinev
Expert
Posts: 1943
Liked: 248 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Network mode performance on vSphere 6.5

Post by DGrinev »

Hi pmaa and welcome to the community!

Can you share the bottleneck stats of the job, that should help to understand the cause of the slow performance. Thanks!

pmaa
Novice
Posts: 7
Liked: never
Joined: Jul 24, 2017 7:44 am
Contact:

Re: Network mode performance on vSphere 6.5

Post by pmaa »

Hi DGrinev,

Bottleneck is source 99% and also target is 98%. Proxy and network are very low with only a few %.

I tested setting up the VMwareNBDCompressionLevel registry key with value 3 and the processing rate went up to 6 MB/s with peaks of 9 and 11 MB/s. Now the bottleneck is target 99%, source 34%, proxy 11% and network 4%.

Edit: This is a replication job between local hosts.

pmaa
Novice
Posts: 7
Liked: never
Joined: Jul 24, 2017 7:44 am
Contact:

Re: Network mode performance on vSphere 6.5

Post by pmaa »

Another replication job using VMwareNBDCompressionLevel of 3 is now running with processing rate of 7 MB/s. Bottleneck stats are: Source 7%, Proxy 9%, Network 0%, Target 49%. So with this one it definitely seems that the infrastructure is keeping up easily but perhaps NBDSSL is the bottleneck.

DGrinev
Expert
Posts: 1943
Liked: 248 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Network mode performance on vSphere 6.5

Post by DGrinev »

Hi,

The reduction shouldn't be significant about 10% - 20% than ordinary NBD.
Overall processing rate depends on different factors such as read or write speed, configuration of NICs and so on, ideally you can expect about 25 MB/s in network mode.

During the last run the bottleneck was Target, which means 49% of the time the job spent writing data to the target storage.
Please review the bottleneck analysis post for additional information. Thanks!

pmaa
Novice
Posts: 7
Liked: never
Joined: Jul 24, 2017 7:44 am
Contact:

Re: Network mode performance on vSphere 6.5

Post by pmaa »

Ok, I'll open a support case, although this VMware KB isn't too encouraging: https://kb.vmware.com/kb/2147768

pmaa
Novice
Posts: 7
Liked: never
Joined: Jul 24, 2017 7:44 am
Contact:

Re: Network mode performance on vSphere 6.5

Post by pmaa »

Support determined that it's indeed a limitation of NBDSSL in vSphere 6.5. So 5 MB/s is currently to be expected with gigabit ethernet.

DGrinev
Expert
Posts: 1943
Liked: 248 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Network mode performance on vSphere 6.5

Post by DGrinev »

In case you are replicating between two local hosts, why don't you consider to use Hotadd transport mode? Thanks!

pmaa
Novice
Posts: 7
Liked: never
Joined: Jul 24, 2017 7:44 am
Contact:

Re: Network mode performance on vSphere 6.5

Post by pmaa »

For the moment we are replicating between local hosts but will move to offsite replication later. Hot-add works with offsite if there is backup proxy on both ends right? Currently we don't have enough Windows VM's to do Hot-add, so unfortunately we have to get Windows Server licenses just for that.

DGrinev
Expert
Posts: 1943
Liked: 248 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Network mode performance on vSphere 6.5

Post by DGrinev »

pmaa wrote:Hot-add works with offsite if there is backup proxy on both ends right?
Yes, the proxy role should be assigned to a VMs on both sites. Thanks!

DonZoomik
Expert
Posts: 226
Liked: 57 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Network mode performance on vSphere 6.5

Post by DonZoomik »

How about a Veeam hotfix to include updated VDDK. It seems that this was fixed in April: http://pubs.vmware.com/Release_Notes/en ... notes.html
6.5U1 would be nice but 6.5b has been out for months.

foggy
Veeam Software
Posts: 19475
Liked: 1768 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Network mode performance on vSphere 6.5

Post by foggy »

pmaa wrote:Currently we don't have enough Windows VM's to do Hot-add, so unfortunately we have to get Windows Server licenses just for that.
You can use any existing VM as hotadd proxy, no need to have a dedicated one.

foggy
Veeam Software
Posts: 19475
Liked: 1768 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Network mode performance on vSphere 6.5

Post by foggy »

DonZoomik wrote:How about a Veeam hotfix to include updated VDDK.
You're right that switch to NBDSSL was fixed in a later VDDK version. Chances are it will be included in v9.5 Update 3.

tmtech
Lurker
Posts: 2
Liked: never
Joined: Aug 01, 2017 1:15 pm
Full Name: Tmtech
Contact:

Re: Network mode performance on vSphere 6.5

Post by tmtech »

I'm running vbr 9.5 u2 on server 2016 datacenter with one proxy, VMware esxi 6. After lots of troubleshooting final verdict was to remove cdrom and sata controller from vbr vm's.

Post Reply

Who is online

Users browsing this forum: No registered users and 14 guests