Slow backup/replication without proxy

[Soft-Master]

I have few Vsphere Vmware servers in my test lab with Infiniband 56Gb/sec interconnection. I performed iperf tests and was able to reach about 40-45 Gb/sec real network traffic between ESXi hosts.
Each ESXi host has local RAID storage and tests shows about 4 Gbyte/sec (32Gb/sec) sequential read/write speed on this storage.
Veeam installed on Windows 2019 virtual server with VMXnet3 10GB NIC. Iperf tests to analogical server on another host shows real 10GB/sec transfer speed.
But when I am performing quick migration or backup job by Veeam I see 100-120MB/sec speed with source bottleneck. Looks like network traffic for one VM transfer limited somewhere around 1GB/sec.
I added few virtual machines to the job and was able to reach 300-400 MB/sec (around 2-3 GB/sec) that much better but still far from real hardware performance. That means that each VM process still limited by this 1GB/sec, but since 4 processes were running simultaneously, the total speed was higher..
What limits this speed? Why single task limited by this low speed? How to change this limitation?
Thanks

[wishr]

Hi Evgeniy,

Source bottleneck means the data mover residing on the source disk spent a lot of time pulling the data from the disk. Please check this sticky thread for more info.

Thanks

[Soft-Master]

Hi
But if I run 4 VMs simultaneously from same host in one job, I get x4 speed summary, which means that there are no speed problems on the hardware side, and the restriction is per VM task is somewhere at the program level. And where is it?
Thanks

[wishr]

What are the bottleneck stats with four VMs being added to the job? Also, what is your backup proxy configuration in regards to hardware, mode, and placement?

[Soft-Master]

Backup proxy is main Veeam server, no additional proxy. Transport mode is network.
Bottleneck is source.
Please explain what you mean "backup proxy configuration in regards to hardware, mode, and placement"

[foggy]

Hi Evgeniy, seems you're using NBD mode, please review this thread for the heads up. Thanks!

[Soft-Master]

VM Host1: R730XD, 26xSAS drives RAID6, Mellanox Connectx3 56GB/s LAN. Veeam installed. HDD speed test on local Veeam disk - 4GB read and 3GB write for sequential access, which basically corresponds to backup load.
VM Host2: Same as host1. R730XD, 26xSAS drives RAID6, Mellanox Connectx3 56GB/s LAN. Few test VM installed.
Network speed test result with iperf between host1 and host2 - up to 40Gb/s
Network speed test result between Veeam server and another windows based server on host 2 is 10Gb/s (limited by VMxnet3 10GB speed)
Backup job with 1 VM - 112MB/s speed
Backup job with 4 VM - 420MB/s speed that means practically x4 scaling, which says that the speed limit is per one job...

[Soft-Master]

Yep, before I was trying to use NBD mode.
Now I installed proxy servers on both hosts and got average 549 MB/sec (with about 800MB/sec in peaks) for one VM backup with bottleneck in proxy . That means that communication via NBD between Veeam server and ESXi host is much slower than communication between Veeam server and Veeam proxy. And has some limitation per stream (as I found on previous tests). But why? And even 549 MB/sec (4.54Gb/sec) still less than available hardware performance...

[Soft-Master]

I found similar discussion in the past
vmware-vsphere-f24/slow-nbd-backup-over ... 39238.html
No solution, just "This is a limitation of VMware... nothing we can do about at NBD at the moment"
But it was 9 years ago... Still no solution?

[foggy]

The thread I've referenced above is a more recent one - yes, this is still the case.

[Soft-Master]

Ok, Got it. Thanks for quick response. So, NBD mode is very slow by VMware design and to take advantage of 56gb network in the absence of shared storage need to install Veeam proxy on each host and use hot-adding mode.
Some question related to it. The network speed between Veeam proxy servers is limited by NIC speed, with VMxnet3 this is 10GB/s. Is it possible to aggregate few NIC's to speed up this transfer between proxies? Microsoft NIC team is not the solution, because it does not provide real bandwidth aggregation in our case (one-to-one connection). It must be supported by Veeam via separating traffic to few streams and distribution these streams between few available NIC's. Does Veeam provide this functionality?
Thanks

[Soft-Master]

Hi
Any information?
Thanks

[DGrinev]

Theoretically, backup\replication jobs can benefit from the nic team, LACP, bonding since VBR is using multiple streams to transfer the data.

You said the bottleneck was Proxy, which means it consumed the most time during the job processing on the deduplication and compression.
Can you try to add more resources for the backup proxies and do the test again?

Please review the bottleneck analysis in the sticky thread here. Thanks!