Slow Backupspeed although 10Gbit network

[mcdaniels]

Hi,
I have got very slow backupspeed of max. 50MB/sec, running on a 10gbit network.
This only happens in VEEAM. When I copy a 2.2GB File from the VM to the physical backupserver 1 get a datarate of 1,2GB/sec.

The managment Network (which is used for veeambackups, if I am right) of the VMWARE ESXI 6.5 is connected to a 10Gbit Intel-nic, the backupserver uses the same Nic.

If I show at the veeam statistics, the bottleneck is the source:

Source: 99%
Proxy: 12%
Network: 2%
Target: 0%

The Esxi is a Dell Poweredge T640, Raid 10, 8HDDs, 10k, SAS.
The Backupserver is a Dell Poweredge T440, Raid 10, 6 x 4TB, SAS 7.2k

ESXI Version: 6.5 (all critical Patches applied)
Veeam Version: 9.5.0/1922

Veeam proxysetting: Transportmode: Network (automatic does no change to speed).

[bdufour]

https://bp.veeam.expert/proxy_servers_i ... twork_mode

Cons
Typically, network mode uses only up to 40% of the available bandwidth of the external VMKernel interface due to throttling mechanisms implemented on the management interfaces.

[nitramd]

How many physical CPUs and how much memory does your backup server have? Veeam uses 1 physical core per disk and about 4GB of RAM per disk.

Is your proxy also located on the backup server?

[nitramd]

Have a look at these proxy specs: https://helpcenter.veeam.com/docs/backu ... r=95#proxy

A proxy uses 2GB memory not 4 -> typo on my part

[mcdaniels]

Hi,
yes proxy = backupserver
16GB of RAM
It is 1x Intel Xeon Bronze 3104, 1.7Ghz, 6-Core

I do no concurrent tasks, I do vm-backup one after one.

BUT: The bottleneck is source, so hm... it cannot be the backupserver? (as you said it is the backupmode of veeam)

Is there a better solution I can use / not using the management-network for backups?

So there is no chance of high speed backup with veeam without altering my environment?

This is also nice:
Generally speaking, network mode is recommended with 10 GB Ethernet.
https://www.veeam.com/blog/vmware-backu ... ation.html

Really, at that speed?!

And in another thread a veeam supporter says: If you have 10GbE for the ESXi mangement network, network mode can easily transfer 300-400MB/s per host.

Hm I am getting only the mentioned max. 50MB/sec

[wishr]

Hi gentlemen,

I would recommend taking a look at that thread explaining what "Source" bottleneck means.

Thank you

[foggy]

Have you tried any other transport mode (hotadd, direct SAN)?

[mcdaniels]

@wishr thank you
@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned)

If I choose to set up a veeam proxy in a VM, do I have to buy additional licenses? I have licensed veeam backup enterprise+ for 4 sockets.

[wishr]

Hi Mcdaniels,

No, you do not have to additionally license any of the B&R components in this situation. The license simply controls the size of the virtual environment that you'd like to back up.

BR,
Fedor

[foggy]

@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned)

Correct, but on 10GB network mode should be faster than you're seeing. Trying some other transport will tell if it is the storage that cannot perform faster or environmental (networking) issue.

[mcdaniels]

@foggy: I will try. Just doublechecked that the management-network of my esxi is running @ 10GBIT Nic -> confirmed. But I have configured a 10Gbit standby adapter, which - in my opinion - should not interfere.
management network setting is looking like (german):



Switch is 10Gbit.

Here you see the harddisc-performance of the backupserver: https://vmware-forum.de/download/file.php?id=1184&t=1
Here you see the harddisc-performance of the esxiserver: https://vmware-forum.de/download/file.php?id=1185&t=1

[nitramd]

@mcdaniels, have you considered running iPerf to test the 10Gb network?

[dimaslan]

Do you perhaps setup any throttling policies? If it was the network that was causing this, it should say so in the Bottleneck description, whereas now it says "Source". Apparently since a simple copy is allowing for the large speeds you expect to get, it should be something else causing this. The backup and the manual copy, were they performed from the same datastore?

[mcdaniels]

Hello,
sorry for coming back to you that late, but I had a lot of other urgent things to do (including illness). The case is not solved yet.

@mcdaniels, have you considered running iPerf to test the 10Gb network?

have to look into this. Have no clue how it works (yet).

@dimaslan: I have not throtteling policies set up.

Yes backup and copy were performed from the same datastore.

Just running a backup-copy-job, which also says that the source is the bottleneck:

Source 76%
Proxy 17%
Network 22%
Destination 12%

Transfer rate about 120Mb/sec

[mcdaniels]

so, just managed to do iperf tests:

ESXI = Client:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec  8626536   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   2.00-3.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   4.00-5.00   sec   395 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec  4286340760   0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
CPU Utilization: local/sender 50.2% (50.3%u/0.0%s), remote/receiver 0.7% (0.1%u/0.7%s)

Backupserver (Windows) = Server:

Code: Select all

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.226, port 16240
[  5] local 192.168.10.219 port 5201 connected to 192.168.10.226 port 29629
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec
[  5]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec
[  5]   2.00-3.00   sec   394 MBytes  3.30 Gbits/sec
[  5]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec
[  5]   4.00-5.00   sec   395 MBytes  3.31 Gbits/sec
[  5]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec
[  5]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec
[  5]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec
[  5]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec
[  5]  10.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
-----------------------------------------------------------

And the other way around:

ESXI Server:

Code: Select all

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.219, port 57808
[  5] local 192.168.10.226 port 5201 connected to 192.168.10.219 port 57809
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   176 MBytes  1.47 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   3.00-4.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]  10.00-10.05  sec  10.9 MBytes  1.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.05  sec  2.20 GBytes  1.88 Gbits/sec                  receiver

Backup (Windows) Client:

Code: Select all

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   184 MBytes  1.55 Gbits/sec
[  4]   1.00-2.00   sec   194 MBytes  1.63 Gbits/sec
[  4]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   3.00-4.00   sec   234 MBytes  1.97 Gbits/sec
[  4]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver
CPU Utilization: local/sender 15.2% (4.1%u/11.1%s), remote/receiver 4.6% (4.6%u/0.0%s)

Seems pretty fast?

[Gostev]

The issue here is either with primary storage array or ESXi management stack (if you're using NBD). The latter can be excluded by trying hot add transport mode instead.

By the way, I noticed you still didn't include a support case ID for this issue. This is against forum rules provided when you click New Topic. Please include the support case ID if you'd like to keep the topic open.

Thanks!

[mcdaniels]

Support Case id: #03912843


Thanks!