Slow Backupspeed although 10Gbit network

[mcdaniels]

Hi,
I have got very slow backupspeed of max. 50MB/sec, running on a 10gbit network.
This only happens in VEEAM. When I copy a 2.2GB File from the VM to the physical backupserver 1 get a datarate of 1,2GB/sec.

The managment Network (which is used for veeambackups, if I am right) of the VMWARE ESXI 6.5 is connected to a 10Gbit Intel-nic, the backupserver uses the same Nic.

If I show at the veeam statistics, the bottleneck is the source:

Source: 99%
Proxy: 12%
Network: 2%
Target: 0%

The Esxi is a Dell Poweredge T640, Raid 10, 8HDDs, 10k, SAS.
The Backupserver is a Dell Poweredge T440, Raid 10, 6 x 4TB, SAS 7.2k

ESXI Version: 6.5 (all critical Patches applied)
Veeam Version: 9.5.0/1922

Veeam proxysetting: Transportmode: Network (automatic does no change to speed).

[bdufour]

https://bp.veeam.expert/proxy_servers_i ... twork_mode

Cons
Typically, network mode uses only up to 40% of the available bandwidth of the external VMKernel interface due to throttling mechanisms implemented on the management interfaces.

[nitramd]

How many physical CPUs and how much memory does your backup server have? Veeam uses 1 physical core per disk and about 4GB of RAM per disk.

Is your proxy also located on the backup server?

[nitramd]

Have a look at these proxy specs: https://helpcenter.veeam.com/docs/backu ... r=95#proxy

A proxy uses 2GB memory not 4 -> typo on my part

[mcdaniels]

Hi,
yes proxy = backupserver
16GB of RAM
It is 1x Intel Xeon Bronze 3104, 1.7Ghz, 6-Core

I do no concurrent tasks, I do vm-backup one after one.

BUT: The bottleneck is source, so hm... it cannot be the backupserver? (as you said it is the backupmode of veeam)

Is there a better solution I can use / not using the management-network for backups?

So there is no chance of high speed backup with veeam without altering my environment?

This is also nice:
Generally speaking, network mode is recommended with 10 GB Ethernet.
https://www.veeam.com/blog/vmware-backu ... ation.html

Really, at that speed?!

And in another thread a veeam supporter says: If you have 10GbE for the ESXi mangement network, network mode can easily transfer 300-400MB/s per host.

Hm I am getting only the mentioned max. 50MB/sec

[wishr]

Hi gentlemen,

I would recommend taking a look at that thread explaining what "Source" bottleneck means.

Thank you

[foggy]

Have you tried any other transport mode (hotadd, direct SAN)?

[mcdaniels]

@wishr thank you
@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned)

If I choose to set up a veeam proxy in a VM, do I have to buy additional licenses? I have licensed veeam backup enterprise+ for 4 sockets.

[wishr]

Hi Mcdaniels,

No, you do not have to additionally license any of the B&R components in this situation. The license simply controls the size of the virtual environment that you'd like to back up.

BR,
Fedor

[foggy]

@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned)

Correct, but on 10GB network mode should be faster than you're seeing. Trying some other transport will tell if it is the storage that cannot perform faster or environmental (networking) issue.

[mcdaniels]

@foggy: I will try. Just doublechecked that the management-network of my esxi is running @ 10GBIT Nic -> confirmed. But I have configured a 10Gbit standby adapter, which - in my opinion - should not interfere.
management network setting is looking like (german):



Switch is 10Gbit.

Here you see the harddisc-performance of the backupserver: https://vmware-forum.de/download/file.php?id=1184&t=1
Here you see the harddisc-performance of the esxiserver: https://vmware-forum.de/download/file.php?id=1185&t=1

[nitramd]

@mcdaniels, have you considered running iPerf to test the 10Gb network?

[dimaslan]

Do you perhaps setup any throttling policies? If it was the network that was causing this, it should say so in the Bottleneck description, whereas now it says "Source". Apparently since a simple copy is allowing for the large speeds you expect to get, it should be something else causing this. The backup and the manual copy, were they performed from the same datastore?

[mcdaniels]

Hello,
sorry for coming back to you that late, but I had a lot of other urgent things to do (including illness). The case is not solved yet.

@mcdaniels, have you considered running iPerf to test the 10Gb network?

have to look into this. Have no clue how it works (yet).

@dimaslan: I have not throtteling policies set up.

Yes backup and copy were performed from the same datastore.

Just running a backup-copy-job, which also says that the source is the bottleneck:

Source 76%
Proxy 17%
Network 22%
Destination 12%

Transfer rate about 120Mb/sec

[mcdaniels]

so, just managed to do iperf tests:

ESXI = Client:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec  8626536   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   2.00-3.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   4.00-5.00   sec   395 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec  4286340760   0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
CPU Utilization: local/sender 50.2% (50.3%u/0.0%s), remote/receiver 0.7% (0.1%u/0.7%s)

Backupserver (Windows) = Server:

Code: Select all

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.226, port 16240
[  5] local 192.168.10.219 port 5201 connected to 192.168.10.226 port 29629
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec
[  5]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec
[  5]   2.00-3.00   sec   394 MBytes  3.30 Gbits/sec
[  5]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec
[  5]   4.00-5.00   sec   395 MBytes  3.31 Gbits/sec
[  5]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec
[  5]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec
[  5]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec
[  5]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec
[  5]  10.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
-----------------------------------------------------------

And the other way around:

ESXI Server:

Code: Select all

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.219, port 57808
[  5] local 192.168.10.226 port 5201 connected to 192.168.10.219 port 57809
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   176 MBytes  1.47 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   3.00-4.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]  10.00-10.05  sec  10.9 MBytes  1.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.05  sec  2.20 GBytes  1.88 Gbits/sec                  receiver

Backup (Windows) Client:

Code: Select all

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   184 MBytes  1.55 Gbits/sec
[  4]   1.00-2.00   sec   194 MBytes  1.63 Gbits/sec
[  4]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   3.00-4.00   sec   234 MBytes  1.97 Gbits/sec
[  4]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver
CPU Utilization: local/sender 15.2% (4.1%u/11.1%s), remote/receiver 4.6% (4.6%u/0.0%s)

Seems pretty fast?

[Gostev]

The issue here is either with primary storage array or ESXi management stack (if you're using NBD). The latter can be excluded by trying hot add transport mode instead.

By the way, I noticed you still didn't include a support case ID for this issue. This is against forum rules provided when you click New Topic. Please include the support case ID if you'd like to keep the topic open.

Thanks!

[mcdaniels]

Support Case id: #03912843


Thanks!

[mcdaniels]

The issue here is either with primary storage array or ESXi management stack (if you're using NBD). The latter can be excluded by trying hot add transport mode instead.

Hotadd transportmode is only available if the Datastore is attached via NFS or Fibre, am I right? Our datastore is a direct attached one.

[Gostev]

No, hot add is available regardless of storage type. But of course, in case of DAS you need to use a backup proxy VM on the same host.

[mcdaniels]

Ok, different information from different parts of the supportnetwork

are there any docs/howtos on how to set up a backup-proxy-vm?

Is this a separate Veeam-Appliance (a VM?)

Or in other words. At the moment I have a all in one installation. All Veeam Services on one server.

How can I "split" of the backupproxy / install a separate backupproxy?

Which OS is required?
Where do I get the setupfiles for the Veeam-Proxy?

[wishr]

Hi Mcdaniels,

Please take a look at this article. System requirements for backup proxies can be found here. I would also recommend checking this BP portal section for additional information.

Thanks and let us know if you have any questions.

[mcdaniels]

Hi,
in fact I am aware of setting up a backupproxy (we have running all components on one machine -> Windows Server 2016 -- proxy&repo).

I only would like to know:

If I want to use hotadd mode I cannot use our configuration, right? (because the veeamproxy has to run on the Vmware-host you do backups from and has access to the datastore)
Do I have to install a VM on the ESXI and install veeam on this machine to use hotadd mode?
So, I only can use our Backupserver (Windows 2016) as a Backuprepo (Backuptarget), right?

Thanks!

[foggy]

Deploy a VM on the required host and add it to Veeam B&R as a proxy using the procedure referenced by Fedor above.

[mcdaniels]

Hi,
so it will be possible to use an existing server-vm on this host to act as veeam proxy? And I do not have to install components of Veeam B&R on this machine?

[foggy]

Correct. Existing VM can be used as a proxy and all required components will be installed upon adding it to Veeam B&R as a proxy.

[mcdaniels]

Ah, I see. I always thought that you will have to install additional software on that existing VM.
Thank you.

I will give it a try.

[mcdaniels]

So,
back from the test:
Proxy "hot add mode": Srv 2k16, 8GB Ram, 2x 2,1 Ghz Xeon, VMXNET3 - 10GBIT.

After the start of the Backup CPU of the proxy permanently at 100% load.
Throughput Veeam: 250 MB/sec.

Source: 3%
Destination:0%
Bottleneck 99% hot-add-proxy
Network: 96% (which I do not understand, cause it is used at max of 1,5 Gbit/sec and is a 10Gbit connection)

It keeps strange for me.

What do you say about that?

Thanks!

[wishr]

Hello,

Thank you for the info. Could you please make sure that it has not failed over to network mode - this can be checked within B&R UI in the job session log. If that does not point you to the root cause it would be better to continue the conversation you had with our support engineer as additional details will be required.

Thanks

[foggy]

I'd also pay attention to the number of parallel tasks during that job.

[oliverL]

Proxy "hot add mode": Srv 2k16, 8GB Ram, 2x 2,1 Ghz Xeon, VMXNET3 - 10GBIT.

Taking the system requirements for VM Backup Proxies:
https://helpcenter.veeam.com/docs/backu ... 95u4#proxy

and assuming this Backup Proxy isn't used as a repository server, you should at least have 3 vCPUs configured for your Proxy.
Since your Proxy is at 99% CPU load it seems that it can't process enough data to fully saturate the 10Gbit Link, therefore you could try to add more vCPUs.


How many virtual Machines and how many vmdks are to be backed up from the T640?