Host-based backup of VMware vSphere VMs.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Hi,
I have got very slow backupspeed of max. 50MB/sec, running on a 10gbit network.
This only happens in VEEAM. When I copy a 2.2GB File from the VM to the physical backupserver 1 get a datarate of 1,2GB/sec.

The managment Network (which is used for veeambackups, if I am right) of the VMWARE ESXI 6.5 is connected to a 10Gbit Intel-nic, the backupserver uses the same Nic.

If I show at the veeam statistics, the bottleneck is the source:

Source: 99%
Proxy: 12%
Network: 2%
Target: 0%

The Esxi is a Dell Poweredge T640, Raid 10, 8HDDs, 10k, SAS.
The Backupserver is a Dell Poweredge T440, Raid 10, 6 x 4TB, SAS 7.2k

ESXI Version: 6.5 (all critical Patches applied)
Veeam Version: 9.5.0/1922

Veeam proxysetting: Transportmode: Network (automatic does no change to speed).
bdufour
Expert
Posts: 206
Liked: 41 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by bdufour »

https://bp.veeam.expert/proxy_servers_i ... twork_mode

Cons
Typically, network mode uses only up to 40% of the available bandwidth of the external VMKernel interface due to throttling mechanisms implemented on the management interfaces.
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by nitramd »

How many physical CPUs and how much memory does your backup server have? Veeam uses 1 physical core per disk and about 4GB of RAM per disk.

Is your proxy also located on the backup server?
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by nitramd »

Have a look at these proxy specs: https://helpcenter.veeam.com/docs/backu ... r=95#proxy

A proxy uses 2GB memory not 4 -> typo on my part
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Hi,
yes proxy = backupserver
16GB of RAM
It is 1x Intel Xeon Bronze 3104, 1.7Ghz, 6-Core

I do no concurrent tasks, I do vm-backup one after one.

BUT: The bottleneck is source, so hm... it cannot be the backupserver? (as you said it is the backupmode of veeam)

Is there a better solution I can use / not using the management-network for backups?

So there is no chance of high speed backup with veeam without altering my environment?

This is also nice:
Generally speaking, network mode is recommended with 10 GB Ethernet.
https://www.veeam.com/blog/vmware-backu ... ation.html

Really, at that speed?!

And in another thread a veeam supporter says: If you have 10GbE for the ESXi mangement network, network mode can easily transfer 300-400MB/s per host.

Hm I am getting only the mentioned max. 50MB/sec :-(
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by wishr »

Hi gentlemen,

I would recommend taking a look at that thread explaining what "Source" bottleneck means.

Thank you
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy »

Have you tried any other transport mode (hotadd, direct SAN)?
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

@wishr thank you
@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned) ;-)

If I choose to set up a veeam proxy in a VM, do I have to buy additional licenses? I have licensed veeam backup enterprise+ for 4 sockets.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by wishr »

Hi Mcdaniels,

No, you do not have to additionally license any of the B&R components in this situation. The license simply controls the size of the virtual environment that you'd like to back up.

BR,
Fedor
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy »

mcdaniels wrote: Jan 14, 2019 7:46 pm@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned) ;-)
Correct, but on 10GB network mode should be faster than you're seeing. Trying some other transport will tell if it is the storage that cannot perform faster or environmental (networking) issue.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

@foggy: I will try. Just doublechecked that the management-network of my esxi is running @ 10GBIT Nic -> confirmed. But I have configured a 10Gbit standby adapter, which - in my opinion - should not interfere.
management network setting is looking like (german):

Image

Switch is 10Gbit.

Here you see the harddisc-performance of the backupserver: https://vmware-forum.de/download/file.php?id=1184&t=1
Here you see the harddisc-performance of the esxiserver: https://vmware-forum.de/download/file.php?id=1185&t=1
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by nitramd »

@mcdaniels, have you considered running iPerf to test the 10Gb network?
dimaslan
Service Provider
Posts: 99
Liked: 9 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by dimaslan » 1 person likes this post

Do you perhaps setup any throttling policies? If it was the network that was causing this, it should say so in the Bottleneck description, whereas now it says "Source". Apparently since a simple copy is allowing for the large speeds you expect to get, it should be something else causing this. The backup and the manual copy, were they performed from the same datastore?
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Hello,
sorry for coming back to you that late, but I had a lot of other urgent things to do (including illness). The case is not solved yet.
nitramd wrote: Jan 28, 2019 2:42 pm @mcdaniels, have you considered running iPerf to test the 10Gb network?
have to look into this. Have no clue how it works (yet).

@dimaslan: I have not throtteling policies set up.

Yes backup and copy were performed from the same datastore.

Just running a backup-copy-job, which also says that the source is the bottleneck:

Source 76%
Proxy 17%
Network 22%
Destination 12%

Transfer rate about 120Mb/sec
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

so, just managed to do iperf tests:

ESXI = Client:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec  8626536   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   2.00-3.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   4.00-5.00   sec   395 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec  4286340760   0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
CPU Utilization: local/sender 50.2% (50.3%u/0.0%s), remote/receiver 0.7% (0.1%u/0.7%s)
Backupserver (Windows) = Server:

Code: Select all

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.226, port 16240
[  5] local 192.168.10.219 port 5201 connected to 192.168.10.226 port 29629
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec
[  5]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec
[  5]   2.00-3.00   sec   394 MBytes  3.30 Gbits/sec
[  5]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec
[  5]   4.00-5.00   sec   395 MBytes  3.31 Gbits/sec
[  5]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec
[  5]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec
[  5]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec
[  5]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec
[  5]  10.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
-----------------------------------------------------------
And the other way around:

ESXI Server:

Code: Select all

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.219, port 57808
[  5] local 192.168.10.226 port 5201 connected to 192.168.10.219 port 57809
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   176 MBytes  1.47 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   3.00-4.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]  10.00-10.05  sec  10.9 MBytes  1.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.05  sec  2.20 GBytes  1.88 Gbits/sec                  receiver
Backup (Windows) Client:

Code: Select all

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   184 MBytes  1.55 Gbits/sec
[  4]   1.00-2.00   sec   194 MBytes  1.63 Gbits/sec
[  4]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   3.00-4.00   sec   234 MBytes  1.97 Gbits/sec
[  4]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver
CPU Utilization: local/sender 15.2% (4.1%u/11.1%s), remote/receiver 4.6% (4.6%u/0.0%s)
Seems pretty fast?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by Gostev »

The issue here is either with primary storage array or ESXi management stack (if you're using NBD). The latter can be excluded by trying hot add transport mode instead.

By the way, I noticed you still didn't include a support case ID for this issue. This is against forum rules provided when you click New Topic. Please include the support case ID if you'd like to keep the topic open.

Thanks!
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » 1 person likes this post

Support Case id: #03912843


Thanks!
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Gostev wrote: Dec 15, 2019 2:09 pm The issue here is either with primary storage array or ESXi management stack (if you're using NBD). The latter can be excluded by trying hot add transport mode instead.
Hotadd transportmode is only available if the Datastore is attached via NFS or Fibre, am I right? Our datastore is a direct attached one.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by Gostev »

No, hot add is available regardless of storage type. But of course, in case of DAS you need to use a backup proxy VM on the same host.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Ok, different information from different parts of the supportnetwork ;)

are there any docs/howtos on how to set up a backup-proxy-vm?

Is this a separate Veeam-Appliance (a VM?)

Or in other words. At the moment I have a all in one installation. All Veeam Services on one server.

How can I "split" of the backupproxy / install a separate backupproxy?

Which OS is required?
Where do I get the setupfiles for the Veeam-Proxy?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by wishr »

Hi Mcdaniels,

Please take a look at this article. System requirements for backup proxies can be found here. I would also recommend checking this BP portal section for additional information.

Thanks and let us know if you have any questions.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Hi,
in fact I am aware of setting up a backupproxy (we have running all components on one machine -> Windows Server 2016 -- proxy&repo).

I only would like to know:

If I want to use hotadd mode I cannot use our configuration, right? (because the veeamproxy has to run on the Vmware-host you do backups from and has access to the datastore)
Do I have to install a VM on the ESXI and install veeam on this machine to use hotadd mode?
So, I only can use our Backupserver (Windows 2016) as a Backuprepo (Backuptarget), right?

Thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy »

Deploy a VM on the required host and add it to Veeam B&R as a proxy using the procedure referenced by Fedor above.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Hi,
so it will be possible to use an existing server-vm on this host to act as veeam proxy? And I do not have to install components of Veeam B&R on this machine?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy »

Correct. Existing VM can be used as a proxy and all required components will be installed upon adding it to Veeam B&R as a proxy.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

Ah, I see. I always thought that you will have to install additional software on that existing VM.
Thank you.

I will give it a try.
mcdaniels
Enthusiast
Posts: 47
Liked: 5 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels »

So,
back from the test:
Proxy "hot add mode": Srv 2k16, 8GB Ram, 2x 2,1 Ghz Xeon, VMXNET3 - 10GBIT.

After the start of the Backup CPU of the proxy permanently at 100% load.
Throughput Veeam: 250 MB/sec.

Source: 3%
Destination:0%
Bottleneck 99% hot-add-proxy
Network: 96% (which I do not understand, cause it is used at max of 1,5 Gbit/sec and is a 10Gbit connection)

It keeps strange for me.

What do you say about that?

Thanks!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by wishr »

Hello,

Thank you for the info. Could you please make sure that it has not failed over to network mode - this can be checked within B&R UI in the job session log. If that does not point you to the root cause it would be better to continue the conversation you had with our support engineer as additional details will be required.

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy »

I'd also pay attention to the number of parallel tasks during that job.
oliverL
Enthusiast
Posts: 82
Liked: 11 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by oliverL »

mcdaniels wrote: Jan 17, 2020 7:52 pm Proxy "hot add mode": Srv 2k16, 8GB Ram, 2x 2,1 Ghz Xeon, VMXNET3 - 10GBIT.
Taking the system requirements for VM Backup Proxies:
https://helpcenter.veeam.com/docs/backu ... 95u4#proxy

and assuming this Backup Proxy isn't used as a repository server, you should at least have 3 vCPUs configured for your Proxy.
Since your Proxy is at 99% CPU load it seems that it can't process enough data to fully saturate the 10Gbit Link, therefore you could try to add more vCPUs.


How many virtual Machines and how many vmdks are to be backed up from the T640?
Post Reply

Who is online

Users browsing this forum: No registered users and 86 guests