Discussions specific to the VMware vSphere hypervisor
Post Reply
mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Slow Backupspeed although 10Gbit network

Post by mcdaniels » Jan 10, 2019 10:15 pm

Hi,
I have got very slow backupspeed of max. 50MB/sec, running on a 10gbit network.
This only happens in VEEAM. When I copy a 2.2GB File from the VM to the physical backupserver 1 get a datarate of 1,2GB/sec.

The managment Network (which is used for veeambackups, if I am right) of the VMWARE ESXI 6.5 is connected to a 10Gbit Intel-nic, the backupserver uses the same Nic.

If I show at the veeam statistics, the bottleneck is the source:

Source: 99%
Proxy: 12%
Network: 2%
Target: 0%

The Esxi is a Dell Poweredge T640, Raid 10, 8HDDs, 10k, SAS.
The Backupserver is a Dell Poweredge T440, Raid 10, 6 x 4TB, SAS 7.2k

ESXI Version: 6.5 (all critical Patches applied)
Veeam Version: 9.5.0/1922

Veeam proxysetting: Transportmode: Network (automatic does no change to speed).

bdufour
Expert
Posts: 205
Liked: 40 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by bdufour » Jan 10, 2019 10:28 pm

https://bp.veeam.expert/proxy_servers_i ... twork_mode

Cons
Typically, network mode uses only up to 40% of the available bandwidth of the external VMKernel interface due to throttling mechanisms implemented on the management interfaces.

nitramd
Expert
Posts: 205
Liked: 48 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by nitramd » Jan 10, 2019 10:30 pm

How many physical CPUs and how much memory does your backup server have? Veeam uses 1 physical core per disk and about 4GB of RAM per disk.

Is your proxy also located on the backup server?

nitramd
Expert
Posts: 205
Liked: 48 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by nitramd » Jan 10, 2019 10:36 pm

Have a look at these proxy specs: https://helpcenter.veeam.com/docs/backu ... r=95#proxy

A proxy uses 2GB memory not 4 -> typo on my part

mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » Jan 10, 2019 10:44 pm

Hi,
yes proxy = backupserver
16GB of RAM
It is 1x Intel Xeon Bronze 3104, 1.7Ghz, 6-Core

I do no concurrent tasks, I do vm-backup one after one.

BUT: The bottleneck is source, so hm... it cannot be the backupserver? (as you said it is the backupmode of veeam)

Is there a better solution I can use / not using the management-network for backups?

So there is no chance of high speed backup with veeam without altering my environment?

This is also nice:
Generally speaking, network mode is recommended with 10 GB Ethernet.
https://www.veeam.com/blog/vmware-backu ... ation.html

Really, at that speed?!

And in another thread a veeam supporter says: If you have 10GbE for the ESXi mangement network, network mode can easily transfer 300-400MB/s per host.

Hm I am getting only the mentioned max. 50MB/sec :-(

wishr
Veeam Software
Posts: 1334
Liked: 134 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by wishr » Jan 11, 2019 11:11 am

Hi gentlemen,

I would recommend taking a look at that thread explaining what "Source" bottleneck means.

Thank you

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy » Jan 11, 2019 12:55 pm

Have you tried any other transport mode (hotadd, direct SAN)?

mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » Jan 14, 2019 7:46 pm

@wishr thank you
@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned) ;-)

If I choose to set up a veeam proxy in a VM, do I have to buy additional licenses? I have licensed veeam backup enterprise+ for 4 sockets.

wishr
Veeam Software
Posts: 1334
Liked: 134 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by wishr » Jan 15, 2019 10:17 am

Hi Mcdaniels,

No, you do not have to additionally license any of the B&R components in this situation. The license simply controls the size of the virtual environment that you'd like to back up.

BR,
Fedor

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by foggy » Jan 15, 2019 4:16 pm

mcdaniels wrote:
Jan 14, 2019 7:46 pm
@foggy: haven't tried it yet. But in fact network mode is the most compatible and the slowest (as mentioned in the docs). If I would like to go faster, I have to use another mode. It is by design. (this is what I learned) ;-)
Correct, but on 10GB network mode should be faster than you're seeing. Trying some other transport will tell if it is the storage that cannot perform faster or environmental (networking) issue.

mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » Jan 23, 2019 8:31 pm

@foggy: I will try. Just doublechecked that the management-network of my esxi is running @ 10GBIT Nic -> confirmed. But I have configured a 10Gbit standby adapter, which - in my opinion - should not interfere.
management network setting is looking like (german):

Image

Switch is 10Gbit.

Here you see the harddisc-performance of the backupserver: https://vmware-forum.de/download/file.php?id=1184&t=1
Here you see the harddisc-performance of the esxiserver: https://vmware-forum.de/download/file.php?id=1185&t=1

nitramd
Expert
Posts: 205
Liked: 48 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by nitramd » Jan 28, 2019 2:42 pm

@mcdaniels, have you considered running iPerf to test the 10Gb network?

dimaslan
Service Provider
Posts: 53
Liked: 7 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by dimaslan » Feb 04, 2019 2:42 pm 1 person likes this post

Do you perhaps setup any throttling policies? If it was the network that was causing this, it should say so in the Bottleneck description, whereas now it says "Source". Apparently since a simple copy is allowing for the large speeds you expect to get, it should be something else causing this. The backup and the manual copy, were they performed from the same datastore?

mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » Dec 15, 2019 9:31 am

Hello,
sorry for coming back to you that late, but I had a lot of other urgent things to do (including illness). The case is not solved yet.
nitramd wrote:
Jan 28, 2019 2:42 pm
@mcdaniels, have you considered running iPerf to test the 10Gb network?
have to look into this. Have no clue how it works (yet).

@dimaslan: I have not throtteling policies set up.

Yes backup and copy were performed from the same datastore.

Just running a backup-copy-job, which also says that the source is the bottleneck:

Source 76%
Proxy 17%
Network 22%
Destination 12%

Transfer rate about 120Mb/sec

mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » Dec 15, 2019 9:58 am

so, just managed to do iperf tests:

ESXI = Client:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec  8626536   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   2.00-3.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   4.00-5.00   sec   395 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec    0   0.00 Bytes
iperf3: getsockopt - Function not implemented
[  4]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec  4286340760   0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
CPU Utilization: local/sender 50.2% (50.3%u/0.0%s), remote/receiver 0.7% (0.1%u/0.7%s)
Backupserver (Windows) = Server:

Code: Select all

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.226, port 16240
[  5] local 192.168.10.219 port 5201 connected to 192.168.10.226 port 29629
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   327 MBytes  2.74 Gbits/sec
[  5]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec
[  5]   2.00-3.00   sec   394 MBytes  3.30 Gbits/sec
[  5]   3.00-4.00   sec   394 MBytes  3.31 Gbits/sec
[  5]   4.00-5.00   sec   395 MBytes  3.31 Gbits/sec
[  5]   5.00-6.00   sec   396 MBytes  3.32 Gbits/sec
[  5]   6.00-7.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   7.00-8.00   sec   393 MBytes  3.30 Gbits/sec
[  5]   8.00-9.00   sec   391 MBytes  3.28 Gbits/sec
[  5]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec
[  5]  10.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec                  receiver
-----------------------------------------------------------
And the other way around:

ESXI Server:

Code: Select all

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.219, port 57808
[  5] local 192.168.10.226 port 5201 connected to 192.168.10.219 port 57809
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   176 MBytes  1.47 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   3.00-4.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
iperf3: getsockopt - Function not implemented
[  5]  10.00-10.05  sec  10.9 MBytes  1.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.05  sec  2.20 GBytes  1.88 Gbits/sec                  receiver
Backup (Windows) Client:

Code: Select all

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   184 MBytes  1.55 Gbits/sec
[  4]   1.00-2.00   sec   194 MBytes  1.63 Gbits/sec
[  4]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   3.00-4.00   sec   234 MBytes  1.97 Gbits/sec
[  4]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   5.00-6.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   6.00-7.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   8.00-9.00   sec   234 MBytes  1.96 Gbits/sec
[  4]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver
CPU Utilization: local/sender 15.2% (4.1%u/11.1%s), remote/receiver 4.6% (4.6%u/0.0%s)
Seems pretty fast?

Gostev
SVP, Product Management
Posts: 25149
Liked: 3704 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by Gostev » Dec 15, 2019 2:09 pm

The issue here is either with primary storage array or ESXi management stack (if you're using NBD). The latter can be excluded by trying hot add transport mode instead.

By the way, I noticed you still didn't include a support case ID for this issue. This is against forum rules provided when you click New Topic. Please include the support case ID if you'd like to keep the topic open.

Thanks!

mcdaniels
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 18, 2014 10:13 am
Contact:

Re: Slow Backupspeed although 10Gbit network

Post by mcdaniels » Dec 15, 2019 3:02 pm 1 person likes this post

Support Case id: #03912843


Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 30 guests