Comprehensive data protection for all workloads
Post Reply
liorme
Enthusiast
Posts: 36
Liked: 1 time
Joined: Mar 12, 2019 6:29 am
Full Name: lior
Contact:

slow backup - bottleneck target

Post by liorme »

vmware environment
3 proxies
backup server
standard sata disks on repository capable of generating about 300 megabytes per second read/write
veeam v11 1261

connection is over 1gbps lan across all network
problem is that backups aren't crossing 300-400mbps

backup job reports :
source - 82%
proxy - 87%
network - 31%
target - 99%

backup job reports bottleneck - target

this also happens on a fresh backup, at the first round (meaning - unrelated to reverse incremental and forever increnetal)
thing is, if i take a large file and try to copy it from any of the proxies or from the backup server to the repo - i get full speed, about a 1000mbps (about 100-120 megabytes per second)
so i can't find a cause why veeam is not pushing full speed. i don't have any throttle enabled - i have disabled everything for testing.

any ideas?
HannesK
Product Manager
Posts: 14205
Liked: 2821 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: slow backup - bottleneck target

Post by HannesK »

Hello,
well, source and proxy are also above 80%, so it's not the "classic" situation where everything is below 20% except target at 99%

How many VMs do you have in the backup job? And which machine is acting as repository? Is it the backup server? If yes, is the backup server also acting as proxy (just by chance, because it's only 4 machines)?

Do you have per-machine files or per-job files? Is a proper RAID controller installed (meaning one with write-back cache)?

Best regards,
Hannes
liorme
Enthusiast
Posts: 36
Liked: 1 time
Joined: Mar 12, 2019 6:29 am
Full Name: lior
Contact:

Re: slow backup - bottleneck target

Post by liorme »

hi, thank you
how many vms ? = it doesn't matter. same problem for a job with 1 and for a job with 10
repository is a separate physical machine than the backup server
do i have per machine files ? no - per job (the default i think)
about the raid. there i remind that if i push a single large file for testing from any of the proxies or the backup server to the repo, i get full speed.
Regnor
Veeam Software
Posts: 929
Liked: 280 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: slow backup - bottleneck target

Post by Regnor »

Instead of copying a file, I would suggest that you use diskspd for performance testing. This will produce a more comparable IO load.
https://www.veeam.com/kb2014

Are you using virtual or physical proxies? Have you tried to disable on by one of those proxies to see if the performance changes?
vmtech123
Veeam Legend
Posts: 235
Liked: 133 times
Joined: Mar 28, 2019 2:01 pm
Full Name: SP
Contact:

Re: slow backup - bottleneck target

Post by vmtech123 »

Could be poor I/O on the target for sure. Copying 1 file is significantly different than actualy using your storage. Same as when I make my SAN use millions of IOPS using IO meter. It's not realisitc.

There will ALWAYS be a bottleneck with Veeam. As soon as you remove one something else becomes the most narrow part of the pipe.

Per-Machine is good because you can use more streams at once. Per job is 1 stream.
HannesK
Product Manager
Posts: 14205
Liked: 2821 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: slow backup - bottleneck target

Post by HannesK »

I can only confirm what Regnor & vmtech123 said. Veeam IO is a mix of sequential and random IO, while the single-file copy is sequential only. Metadata operations are random. Although it looks like "one big backup file", there are data and metadata sections inside. That's why the single file copy is faster than a Veeam backup. File copy is still a good first test, but the diskspd would be the next step now.

Switching to per-machine chains usually helps with good RAID controllers / storage arrays, because single stream performance is usually worse than multi-stream. So it might be worth a try.
standard sata disks
okay, how many?
same problem for a job with 1 and for a job with 10
good to know. sounds like an IO limitation on the repository then.
liorme
Enthusiast
Posts: 36
Liked: 1 time
Joined: Mar 12, 2019 6:29 am
Full Name: lior
Contact:

Re: slow backup - bottleneck target

Post by liorme »

i did a test of speed with hdd mark
which of there are important for me to do forward incremental (and later will be also merge)

i have tested with crtystal disk mark

seq1m-q8t1 - read 350mb/s write 208mb/s
seq1m-q1t1 - read 153mb/s write 112mb/s
rnd4k-q32t1 - read 55mb/s write 39mb/s
rnd4k-q1t1 - read 7.3mb/s write 3.2mb/s
HannesK
Product Manager
Posts: 14205
Liked: 2821 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: slow backup - bottleneck target

Post by HannesK »

Veeam has a mix of block sizes. That's why it's a good idea to use the example values from https://www.veeam.com/kb2014 (as posted above).

With forward incremental, the question is, whether there are merges happening and whether REFS / XFS is used. I'm still missing the information on how many disks you have.

You can try "active full" backup. That would stop merges and write more sequential.
liorme
Enthusiast
Posts: 36
Liked: 1 time
Joined: Mar 12, 2019 6:29 am
Full Name: lior
Contact:

Re: slow backup - bottleneck target

Post by liorme »

thank you
Post Reply

Who is online

Users browsing this forum: No registered users and 111 guests