slow backup - bottleneck target

[liorme]

vmware environment
3 proxies
backup server
standard sata disks on repository capable of generating about 300 megabytes per second read/write
veeam v11 1261

connection is over 1gbps lan across all network
problem is that backups aren't crossing 300-400mbps

backup job reports :
source - 82%
proxy - 87%
network - 31%
target - 99%

backup job reports bottleneck - target

this also happens on a fresh backup, at the first round (meaning - unrelated to reverse incremental and forever increnetal)
thing is, if i take a large file and try to copy it from any of the proxies or from the backup server to the repo - i get full speed, about a 1000mbps (about 100-120 megabytes per second)
so i can't find a cause why veeam is not pushing full speed. i don't have any throttle enabled - i have disabled everything for testing.

any ideas?

[HannesK]

Hello,
well, source and proxy are also above 80%, so it's not the "classic" situation where everything is below 20% except target at 99%

How many VMs do you have in the backup job? And which machine is acting as repository? Is it the backup server? If yes, is the backup server also acting as proxy (just by chance, because it's only 4 machines)?

Do you have per-machine files or per-job files? Is a proper RAID controller installed (meaning one with write-back cache)?

Best regards,
Hannes

[liorme]

hi, thank you
how many vms ? = it doesn't matter. same problem for a job with 1 and for a job with 10
repository is a separate physical machine than the backup server
do i have per machine files ? no - per job (the default i think)
about the raid. there i remind that if i push a single large file for testing from any of the proxies or the backup server to the repo, i get full speed.

[Regnor]

Instead of copying a file, I would suggest that you use diskspd for performance testing. This will produce a more comparable IO load.
https://www.veeam.com/kb2014

Are you using virtual or physical proxies? Have you tried to disable on by one of those proxies to see if the performance changes?

[vmtech123]

Could be poor I/O on the target for sure. Copying 1 file is significantly different than actualy using your storage. Same as when I make my SAN use millions of IOPS using IO meter. It's not realisitc.

There will ALWAYS be a bottleneck with Veeam. As soon as you remove one something else becomes the most narrow part of the pipe.

Per-Machine is good because you can use more streams at once. Per job is 1 stream.

[HannesK]

I can only confirm what Regnor & vmtech123 said. Veeam IO is a mix of sequential and random IO, while the single-file copy is sequential only. Metadata operations are random. Although it looks like "one big backup file", there are data and metadata sections inside. That's why the single file copy is faster than a Veeam backup. File copy is still a good first test, but the diskspd would be the next step now.

Switching to per-machine chains usually helps with good RAID controllers / storage arrays, because single stream performance is usually worse than multi-stream. So it might be worth a try.

standard sata disks

okay, how many?

same problem for a job with 1 and for a job with 10

good to know. sounds like an IO limitation on the repository then.

[liorme]

i did a test of speed with hdd mark
which of there are important for me to do forward incremental (and later will be also merge)

i have tested with crtystal disk mark

seq1m-q8t1 - read 350mb/s write 208mb/s
seq1m-q1t1 - read 153mb/s write 112mb/s
rnd4k-q32t1 - read 55mb/s write 39mb/s
rnd4k-q1t1 - read 7.3mb/s write 3.2mb/s

[HannesK]

Veeam has a mix of block sizes. That's why it's a good idea to use the example values from https://www.veeam.com/kb2014 (as posted above).

With forward incremental, the question is, whether there are merges happening and whether REFS / XFS is used. I'm still missing the information on how many disks you have.

You can try "active full" backup. That would stop merges and write more sequential.

[liorme]

thank you