bottleneck determination: physical endpoints only. concurrent tasks (?)

[edison5000]

Our setup is simple and new, but appears very bogged down. Our previous EOLed product and much older server was crushing our Veeam rig's current performance. So I'm assuming either
a) that we're doing something wrong b) Veeam just needs more resources to do the same thing, and we'll have to buy 'em.

I realize this is likely a system engineer question (so I have ticket open), but the ticket title doesn't match the issue here since some prior tickets got merged. (Case #05482078)

Our rig is simple

We have a brand new Dell r7515 epyc AMD 24 core server
64gb ram
local raid array with 150Tb storage using ReFS & Raid 60, ssd's, so far a single repository. No additional software raid, if that matters.
10gb network and (1) 10gb card in the server, though endpoints mostly all have 1gb cards.

C: 480gb raid 1 using dell boss card, Win Server 2022, (1) Veeam B&R & Veeam One.
D: 480gb raid 1 using dell perc controller, SQL
E: is the repository, a single 150TB repo w raid 60 & megaraid controller. I have also redirected Veeam logs from C to E via Veeam documentation (reg keys) though the logs are not very big since we don't even have a week of backups.

We will ultimately have around 800+ machines here; previously we were backing up every machine every 20-30 min, and this was clearly messaged when we inquired about meeting our use case with our Veeam sales person and engineers. We are only backing up hardware desktops and notebooks. No vm's, no servers, no esxi. Strange, yes, but here we are. ; )

Right now we maybe have 350 desktops/notebooks online, but there will be more.

Because we must use Workstation mode to get enough licenses, each agent is set up to back up 'on action's - user logoff or lock workstation, otherwise we can't get enough backups per day. (At least, not from the veeam gui. You can't set multiple jobs on same protection group). In any case, it is struggling.

The number of machines 'waiting for available backup infrastructure' keeps slowly increasing up to 140+. This should be primarily incrementals because I spent all weekend making sure all the available boxes got their full initial backups done.

SQL is only using about 8gb of ram.
Task mgr shows total ram use is around 44%.
cpu is vacillating between 5-8%

We have about 6 jobs set up for the different departments, that all do the same thing. Figured it was easier this way that one huge job for everyone.

I had originally set the repo max concurrent tasks to 22, 2 less than total processor core number. As an experiment, I have been stopping the jobs and restarting them using slowly increasing # of concurrent tasks. I've got it up to 30 now, though the speed of tasks waiting in line "for available backup infrastruture" doesn't seem radically better. I feel like using less than the # of cores won't be better... should I be using even more? I've read multiple threads with people discussing this, some pertaining maybe only to vm's or proxy's, some not sure. 2:1 task:core?

How about 'use per-machine backup files'? We've got this turned on, but we're not using any actual vm's. Yea or nay?

Any other input would be super appreciated. Thanks!!

[PetrM]

Hello,

At first, I think the only way to shorten waiting time of backup infrastructure resources is to increase the number of concurrent tasks on repository. It's recommended to stick with the following rule: 1 task = 1 CPU core, for example you may have a look at this page. It seems that your current setup is fine, but according to the shared resource usage statistics (SQL - 8 GB, RAM - 44 %, CPU - 5-8 %), there is a room for gradual increasing of concurrent tasks of the currently used repository. On the other hand, I wouldn't go this path and would suggest to provision additional hardware so that you can increase concurrent tasks number and stick with the recommended sizing rule at the same time. Also, I don't think you should disable vbk-per-vm as it may affect data processing rate during backup and restore.

UPD 30/06/2022: Please see actual information regarding task limitation for repository in this post.

Thanks!

[edison5000]

PetrM - thank you very much for the quick response. So when you say "provision additional hardware" - meaning..

1. what hardware would you recommend? Just seems like performance is *radically* less than our previous rig, which overall was much older, processor wise, than the gear I mentioned above. We had no special extra deduplication hardware or proxy servers, or extra separate repo servers for that product.

2. why is increasing the concurrent tasks not recommended? Is it because these are or are not vm'?

Thanks again.

[PetrM]

Hello,

1. I was meaning to deploy one more repository server so that you can process more concurrent tasks in parallel. I think local RAID with REFS would be fine, maybe similar to what you have already.

2. These recommendations are based on results obtained in our internal labs during performance testing and are applicable to the majority of deployments. Obviously, it may depend on specific hardware so the best way to identify an optimal number of concurrent tasks is to test it in real environment. Anyway, I would try to keep everything as close as possible to what is recommended in order to prevent potential issues in future.

Thanks!

[edison5000]

Thanks again. One more question on this: over here in Best Practices

https://bp.veeam.com/vbr/3_Build_struct ... ositories/

it says 'start with 3 concurrent tasks per cpu core'. Is this only for vm's?

[PetrM]

Hello,

No, it's not for VMs only but for any type of workloads including physical servers.

Thanks!

[edison5000]

Petrm - Thanks again.

Now we are up to 52 (!) concurrent tasks despite only 24 physical cores. It is handling them easily. Now, it is finally, periodically using more cpu - but the highest it has gotten is maybe 20% max, even when all 50+ tasks are engaged. That seems quite different than wherever the Veeam docs or threads state "1 physical core per concurrent task is recommended".

Is it possible that the lab tests were underestimated? (or need updating?) As mentioned, we have no additional repo servers or proxy servers, it's just one Dell r7515 amd server with a local raid controller + array in it. Just putting that out there.

[PetrM]

Thanks for sharing this information with us. I wouldn't say that the tests are underestimated, the final results may depend on the specific hardware. But your case makes me think that it's worth discussing with our performance team. I'll update this topic as soon as I have more information.

Thanks!

[PetrM]

Hello,

Looks like the recommendation I was referring to is obsolete and 2 tasks per 1 CPU core for repository is totally fine in the vast majority of cases. We'll update our documentation soon, many thanks for paying our attention to this.

Thanks!