SOBR: load balance performance over extents

[pirx]

Hi,

is there any way to force Veeam to distribute the load over multiple SOBR extents as evenly as possible? In my case those extents come from 3 high density servers, each with 2 RAID volumes = extents. If those 6 extents were volumes from a large SAN storage with distributed RAID, the available IOPS would be the same, no difference if 3 or 6 extents are used by Veeam. But the single extents of a high density server provide only limited IOPS/throughput. So it would be ideal if Veeam would distribute the jobs over as many extents as possible (if there is enough free space). I don't want to create a lot of different SOBR's for the jobs, I hope there is a way to achieve this with just one.

[HannesK]

Hello,
nice to hear that you already got your servers

So it would be ideal if Veeam would distribute the jobs over as many extents as possible (if there is enough free space)

That happens automatically. Out of the box. One SOBR with 6 extents is fine. The backup chains must be on the same extent anyway to allow block cloning.

Best regards,
Hannes

PS: I'm curious whether you have already seen IO limitations, or whether you are predicting the future. If you are trying to predict the future based on the infrastructure you have today, then I would say that's impossible. 3 servers with XFS / REFS are completely different architecture that is much easier from a performance point of view. I recommend avoid thinking in "NAS issues" when you have block based storage.

[pirx]

Hi,

nice to hear that you already got your servers

well... no yet, but almost the ones for the copy repositories. As you might have seen in the other thread I'm still looking into different options for backup repositories.

PS: I'm curious whether you have already seen IO limitations, or whether you are predicting the future. If you are trying to predict the future based on the infrastructure you have today, then I would say that's impossible. 3 servers with XFS / REFS are completely different architecture that is much easier from a performance point of view. I recommend avoid thinking in "NAS issues" when you have block based storage.

One point I'm struggling with in this scenario is that a repository server is bound to the local storage in terms of CPU/MEM, which then limits the possible concurrent tasks. I can easily add more proxy resources, but for repository I can only add a new server with additional storage. So it's important to choose the "right" building block. Not only in amount of storage, also IOPS. I still can't imagine how such a high density server with so few disks in RAID6 will be able to write backups, read backups and copy it to other repo + S3 bucket. And on top of that maybe sureback.

[HannesK]

I only check unanswered threads

as I said... please avoid mapping your current NAS issues on proper block hardware. It's a different level of storage quality.

The most conservative calculation we do is 5x. That means 60 disks with REFS / XFS equal 300 disks with NTFS and probably 400 disks on NAS.

Sure, there are physical limits. I believe that we talked about that in the RAID-rebuild earlier. So you probably will go with less than 10TB per disk instead of 18TB per disk (or whatever the largest disk available is)

In general, I recommend to size in a way that backup / offload to S3 and SureBackup run at different times. I usually recommend to plan backup resources for an 8h backup window, then there is 16h for copy / sure backup. With your internet connectivity, it should be much faster then, because you are above average with internet speed.

Best regards,
Hannes

[pirx]

That happens automatically. Out of the box. One SOBR with 6 extents is fine. The backup chains must be on the same extent anyway to allow block cloning.

I don't see how this would work here: https://helpcenter.veeam.com/archive/ba ... ement.html and in the past I've seen some extents used more others less. Even if slots and free space was available.

I think the only rule regarding performance is that an extent is chosen by available task slots. So If I set a lower number of concurrent tasks the tasks will be spread over all available extents. But I don't think this is real load balancing. I may have to set this limit even if a single server/extent is capable of delivering more tasks/IOPS for read operations. For example as source for copy/offload jobs. In this case I don't want to have the limit that I've set for ingress, I want all possible resources of an extent to be available.

How would the tasks be spread over the extents if I have the following:

- one SOBR, 3 server, each with 2 extents and 26 concurrent tasks/extent (52 cores per repository server, that would be maximum with the golden rule of 1 core for 1 task )
- all server have approx. same amount of free space
- or (for whatever reason) some are more utilized than others
- 50 Veeam tasks
- does Veeam check latency of an extent? Maybe CPU resources/slots are sufficient, but storage latency explodes...


From the Extent Selection part in the manual I don't see how a decision will be taken (which order) and how this will be a "real" load balancing.

[HannesK]

https://helpcenter.veeam.com/docs/backu ... ml?ver=110 - yes, task slots are checked before free space. point 3 vs. 4

It's up to you, how complicated you make the administration of the system. I would just go with one extent per server and set the task slots the same on each host. If you believe, that you can do better manual load balancing... the software allows manual placement

If the machines are repository-only, then you can set the task slots higher. 2:1 or 1.5:1 is the range most customers do. If the proxy runs on the same machine, then proxy tasks need to be reduced (proxy should have 1:1 cores, or only very little overbooking)

I repeat: IOPS are no problem with block cloning. Realistically we are talking about 10x-100x speed improvement for merge processes (remember that I calculated with 5x earlier). The only challenge customers with block cloning have is "free space balance".

does Veeam check latency of an extent?

no. That value is unreliable as standalone value.