Performance using direct2object (on prem)

[Markus M.]

Hi there,

we are currently running a PoC using Veeam V12 and Datacore Swarm object storage.
Storage layer is running on 3 phys. hosts (DL360 G8 with 8*1.2 TB SAS, each); configuration is EC 4:2
Discovered poor performance when running a direct2object (Active full of a 5 TB VM just processes 120..140 MB/s)
Same backup running into a SMB-Repo (where the host just has 8*8TB NL-SAS) runs with a processing rate of nearly 600 MB/s
Even Hardware isn't very up to date, I'd expect more throughput with this setup.
I am in discussion about this with DataCore, but no finding so far.
Can't see any bottlenecks when looking at perfmon counters of the gateways and storage nodes, looks ok.
Storage network is 10Gb, jumbo frames enabled, encapsulated in separate VLAN, no congestion on switch.
Does anybody has a similar setup running on prem (maybe with a different type object storage) and would share the processing rate?

Thanks!
Markus

[veremin]

The best course of action will be to open a support ticket for this issue and let the support team find out the weakest chain, based on the debug logs and detailed investigation. Thanks!

[Markus M.]

Hello Vladimir,

ok - done.

Thanks,
Markus

[veremin]

Thank you. Kindly, share the ticket for the convenience of future readers.

[Markus M.]

Here the Case-ID: #05996827

[karsten123]

any news on whats wrong?

[veremin]

The support engineer has requested a new set of logs recently, so the investigation goes on.

Are you struggling with the performance of object storage repositories yourself, or it is a question of general interest? In the former case, you might want to reach the support team on your own to speed up the process.

Thanks!

[karsten123]

it is just a question of general interest

[Markus M.]

Hi there,

no reply from Veeam Support so far, further there was no request regarding logs in the case?!
With following changes the performanve could be increased to now approx. 320 MB/s:

- installed faster CPU in storage nodes (was 6, now 10 core and 2.5 instead of 2.1 Ghz) --> there was a misinterpreting of Grafana perfmon counters regarding CPU load with old CPUs, they were constantliy running at 70...80% during active full backup.
- reviewed settings for jumbo frames in storage network layer --> doublechecked functionality between all nodes and content gateways.
- changed Swarm cluster setting: policy.eCMinStreamSize from 1Mb to 8Mb
- changed Backup Job setting "storage optimization" from 1MB to 4MB

This seem to be the most I can expect with this old hardware setup (3* Proliant DL360 G8, (*1,2TB 10k HDD)
I'd be glad if anybody founds this helpful.

Thanks,
Markus

[Gostev]

Please note that using 4MB block in Veeam doubles your backups size (on average). I personally never saw this as a good trade off for the lack of storage performance. Only good for storage vendors because this means you will need 2x more storage!

[Markus M.]

@Gostev:
Thank you for pointing out this correlation. Another backup ran with the 4MB storage optimization setting and I couldn't see a doubling of space used. Anyway, I will keep this in mind.

Currently I am testing with latest patch V12 Build 20230412 and additionally we discovered another bottleneck on the gateway servers of SWARM (high CPU load), which could be (partially) reliefed.
I have started another active full, with this changes a further increase of processing rate was achieved, now the job runs with almost 450 MB/s (processing rate), not too bad for the (already retired) 9 years old server systems.
And I have to admit that when I first recognized the performance issue, I was too focused on the storage nodes as possible culprit, but as always, a "broader" view into the whole systems reveals the real reason - in this case network-, CPU-ressources and job-settings.

[Gostev]

To clarify, your incremental backup sizes will double (on average, in a typical environment, across all machines). And since Veeam is forever-incremental, it translates into the entire backup footprint doubling.

The reason for this doubling is very simple: image-level backup is done block by block and so every tiny disk change will require storing a 4MB block with unchanged surrounding data (vs. only 1MB with the default settings). This is however balanced by the fact that tiny isolated disk changes are not common, so at least some part of this block will have other changed data. As a result, instead of mathematical 4x increase of incremental backup size according to difference in block sizes, it is typically only 2x. By the way, this number was questioned internally as well a number of times and because of that there were many tests performed in different environments in the past years which did confirm times and again this to be a good estimation.

[vmtech123]

Thanks for the clarification Gostev!

Out of curiosity, how much space (Average) is saved going to 512 or 256? Or is the performance trade off not worth it?

[Gostev]

Space savings should always be more or less half of the mathematical difference between the two block sizes. Performance will suffer indeed with smaller blocks, however you should never use blocks smaller than 1MB with object storage for a different reason: it will cause bad scalability issues with most object storage out there due to 2x or 4x more objects required to store the same amount of data. As the number of object per bucket is the primary architectural bottleneck for most object storage out there.

All things considered, default 1MB block size is really the golden middle to stick with when backing up to object storage. And while you can gain something from using a different block size, like faster backup performance or lower object storage space usage, you will always lose much more and it's rarely worth it.

BTW, one other big drawback of large blocks which I did not mention is much worse performance of file/item-level restore and instant recovery. It's easy to overlook just like I did because we always tend to worry about backup performance first and foremost, while what matters even more is how fast can you restore.

[vmtech123]

Very good point.

Interesting ideas, and I'll leave it at the default.. It never hurts to have this knowledge if for some reason a specific use/case comes down the road though.