S3-integrated Storage - Block Size

[tyler.jurgens]

With S3-integrated storage (Eg: minio) the storage tells Veeam what block size to use (in this case, Minio recommends a 4 MB block).

However, when I check the job settings on a new backup job, using the defaults, the block size is set to 1 MB.

Which block size will be used on the backup job? The size the Veeam job reflects, or the size the storage system recommends?

Assuming the block size used is the block size the storage system recommends and the settings in the veeam job are ignored: If I wanted to override the block size, how is that done?

[sfirmes]

@tjurgens-s2d The block size that you specify in the backup job is what is used when putting objects to object storage. The object storage vendor may have their best practice size recommendations and if you choose to use them that is where you would select the block size to be used.

[tyler.jurgens]

On Luca's blog post (https://www.virtualtothecore.com/a-firs ... am-sosapi/) it looks like it might be planned capability to force a default block size. I'm guessing that's not in place yet?

[Gostev]

No, this has not been implemented yet. Also, some people at Veeam argue if it's even a good idea to let storage vendors control this

For example, changing the default 1MB block to 4MB will approximately double your disk space consumption in the long run. This is obviously great for a storage vendor as they get to sell 2x more nodes, but not so great for your wallet or data center footprint.

[tyler.jurgens]

As with anything, let it be configurable. If the storage vendor's SOSAPI implementation forces 4 MB blocks, let the Veeam client override it. Eg: If I create a job against a S3-integrated bucket, the block size defaults to the SOSAPI block size. If I change it, it uses the block size I set.

For our Minio implementation, we'd take 4 MB blocks over hundreds of millions of tiny blocks negatively impacting our performance. We have one client backing up 45 TB using 223 million objects! It has made my leadership team want to avoid Veeam, while I've been trying my best

The storage size issue only really gets amplified if the clients retain long term incremental backups. Yes, it takes more space, but its a tradeoff we (at least in our specific situation) need to make. Given each GFS point already is effectively free, its not so bad.

[sfirmes]

Also when the SOSAPI starts to use the object size as noted in the blog, it will be at the bucket level and every job pointed to that bucket will use that object size. As @Gostev mentioned, you can see significant storage growth when increasing the object size from the 1MB default/best practice value. Due to that, you should try sticking to changing the object size on a workload/job level and monitor the effect of the change.

[tyler.jurgens]

That's all well and good, and we've been changing block sizes on individual backup jobs. However, when you're talking many customers, each with many jobs, changing these all takes a significant amount of time.
I agree that we need to change and monitor, which we have. It just doesn't scale well and is prone to human error.

Also concerned with how Capacity/Archive tier block sizes get set. For a direct to S3 backup job, its straight forward - change the block size on the job, run an active full and call it a day. Backup Copy jobs or Capacity/Archive tier jobs are more difficult - one has to change the primary backup job block size setting.

Unless the SOSAPI knows enough to change the block size on any jobs that either have a backup copy or are part of a SOBR with offloading to S3, forcing the block size on backup jobs alone isn't going to have a major impact, as I suspect most customers will use S3 as a backup copy target, or SOBR offload target. I may be wrong here though, Veeam may have more clear metrics on how S3 is being used.

[k00laid]

@gostev @sfirmes, I hate to resurrect this topic but I'm trying to understand the "why" of the storage increase when changing from a 1 MB to 4 MB block size. Is it just a matter of creating a higher liklihood of "misses" as deduplication is applied through block generation because 4x as much data has to match?

[Gostev]

Because incremental backups are block level so when a given data block had say 200KB of data changed in it, in the first case incremental backup will need to store only 1MB of data (which will include unchanged data that surrounds the changed 200KB), but in the second case 4 times more data will need to be stored (4MB block).

[emachabert]

This is in fact no so trivial when you have huge amount of data, metadata operations surrounding the data block (like retention information) are something to take into account.

We are dealing with perfomance (job duration) issues when you do immutable backup copy to an S3 object storage with mutli TB VMs (which are included in job with regular VM..). You can spend hours (like 5 to 10 hours) where veeam will not transfert any data but just storm the storage with millions of putObjectRetention api calls since there are so many objects to put the retention information on. We are currently in an extensive analysis of the pattern, fine tunning of Veeam internals and object storage internals with the help of both vendor support teams to mitigate this side effect.
At least this is interesting as deep diving the thing makes you understand how everything works under the hood.

[tyler.jurgens]

We have had the same challenges @emachabert. The small objects Veeam creates are definitely a challenge for any HDD based Minio deployment. Sure, we could solve those by switching to NVMe drives, but the cost of that is prohibitive.

Our best solution has been to increase block size to 4 MB (or even 8 MB, which is unlocked with a regedit). I would prefer if Veeam could allow us to set this value on Backup Copy Jobs and SOBR Offload settings, but as of yet that feature request has not been implemented. I understand the desire by Veeam to keep object sizes small, but it makes things very challenging for us.

We've been very honest with customers when telling them to go to 4 MB blocks and what the consequences are (increased size, fewer api calls, etc). Haven't had push back on it yet.