S3 On Premise OS block size

[efd121]

I have an on premise S3 appliance that was setup using 1M block size on the underlying OS. My backup jobs are also set to 1MB. We are seeing over 3x the amount of space being taken up on the OS compared to the backups. I was expecting the data written to the appliance be mostly 1MB in size but a random sampling I’m finding most files are much smaller than 1M.

I’m working with the vendor to determine what would be the most efficient block size, so we are not consuming 1M for every file. Is there a recommended block size I can have them consider that would be the most effective for Veeam? I want to be careful not to cause performance issues if the block size is set too small.

The last count there are almost 1.5B objects and if my sampling is anywhere close to the normal its very easy to see why the OS is using 3x times the space with the 1M block size.

Our backups are first written to hardened Linux repositories then offloaded to the appliance using the Copy backups to object storage as soon as they are created. The hardened Linux repositories are set for 7 days of immutability and the appliance is set to 30 days.

Dave

[sfirmes]

Our recommended block/object size is 1MB. Here is a great explanation of why your seeing a larger sized object storage repository than you are seeing for the linux repository: Object Repository with Immutability size explanation

Hope this helps.

Steve

[efd121]

I do have all the backup jobs set to 1MB for the Storage optimization. I'm looking for assistance what the vendor should set the underlying Linux file system block size. If Veeam was writing most/all files at 1MB or larger the 1MB block size would be fine but when I look at the files that are being written the vast majority are much smaller.

[sfirmes]

The objects will typically be smaller due to the effects of compression and deduplication. What object storage platform are you using? Is it an appliance or an opensource product you are implementing yourself?

Thanks

Steve

[efd121]

Its a Veeam certified appliance.
The vendor is collecting data to determine the optimal size, I was hoping that data was already available so I didn't have to wait several days for them to collect and analyze the data.

The longer I keep writing data using the 1MB block size the more time it will take to rewrite it with the updated size.

Dave

[sfirmes]

I work closely with all of our on-prem object storage partners. If you want to send me a private message and provide me the name of the partner, I might be able to help them provide you an answer.

Thanks

Steve

[karsten123]

just share the vendor name. what is your problem?

[sfirmes]

There might not be an issue with the object storage.

Based on our immutability sizing guidance How Immutability Works there can be a discrepency with the storage utilization of the performance tier compared to the capacity tier.

Let's use an example where the performance tier is block/file and the retention is 30 days and the capacity tier has a retention of 30 days with immutability set for 30 days.

In this example, the retentions are:

• Performance tier = 30 days
• Capacity tier = 70 days (30 days retention + 30 days immutability + 10 days block generation)

In this example, it wouldn't be surprising to see the capacity tier be approximately 2x the performance tier when comparing the storage utilization of the respective tiers.

Hope this helps.

Steve