Object Storage Server Example

[Murigar]

We are looking at migrating our 100% on premise environment to M365.

I am working on digesting VBO for the first time. If I can I would prefer on premise object storage.
It looks to me like I can spin up MinIO on some local servers, setup a proxy or two.

Let's say 1000 users, and how about 300 TB of data.
Probably setup two storage servers. Primary and backup copy.
What would some server builds look like for this?

This could be easily achieved with OK performance in TrueNAS Scale, with the MinIO container.
Or, an Ubuntu server with a bunch of spinning disks and XFS or a RAID card. Performance much worse.

Specific questions:
Is this crazy talk?
Do I need the IOPS of flash?
Do any of you have examples of your hardware build/disk layout?

Thank you.

[Mildur]

Hi Murigar,

I don’t have experience with MinIO performance as a backup repository, as it’s not part of our lab environment.
Therefore, I can’t provide specific design recommendations beyond suggesting the use of SSD-based storage for better performance.
Let’s see if other customers or partners can share their insights on MinIO usage. You can also find related discussions in this subforum using the search function. One of our Veeam Legends, @edh, shared his experience with MinIO last year — perhaps he can provide additional input.

Please note that while MinIO should work via its supported S3 API implementation, it hasn’t been tested yet under the Veeam Ready program (link) for version 8.0.

Alternatively, instead of operating your own object storage, you could consider subscribing to a Microsoft 365 backup service from one of our Service Providers or use Veeams own service offering Veeam Data Cloud for M365.

Best regards,
Fabian

[edh]

Hi Murigar,

MinIO works well with HDDs, but it’s important to understand that every S3 operation involves disk access.
For example, every HEAD request triggers a read from disk. Since HDDs have relatively low I/O performance, this can easily become a bottleneck. Minio dont use a "database" for HEAD operations like CEPH.

In our experience, we’ve run several large multi-node MinIO clusters for Veeam Backup for Microsoft 365 (VBO 365), hosting around 1 PB of data in each one.
Overall, it works great for daily operations — but there are some situations where you’ll need patience:

• Rescan/Resync operations: every metadata lookup or validation involves I/O, which makes these operations slow. Our biggest tenants got a bucket of 120TB and resync can take hours.

• Deleting organizations: deletion is always an expensive process, especially with SMR disks (I strongly recommend using CMR drives instead).

• Small object size: VBO stores a huge number of very small objects (often <1 MB). This leads to billions of files and can cause performance degradation due to object fragmentation.

• Minio DONT allow cluster expansion of a "single" node, if you start with blocks of 4 nodes, you always will need to expand in silos of 4 nodes.

For reference, a single cluster with 48 × 24 TB drives (≈ 860 TB usable capacity) can deliver around 11 Gbps, dropping to about 5 Gbps as it fills up.
A key recommendation: don’t let your object storage get too full — once it does, cleaning up or expanding the cluster becomes slow, difficult, and very manual. That’s one area where MinIO really struggles.

The biggest problem with object-based data storage is the type of data you store, and by this I mean the average object size. An optimal object size is 8MB to 32MB, but as we all know, few, if any, emails are that large.

Finally, note that MinIO always recommends using SSDs or NVMe drives to avoid I/O bottlenecks.

My last two cents: Please read ALWAYS the minio changelog, because minio is a company movement to force their customer go to paid-ultra-expensive minio services, removing features in their community server.

Veeam VBO Team, made the last years greats improvements with the performance of the S3 and the backends.

Hope my experices help! Thanks!

[edh]

Do any of you have examples of your hardware build/disk layout?

For HDD:
12 / 24 driver MAX per Node.
At least 128GB RAM / 256 recomended
2x10 Gbps , bonded will be good.
Dell R740XD2 / Dell R760

With NVME:
16/24 NVME per node... for cost effective i recomend 7.68 to 15.36/61TB per drive. But this cluster will be expensive.
Never never never is drives without PLP (Power Lost Protecction) this kind of software always use direct_io to prevent issue with caches.
In minio this behaivor can be changed with a "cli" but i prefer dont share to prevent you of weird issues if you got a unexpected shutdown.

Minio like all S3 storage got a gold rule, more drives in case of HDD, better performance. In some setup is better to get 12TB drives instead 24 TB drives and double the number of I/o available.

Dont be "docker" guy for this, docker add another layer of complexity when things goes wrong, baremetal just works.
Dont use RAID card in RAID 0, put the RAID card in HBA mode. RAID 0 is not a real HBA by-pass. I case of new dell, you can put drives in "non-raid drive".

Run a load balancer like HaProxy / Nginx for loadbalance properly for every node. I know that a single node can redistribute the re-write to other nodes, but this kill the setup, due each step will add some "ms" for the ack write of remote nodes.

DoNot Try geo-remote minio, it dont work. Its too sensitive to latency.

Fell free to ask anything @Murigar

[Murigar]

@edh Wow! What a gold mine.

This is perfect and exactly the type of feedback I was hoping to get.
I will digest this and post back with any follow up questions.