Slow S3 Offload to Local MinIO

[daw_idee]

Hello,
I have deployed a Single VM with MinIO in distributed mode (to obtain the object lock capability) with a ISCSI Storage on the background, all on a 1 Gbits network
I have successfully setup a sobr repo that offload the backups as soon as they are created, the problem is that I have really bad performance, with an average of 20 MB/s



also if I don't limit on the s3 repository the concurrent tasks i get this error when the offload starts:
1/22/2022 10:07:12 PM :: REST API error: 'S3 error: A timeout exceeded while waiting to proceed with the request, please reduce your request rate. Code: SlowDown', error code: 503. Other: Resource: '/', HostId: 'bbf7f867-9d2c-46a6-bfa3-8573403503a9'

There is something that I'm missing, some tuning or similar?
I've tested the write speed directly on the vm and reach consistent speed for a a 1 Gbits network

[sfirmes]

@daw_idee you did the correct thing by limiting the number of concurrent tasks. Each concurrent task can generate up to 64 simultaneous connections to the object storage. If you don't set a limit, the object storage can get overwhelmed by the number of connections and you will see those 503 error codes.

Are you using SSD drives for the storage being used by MinIO? We typically see better performance when using SSD with on-prem object storage.

Also Using MinIO with Veeam is a great page that may help you.

[Eco]

I have done a fair amount of testing with Veeam and MinIO, working together they are both capable of saturating whatever hardware they are running on. So, the first thing you would need to test is what the speed of the iSCSI disks is. 1Gbps network really isn't sufficient to do any sort of real world workload with Veeam or MinIO in my opinion, but there are no technical limitations in MinIO to prevent it (and nothing I am aware of in Veeam). After you have confirmed the disk speed, you should see what is happening on the MinIO cluster outside of Veeam. If you are running any recent version, you should be able to run `mc admin speedtest <alias>` and get a sense of what the MinIO node is capable of.

My guess is going to come down to a disk issue. Most likely, is that you are using multiple folders on the same disk (something like minio server /mnt/disk{1...4}) where there are not actually four disks physical disks underneath. This would cause significant write amplification and destroy performance. In any case, this setup isn't really suitable to test performance so I would not rely on it. I can tell you in the testing I did, Veeam and MinIO together were able to saturate the network on a 4 node cluster with 10Gbps network, and even 10Gbps is considered slow for MinIO.

Last bits of advice - if you are using a fairly old release of MinIO, you should upgrade as there are numerous performance improve,ments incorporated on a frequent basis. A lot of attention has going into small file performance increases, which would be something a setup this small would benefit from with Veeam. In the same vein, making sure you are using a newer kernel is beneficial any time you are testing performance in linux. Older releases of various enterprise linux distributions tend to use very old kernels (3.x). I would recommend at least a 4.x kernel and preferably 5.x as the autotuning of things like disk and network make a huge difference. You didn't really mention about your setup as well, but you should be using XFS (not ext4, def not ZFS) and not doing anything like RAID/LVM underneath. Since MinIO provides it's own consistency and durability guarantees via erasure coding, having other systems on top of it is only detrimental.

Happy testing and good luck.

[daw_idee]

the backend storage has near line 3,5 disks, usually we can get better performance if used as backup taget

[Eco]

Understood. The other points still stand, I would recommend taking a look at them. In any case, you should expect to see MinIO bottlenecked on I/O, preferrably on network, with the small file exception I noted earlier, and that only if the disks are not able to provide enough iops to satisfy requests (so it will more often be seen in hdd setups).

[daw_idee]

I have done a fair amount of testing with Veeam and MinIO, working together they are both capable of saturating whatever hardware they are running on. So, the first thing you would need to test is what the speed of the iSCSI disks is. 1Gbps network really isn't sufficient to do any sort of real world workload with Veeam or MinIO in my opinion, but there are no technical limitations in MinIO to prevent it (and nothing I am aware of in Veeam). After you have confirmed the disk speed, you should see what is happening on the MinIO cluster outside of Veeam. If you are running any recent version, you should be able to run `mc admin speedtest <alias>` and get a sense of what the MinIO node is capable of.

My guess is going to come down to a disk issue. Most likely, is that you are using multiple folders on the same disk (something like minio server /mnt/disk{1...4}) where there are not actually four disks physical disks underneath. This would cause significant write amplification and destroy performance. In any case, this setup isn't really suitable to test performance so I would not rely on it. I can tell you in the testing I did, Veeam and MinIO together were able to saturate the network on a 4 node cluster with 10Gbps network, and even 10Gbps is considered slow for MinIO.

Last bits of advice - if you are using a fairly old release of MinIO, you should upgrade as there are numerous performance improve,ments incorporated on a frequent basis. A lot of attention has going into small file performance increases, which would be something a setup this small would benefit from with Veeam. In the same vein, making sure you are using a newer kernel is beneficial any time you are testing performance in linux. Older releases of various enterprise linux distributions tend to use very old kernels (3.x). I would recommend at least a 4.x kernel and preferably 5.x as the autotuning of things like disk and network make a huge difference. You didn't really mention about your setup as well, but you should be using XFS (not ext4, def not ZFS) and not doing anything like RAID/LVM underneath. Since MinIO provides it's own consistency and durability guarantees via erasure coding, having other systems on top of it is only detrimental.

Happy testing and good luck.

thank you very much ill'do some testing!

[Gostev]

That's some great knowledge, thanks Eco for sharing indeed. I was under impression based on some other feedback that MinIO is not efficient enough with Veeam, and that saturation is only possible with synthetic tests. Can you please share what performance did you see with Veeam and MinIO for a single VM offload and restore direct from MinIO? And in what cluster configuration. Just so that there's a baseline for everyone to refer to.

[oscaru]

How is the iSCSI storage provisioned to the MinIO VM? an iSCSI initiator inside the VM OS? or an iSCSI initiator on the hypervisor, and then added like a virtual disk to the VM?

[Eco]

That's some great knowledge, thanks Eco for sharing indeed. I was under impression based on some other feedback that MinIO is not efficient enough with Veeam, and that saturation is only possible with synthetic tests. Can you please share what performance did you see with Veeam and MinIO for a single VM offload and restore direct from MinIO? And in what cluster configuration. Just so that there's a baseline for everyone to refer to.

@Gostev I was mostly testing 10 VM's at a time with 100GB randomly generated values. I probably did some single VM tests along the way as well. I tested on a few different instances with Equinix metal, with HDD and nvme in various configurations. The MinIO clusters were generally four nodes and a single instance of VBR running on a different ESX host. AFAIR I tested both bare metal instances of MinIO and running in VM's. In all cases, the bottleneck was I/O, so for example some of the configurations were single HDD per MinIO node, in which case aggregate disk speed was the cap, in other cases there were either enough drives or fast enough drives to saturate the connection. Unfortunately I can't recall the nvme instance type, I *believe* the HDD instances were this (or something similar) - https://metal.equinix.com/product/servers/s3-xlarge/.

Specific hardware itself isn't particularly relevant though, since MinIO is I/O bound, it really matters more what the disks/network can do. So, some baseline testing with dd/fio for the disks will show you what a given box can do at the disk level, and then you hopefully want to meet or exceed what the nic is capable of so that your network (ideally) is the bottleneck. You can use iperf/vnstat to confirm throughput, and the MinIO team just wrote a handy tool that will do essentially a distributed iperf - https://github.com/minio/dperf.
There is some discussion to be had for how Veeam persists objects to object store, and how MinIO handles erasure coding those objects, but that is probably a bit deep for this discussion. At a very high level you can think of a Veeam workload in MinIO as being a "small file" workload, and provision the hardware accordingly.

[sf4692]

Hello All.

i think my question is regarded to the one above so i post it here:

we offloaded from Veeam successfully to a Minio Testserver with 8 x 6TB SATA 7200 disks in Erasure Coding Mode. Until we hit the max. allowed restore points in Veeam everything was working fine and really fast (much much faster than the internet connection for upload was) but since Veeam needs to merge restore points to keep max. allowed quantity of restore points it does 30GB in 24hours which before it has done in like 30 mins..

Anyone any clue what that could be? Could some metadata from minio be placed on a NVME SSD to gain speed? thank you

[sf4692]

OK so from githut i got that information.

Using HDDs with Veeam you will only get so much and you also gave MinIO only 8 drives - that's hardly any spindles to do anything useful for Veeam like workload.

[Gostev]

Veeam needs to merge restore points to keep max. allowed quantity of restore points

Actually, merge never happens with object storage. Unlike with regular backup storage there are no backup files, so there's nothing to merge to start with. In case of object storage, retention processing comes down to deleting objects which no longer to any restore point under the retention policy.

Using HDDs with Veeam you will only get so much and you also gave MinIO only 8 drives - that's hardly any spindles to do anything useful for Veeam like workload.

Since we're talking strictly delete operations, there should be little to no I/O involved so the number of spindles should be largely irrelevant for this specific operation (retention processing). Plus he said he was happy with the offload performance anyway, which IS on the other hand an I/O heavy operation. So I suspect he might facing some architectural limitation that makes object deletes very slow on MinIO, but I can't be sure.

But the hardware itself actually makes for a decent entry-level backup repository, and he would see consistently great performance with any backup management operation if for example he used the same server as a hardened repository with XFS on top of regular RAID. I guess I just don't want anyone to have an impression that 8 spindles is "hardly anything" for Veeam. I mean, servers like DL380 have been a proven workhorse for our SMB customers for over 10 years now, and they only fit 12 LFF drives.

[Eco]

Without knowing much about the setup, 30GB in 24 hours is abyssmal, and 30GB in 30 minutes isn't much better. I have to assume you mean 30TB here? One thing that might help to understand a bit about how MinIO handled erasure coding - https://docs.min.io/minio/baremetal/con ... ure-coding. MinIO constructs one or more erasure sets per cluster based on the number of disks and servers. The number of disks per erasure set is between 4-16, with the highest common denominator being picked to allow for the best durability. The salient points to mention would be that for deletes, a seek operation has to happen to all disks in a given erasure set (in this case is sounds like you only have one of 8 disks) which of course will be more expensive with HDD than SSD. If you add versioning on top of this, then each object may actually have several versions that need to be deleted. If you had massive amounts of data constantly saturing 100Gbps links, then you will benefit from SSD's, but to @gostev's point, there is no reason I can think of offhand that HDD would not work for the amount of data you are describing. And, even for very large amount of data, if the request rate itself can be satisfied by the hardware, then you could have PB's of data on HDD (this is actually true for many deployments). If it was not a typo and you really mean 30GB in 30 minutes, that sounds like something off in the setup itself. The last point I would make is that MinIO is strictly consistent, so a 200OK is not returned until the operation has successfully completed on all available disks in an erasure set. If one of your disks is significantly slower due to hardware malfunction (or any other reason) then each delete operation will be slow as well. If you are running a newer version you can run `mc admin speedtest <alias>` and get a sense of what your cluster is capable of.

[sf4692]

Hi all, i can confirm that a DL380 Gen9 with a 8 Core CPU and 128GB + RaidController (Raid 6) with 1 SSD Drive as Read Cache in Front works perfectly with VEeam. Absoluteley no resource usage and super fast copy offload speeds.

Whereas Minio with 1 node (i must fairly say i use a homeserver Setup AMD Ryzen 5700G 8 Core with 128GB Ram and the Disks in HBA Mode to present to Ubuntu directly) works perfectly fine also in speed test until you have to merge restore points. The 30GB is true (it is not Terabyte) VM Jobs keep stuck in preparing index for offloading forever. We downgraded togetether with Veeam support the streams and also the timeouts, but it doesnt help.

see here the coresponding Github Discussion:
https://github.com/minio/minio/discussions/14317

What i have done now really is to get rid of Minio and use the Hardened Veeam Linux Repo offsite.