Performance of Backup of S3

[mkretzer]

Hello,

we are trying to backup a CEPH S3 bucket with currently about 400 GB in preparation of backing up a 40 TB CEPH System.
Sadly, even incrementally backing up (without reading any changes) the bucket takes nearly a hour for only 1,6 Million objects. Our production storage has 130 Million objects!

At that speed even daily backup in unrealistic. We tried adding a Veeam infrastructure appliance as local proxy in the same network and it got even slower... We checked latencies and CEPH shows GET latency of 1,5 - 2 ms.
What performance can we expect from Veeam here?

Markus

[HannesK]

Hello,
I asked internally and one colleague had Ceph before at a customer and the root cause was a rate limit on Ceph side https://docs.ceph.com/en/latest/radosgw ... management

If that's not the case, then please open a support case and post the case number

Best regards
Hannes

[mkretzer]

Case # 08237241

[mkretzer]

It seems concurrency in one job is very limited. If we clone the same job two times and run all three jobs at the same time the total runtime in only slightly slower as with one job run. So the source system can deliver the speed without issue and Veeam just seems to do the S3 backup very inefficiently.

Do any of you backup S3 systems with many million objects?

[osnexus]

It's super important to offload the rgw.bucket.index to all flash media. In addition you'll want to setup two storage classes, STANDARD should be replica-3 (all-flash) and your EC pool should be STANDARD_IA. With that you can setup a preRequest rule in the RGW to place objects > 64K into the STANDARD_IA pool. Hope that helps, you can handle billions of objects in your ceph cluster, just need to get it all setup correctly.

[Bjoern_ch]

Hi,
we use CEPH too.
In total we have billions of objects and many PB of data. Works quite well.
But as said before setting it up correctly can be a bit tricky.
Small objects can be become problematic with CEPH in our experience. Therefore, we are using large object sizes (4MB) with vbr only.