Performance of Backup of S3

[mkretzer]

Hello,

we are trying to backup a CEPH S3 bucket with currently about 400 GB in preparation of backing up a 40 TB CEPH System.
Sadly, even incrementally backing up (without reading any changes) the bucket takes nearly a hour for only 1,6 Million objects. Our production storage has 130 Million objects!

At that speed even daily backup in unrealistic. We tried adding a Veeam infrastructure appliance as local proxy in the same network and it got even slower... We checked latencies and CEPH shows GET latency of 1,5 - 2 ms.
What performance can we expect from Veeam here?

Markus

[HannesK]

Hello,
I asked internally and one colleague had Ceph before at a customer and the root cause was a rate limit on Ceph side https://docs.ceph.com/en/latest/radosgw ... management

If that's not the case, then please open a support case and post the case number

Best regards
Hannes

[mkretzer]

Case # 08237241

[mkretzer]

It seems concurrency in one job is very limited. If we clone the same job two times and run all three jobs at the same time the total runtime in only slightly slower as with one job run. So the source system can deliver the speed without issue and Veeam just seems to do the S3 backup very inefficiently.

Do any of you backup S3 systems with many million objects?