Slow S3 backup-remove

[ortec-RW]

We use Cloudian S3 compatible object storage with immutable backup option via SOBR in Veeam. Backups are retained for however long we have set them, including the period they should be immutable in the S3 buckets.

However, objects are added and removed from backup schedule as time moves on. Machines (VMs) that no longer need to be backed up are removed from the jobs. But the old restore points of those VMs remain in the repositories. They will not be deleted. You have to do this manually.

On normal repositories this is pretty easy and quick. But removing old data from SOBRs is very very slow. I'm currently looking at a delete job I started 24 hours ago. It contains 2 VMs that need their restore points deleted (4 RP's per VM), in total about 460GB. The progress indicator is at 0% still, after 24 hours.

A similar job using a regular repo will do this in less the 2 hours. Why is the SOBR, including the S3 extend) taking so much longer ? Nothing else is running at the moment. Eventually the delete job will probably fail, and it would still not have removed all the old data. I can also no start another delete job, until the 1st one has finished.

What am I doing wrong here ?

[Mildur]

Hello Richard

We cannot troubleshoot deletion performance in this forum.
Please note, this forum is not an alternative place to get support for technical issues and unexpected behavior. Veeam employees in this forum cannot investigate the issue for you.
This is all explained in a banner with red letters (together with our forum policy) when you have created this topic.

So the steps you should do are:
1.) Export a current log package
2.) Open a support case
3.) Upload the exported logs to the case

Then follow the instructions you get from the support engineer in your case. It's also worth to check the Cloudian CPU statistics.

Thank you for understanding,
Fabian

[Gostev]

From the product perspective however SOBR is no different from "normal" object storage repositories in how it stores data in object storage, or how old data is removed. There's no special data storage format or dedicated code just for SOBR or anything like this, it's the same format and code.

Thus my only guess is that Cloudian S3 buckets behind your SOBR simply do not perform as well as buckets behind "normal" repositories. Perhaps they are overloaded with too many objects for Cloudian to handle per single bucket, or something along these lines? Did you follow your storage vendor's recommendations of max number of objects and/or maximim data size per bucket?

If that's the case, try creating multiple separate buckets for your SOBR tier to use, this will make Veeam place each newly added machine into a different bucket (round robin).

[ortec-RW]

First thanks to Fabian and Gostev for your replies. I do have a couple of comments.

I consider this Forum to be a mirror for my knowledge and usage of Veeam products. I think we can learn from each other if we discuss or talk about things going wrong, or even things working fine. I do not expect a solution to every problem I run into. It's like having a collegue you can talk to and check if you missed anything and for someone to have a fresh view on the matter. I don't asume to know every aspect of the Veeam product, others might know more. There's always more to learn.

I do not get that same experience in co-op when opening support calls. At any vendor for that matter. Its much more official and you need to follow the procedure for opening ticket, adding logs, and then hope the 1st line support even took the effort to read what you've put into the ticket. I understand why it's organized that way, we have the same within our org. And some customer support desks are better then others, Veeam is definitely one of the better ones. But it's not the same as asking around in the Forum. Often something simple you've overlooked or did not think about can help. Especially if someone else has run into the same problem. The collective knowlegde in this Forum is always bigger than what any Supportdesk can deliver. It's a lot quicker this way. And at the same time, if nobody comments or shares new light on the matter that is fine too. Like I said, I don't expect a solution in the Forum. When opening a support-ticket I have a lot more expectation for it to be solved.

Talking about that last comment, I think many of us know when using multiple vendors there's a big chance they point at each other for getting the issue solved. In this case Veeam and Cloudian. I also understand that, both should be consulted. But I could very well end up in the middle, with no help at all. Here's where a Forum could help you get things moving.

As for the issue itself. The CPU stats on the Cloudian are ok. Not higher or lower compared to when jobs put data into the buckets, instead of deleting them. If the code for SOBR is no different then any other repo, and if the number of buckets is not a problem when putting data into them, it's even stranger that deleting data from those same buckets is so slow. The job I talked about yesterday is still running, after 43 hours. And still at 0%. Even if it's actually doing something, that % counter should show.
A side effect is that while the delete job is running, the SOBR is locked for other jobs. So the daily offload of new backups that need to be added also fails. Probably because there's no multiple buckets. That would require a re-design of our environment.

But lets see what the Supportdesk of Veeam and Cloudian make of this. Oh and btw, I'm not running the latest version. So need to update first. After that job is finished.

[Gostev]

Richard, we simply ask not to bring environment-specific issues here, as they do not add any value for this community (no users care to sort through and read about those) and yet they are extremely distracting for the small team behind these forums and take meaningful time of our day to attend to. So please, don't just ignore the forum rules linked when you click New Topic with your future contributions to this community.

Look - you're not even using the latest product version, you did not engage either support orgs yet, and yet before doing any of these obvious first steps you decided to bring your issue for discussion straight with the R&D team. Obviously, this can never scale to the number of customers we serve... it's akin to DDOS for this unique resource, with more and more users doing this lately.

Thank you for understanding.