SOBR Offload and Direct to Object Rescan Issues

[EricinIT]

I'm creating this post to get some insight into what could be causing SOBR object repository offloads and direct to object repositories to continually need rescans. As a Service Provider, we've had numerous cases into Veeam Support over the last few years due to SOBR offload issues and direct to object storage backups needing rescans. Sometimes rescans will fix the issue but there are also times where we have to reseed the jobs or totally replace the capacity tier extent. This is no small task especially when these tiers sometimes hold upwards of 250TB. I don't know if this is just expected with the amount of SOBRs and direct to object repositories we support, but it's a continuing issue. The issue affects numerous different Veeam servers even some in tenant sites so it's not environment specific. However, the object storage is mainly Wasabi us-east-1.

Below are some of our case numbers that I found over the past few years with a quick search through existing and closed tickets related to the object storage. All of them are slightly different but all related to the SOBR or direct to object backup functionality in Veeam.

Case #06032122
Case #07047854
Case #07383849
Case #07528925
Case #07828798
Case #07828667

These are not the only times we have the issues as we will put a ticket in usually if the simple rescan doesn't resolve the issue. We see the rescan issues with SOBR offloads and direct to object backups on almost a daily case to which we have to continually rescan these repositories. I'd appreciate any input as to what could resolve this permanently.

Eric

[david.domask]

Hi EricinIT,

Thanks for sharing the cases and sorry to hear about the challenges.

I would advise continue with Support and request a root cause analysis on the errors -- the need for the rescan happens when there is an inconsistency between the Backup Server's understanding of the state of the backups on Capacity Tier and what is found when actually checking Capacity Tier -- the check is done to ensure backup integrity and to avoid creating inconsistent states.

However, there can be quite a few reasons why this might happen, typically related to connectivity to the S3 provider and / or issues on the S3 provider side, and this will require a review of the debug logs to understand, which Support can help with.

From experience, I've seen this happen a lot when the S3 target handles situations like deletes or API limits in a non-ideal way (returning incorrect information to the S3 client (Veeam in this case), such as a CRUD operation failing that actually succeeded on the S3 side), but as noted before this is very much so contextual.

I see a few of the cases are already with Advanced Support (T2), so please continue working with them and ask for root cause analysis to better understand where and why the issues appeared.

[EricinIT]

Ok, will do.

Eric

[Gostev]

Also, you may want to consider adopting Veeam Vault in future, as to remove any Wasabi unknowns from the picture and have the entire solution delivered by the single vendor. Since Wasabi is a black box for us, understandably there's only so much we can do when things don't work well. With one of very few options being the above-mentioned annoying rescans.