SOBR Offload failing to offload to Wasabi

[ahoyle]

"Hello,

We have been having this issue with one of our clients for over a month now. Their SOBR Offload is failing and is causing the storage media to become full and we can no longer backup any new data.

Support found some errors in the logs so we upgraded to the latest cumulative update to resolve those. ( We confirmed that the root cause of the issue is per a known issue for the v12 Build that you upgraded to (CF-12.0-CP1-20230223) where a conflict between SOBR offloads and Backup Chain "Upgrades" to "True Per-VM Backup Files", where the offload job processes backup files to the capacity tier that the chain upgrade process still thinks should exist on the local performance tier.)

Did a rescan of the Wasabi Repo with all jobs disabled.

Uploaded new logs and it was found there was a an issue with some metadata that was in the Wasabi Repo - Was asked to delete items from Wasabi to fix upgrade chain errors that were found.
The deletion process from Wasabi is very time consuming as it took almost 11 days to delete 2 folders, granted there was a LOT of files in those folders. (We used Cyberduck to remove items from Wasabi repo).

Finished the removal of items from Wasabi and once I re-enabled the SOBR from Maint mode it started an offload task updating the backup chain of those two jobs that I removed the metadata from. The task failed as it ran out of available memory (we had it set to 32GB). We adjusted memory to 96GB to see if it would help.

After adding in RAM and rebooting server it started a new offload job. As that one was running it added a new offload job every 4 hours or so that just cancelled itself at the end as the original offload was still processing. The original offload was upgrading the backup chain each taking about 30min per backup file/restore point. We lost network connectivity to this site and the job ended up failing.

Support suggested we attempt a manual offload of a single chain to see if we could just clear up some space to turn backups back on. It ran for about 65 hours then failed out due to the server restart.

All of the jobs since (from over the weekend) have finished "successfully" but nothing is moving, it finishes with the following message - "Timed out waiting for the index lock to release, backup chain will be processed during the next offload cycle."

Currently I am awaiting a reply from my last log upload from this morning.

Anyone see this before, or perhaps, have I overlooked a similar post in the forum?

Case ID #06126003

B&R v12

[Andreas Neufert]

This is sad to hear. I will escalate your support case from our side. As well I will inform the responsible PM. Maybe he has an additional insight/idea.

[ahoyle]

Hello,

Thank you for getting that escalated. It appears it was escalated on Tuesday and we had yet to hear anything back. I have followed up within the case and asked for an update. We are currently stuck with no new backups until we can get the offload issue resolved as we have no space available and unable to add more. We are breaching SLA requirements by not being able to add any new backups. I was requested to add additional logs this morning, but have not had any updates on any progress on this issue.

[ahoyle]

Hello.

We are unfortunately unable to add more space for the SOBR for this specific client. Has anyone else had a similar experience with this because we have been dealing with this issue going on almost 2 months now and it's impacting our ability to perform regular backups.

@Andreas Neufert any other options you can try by chance?

Thanks!

[Andreas Neufert]

I would work with Support on fixing the situation. The support case noted that you canceled one of the tasks that support asked you to perform (can take a longer time I think).

Other than this I would update to the latest Veeam patch/version as the team is constantly adding improvements.