Backup Copy Stuck On Storage Initialization, Slow, Hours

[cgsm]

I have a Backup Copy job that copies from my local VBR server repo to Wasabi. This job runs every 6 hours after the Backup Job runs. 20 VMs are backed up and copied. Every so often, and twice this week so far, the job gets stuck on the "Initializing storage" step for a few VMs for hours. Eventually, the "Initializing storage" step completes, again after hours, and the Backup Copy completes.

I cannot see any consistent cause or reason for "Initializing storage" to take forever. My Wasabi repo isn't full, I don't have very long backup chains (14 days on the main repo, 7 days + 2 weekly on Wasabi, weekly synthetics). There is near zero CPU, memory, disk, or network usage.

I haven't opened a support case yet because I haven't been able to reproduce the stuck/slowness.

Any ideas of things to check?

[Dima P.]

Hello cgsm,

Please make sure that source repository gateway has access to Wasabi storage. Can you please clarify how Wasabi storage is configured (gated or direct connection mode)? Thank you!

[cgsm]

@Dima,

Connection mode is "Direct". The VBR server is the gateway and does have access to Wasabi. I know the configuration is okay since Backup Copy jobs work most of the time.

[Dima P.]

cgsm,

Thank you! I'd recommend to go with support team as the log investigation is required. Please share the support case with us once you have it!

[cgsm]

Case # 06033960

[mortalfoil]

I have similar "storage initializing" delay issues with Wasabi, and constant http 12029 errors. I haven't yet submitted a case. Will watch to see if this gets any kind of resolution.

[cgsm]

I have gone back and forth with Support a bit. It was suggested to set the S3ConcurrentTaskLimit regkey to 8. I did so and have been monitoring. Since the issue is so intermittent, and I am not sure what causes it or on what schedule it occurs, I don't have much to say yet.

Since implementing the regkey, I did have a few VMs that took 40-60 minutes on the "storage initializing" step. Far better than a couple hours, but still extreme when most VMs complete in a few minutes. Support is taking a look at my latest logs from when these most recent events occured.

Other than that, just monitoring...

[CSmith686]

I am experiencing the same issue. Did changing the registry fix the issue?

[cgsm]

Changing the registry value did not seem to help. I still have jobs that get stuck at "Storage Initialization" for multiple hours. Veeam support has narrowed this down to block deletions on the object storage (Wasabi). We (myself and Veeam) are now in touch with Wasabi to try and compare logs.

[Dima P.]

Hello folks,

Thank you for sharing the update! I've asked QA team to review logs as well, stay tuned.

[Maddin21]

Same here, VBR direct connection to Wasabi storage. Yesterday copy job of our SQL instance stuck at 0% at "initialising storage". Prior to this the vSphere cluster copy job run on one VM into errors. Several "Cannot connect to target backup repository" errors followed by 'Item xxxx is locked by running session Retention job [Retention job]'

Occasionally we have 'Error: HTTP exception: WinHttpQueryDataAvailable: Error code: 12152 Agent failed to process method {Cloud.DeleteCheckpointNoSession}'

[cgsm]

I have been in contact with Wasabi and Veeam support about this. We have not determined anything yet. The issue does seem to have something to do with multi-block delete calls.

Wasabi said that there is some sort of rate limit, I think 250 connections, but was unsure if I was hitting it (they had to escalate at their side and I haven't had a reply yet). This has to do with the VBR S3ConcurrentTaskLimit regkey and the "limit concurrent tasks to" field for the repo. VBR had me set the S3... key to 8 but we missed that that the "limit concurrent tasks to" was unchecked meaning unlimited, therefore the S3... key had not effect supposedly. Anyway, the "limit concurrent tasks to" field was set to 4. Per my understanding this means 4 * 8 = 32 connections to Wasabi will be used, way under the limit.

Regardless, the issue still occurs with VMs stuck on "initializing storage" for 6+ hours. There is no schedule to when these delays occur, but it does seem to affect VMs with higher amounts of changes (not total GBs, but lots of little changes). Veeam had me enable some extra S3 related logging to see if there is any more info we can gather.

[Maddin21]

This is getting more and more ridiculous with Wasabi performance. The SQL copy job from yesterday broke due to transfer window restrictions. Current job is stuck on 'initializing storage', several other VMs from the job failed with error 12152 again. Only 7 out of 30 workloads are good. I´m already thinking about to move to another S3 storage.

[galbitz]

It might be cleanup related, i am not sure what to make of the wasabi/s3 support. I had a 1tb VM that was just doing small incrementals for 30 days. We had to reseed it, removing it via veeam was not possible, the operation would time out after 12 hours. Support (wasabi's) gave me a script to run and that took 4 days of constant running to delete the backup files. Based on that I have concerns about the long term viability of this process. We moved to a new bucket so we were able to just trash that bucket and continue on. I had a similar incident the other day where we scrapped a backup chain of a similar-sized VM and restarted a new fresh copy in the same bucket. The s3 objects went orphaned and I tried again to remove it from the veeam console. The removal job ran for 12 hours and then threw a bunch of timeout errors. The storage is no longer listed in veeam, but I assume I will be paying for some of that in wasabi forever now. I cant run the script because it doesn't accurately detect files associated to a job or not it only removes the entire bucket..

[Maddin21]

The last couple of days (since 6th of July) the performance significantly increased. Whatever have been done, this is not related to https://www.veeam.com/kb4420 as I do not have this patch applied yet. Seems Wasabi did some work on there backend?

[patrickvs]

I also experienced a lot of issues with Veeam deleting objects on de Wasabi storage.
It seems that Veeam by default limits the maximum concurrent object to be deleted to 1000.
In some cases this ends up in time-outs or even corruption.
I changed (added) the S3MultiObjectDeleteLimit registry key to 500 and also the S3RequestTimeoutSec to 900.
After this all jobs run successfully.

[cgsm]

@patrickvs

So by reducing the number of blocks deleted at one time (default 1000 -> 500) the performance improved? I would have expected the opposite since more API calls to Wasabi need to be made.

[patrickvs]

That is right.
By limiting the amount of simultaneous deletions you prevent time-outs to occur which improves the overall performance.
On some locations a higher value works out fine and in some cases even the default value works fine

[Maddin21]

On our side this Regkey setting did not changed anything. Five days it was running without any error and today I have the same again: 'error code: 12152 Agent failed to process method {Cloud.DeleteCheckpointsNoSession}'

Really thinking about to change the S3 storage provider.

[cgsm]

@Maddin21
I think your issue is different than mine and what we have been discussing. Yes, both are Wasabi related, but from some quick research on your errors I think it is something different. I do not experience your error message at all.

[cgsm]

So, so far we are thinking that getting stuck on "Storage Initialization" is due to a combination of the repo setting "limit concurrent task to" checkbox/field and the S3ConcurrentTaskLimit regkey. Leaving the checkbox unchecked is a potential cause of the issue; the default value being too high (not sure what it is) and thus causing issues with Wasabi.

Still working through this will support...

[Tic]

We have been struggling with this same issue for weeks

We manage two different servers from two different companies, with different SO, communications lines, firewall, antivirus…., but a very similar problem, with big MVs, every x days, the job stuck on storage initialization.
Our record so far is +65h.


First we opened a case and we were asked to add those keys
#Registry keys removed by Mod. Please don't share them publicly.


After some time we opened a second case for the same problem

We were asked to add
#Registry keys removed by Mod. Please don't share them publicly.

But the problem persist
Then, in different steps, support asked us to remove the backup window from the Copy Job, change the Copy job from “Immediate Copy” to “Periodic copy” in order to change the Schedule of the backup Copy and finally to disable firewall and antivirus on backup and Gateway servers…
Nothing worked

To make matters wose, on V12 Veeam did not implement the option to make Backup Copy Jobs from other copy Jobs, so when the Copy Jobs is stalled for 60h the actual MV backup job is unable to merge the backup chain files, since the file is locked by the Copy job, and fails

I came to this post 1 week ago looking for hope, and after reading CGSM´s last post, I checked the “limit concurrent task” on the repository and set it to 2. The job stalled for more than 8 hours at the time, just started to work
After more than a week, the problem have not come back

We have also updated Veeam B&R to the last patch, but in my opinión, that “limit concurrent task to” check did the trick

Thanks @cgsm!

[Mildur]

Hello Tic

I'm glad you could resolve the issue with our customer support team.
I removed the keys from your post. Those keys should only be made available via our support team. If you leave your case number, other customers can reference to them in their own support case.

Thanks,
Fabian

[cgsm]

Just a note to everyone reading this, my issue kind of just "went away". Veeam support and I could not reproduced the issue, even after unchecking the "Limit concurrent tasks to" field/input for the repo and removing any S3 related regkeys (reverting my settings to how they were when I opened my case). Support and I are unsure of what happened, but we think maybe Wasabi changed some settings on their end that were reducing throughput or enforcing stricter API call limitations. I continue to monitor my jobs to Wasabi but haven't experience an issue in a few weeks now.