Backup Stuck On Storage Initialization - Wasabi

[TDG]

Backup Stuck On Storage Initialization - Wasabi
Case #06152757

It's been working all this time & bucket size is 48TB

Any ideas pls

[Andreas Neufert]

Thanks for sharing. I can not remember seeing this somewhere.

I would maybe check general things in addition to the support case:
1) Are the Veeam servers at correct time (use NTP) so that certificates for SSL encryption work correctly.
2) Does DNS work with for external URLs
3) Check firewall settings regarding the used port
4) Try to ping around in the internet and do a tracert. Do you go through the correct path to the internet in general and to the wasabi server?
5) Try another Wasabi region for a test.

[TDG]

Hi this morning all are fine
But a nice gentleman Jeremy Wolfe from Veeam US advise the below - Keep up the good work & a big thanks to all

Have a good day

Here is a link that outlines the patch notes for v12 and improvements for working with Object Storage repositories that may improve the issues you are currently facing:
https://www.veeam.com/kb4420
Object Storage
- Poor retention policy processing and backup deletion performance due to the loss of multi-threading in the backup removal process.
- Upgrading the metadata of backup chains with immutable copies stored in the Capacity Tier takes an extremely long time.

As for the "xxx_4_hour_to_wasabi" job, it's currently waiting on resource availability. To note, by default, Veeam only allows 4 VMware snapshots per datastore at any one time to prevent causing issues with the storage. This is a built-in limitation that can be increased via the registry key:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
MaxSnapshotsPerDatastore
• Type: REG_DWORD
• Default value: 4
• Description: specifies the number of snapshots that can be simultaneously created on the same datastore

This key will likely not exist unless it's been added manually previously. Once you create the registry key, you can set the desired value. The Veeam Backup Service will scan the registry nearly every 15 minutes and find the change, so no services or server reboots are required for this to take effect.

[Andreas Neufert]

Thanks, yes, updating to latest version / Patch level makes total sense.

[Maddin21]

The stuck on storage initialization with Wasabi is topic in this thread: veeam-backup-replication-f2/backup-copy ... 86650.html
Me as other customers have this issue not every day but in the last few days this is getting worse and worse. Running V12 since March and it was stable and fast with Wasabi so I don´t think this issue will be gone after the update but I will give it a try.

[TDG]

Thanks Maddin21. I killed the job & reboot Veeam & all are back to normal. I may open a case with Wasabi. I have a feeling V12 + Wasabi = Still issues which may be sorted over the time ?

[TDG]

in order to reduce the number of http requests to Wasabi I was asked to do this regedit

S3ConcurrentTaskLimit
1. Type: REG_DWORD
2. Default value: 32
3. Description: Amount of parallel HTTP requests for data upload (tasks) to the S3-compatible object storage (archive tier). Applied either on VBR server and applied automatically to all extents/gates, or can be set specifically on the extent/gate. Key on VBR server has a higher priority. (NOT APPLICABLE TO AZURE)

[Andreas Neufert]

With version v11 and newer no reg key modification is needed. You can control the load based on "maximum number of task slots used" setting for the bucket.

[cgsm]

This is the same issue I am having, I started the thread at veeam-backup-replication-f2/backup-copy ... 86650.html.

Setting the S3ConcurrentTaskLimit did not resolve the issue for me.

Setting the repo "Limit concurrent tasks to" checkbox/input to 4 (it was previously unchecked) seems to have alleviated my problem. I am not sure what the default value is when this setting us not checked/unused but maybe it is just way too high for Wasabi.

Still investigating with support...

[Andreas Neufert]

If this checkbox is not checked we use up to the number of task slots configures at the Repository (or SOBR extents). Each task slot that transport data does 64 parallel S3 operations.

[cgsm]

How do I determine "the number of task slots configure[d] at the Repository"? The "Limit concurrent tasks to" checkbox/field is at the repo level. Are we talking about the same thing here?

[bluj83]

I am having issues with the job getting stuck on storage initialization as well. The only way to fix it is to stop Veeam or reboot our Veeam server. If we force stop the job it takes forever to stop. We have one job going to Wasabi now that won't even run. Is this a Wasabi problem or a Veeam problem? We are on us-east-2.

[cgsm]

@bluj83 Try checking and setting the "Limit concurrent tasks to" field for the repo (Backup Infrastructure > Backup Repositories > choose repo > Edit Repository) to 4.

This might have helped, but I could never really identify a specific thing that solved my issue. I also looped in Wasabi's support at the time so I am not sure if they tinkered with some settings on their end.

[bluj83]

We have it checked and set to 4. The stuck job we had yesterday fixed when I told it to run an Active Full. This seems intermittent. Its like we could go a week or 2 without the issue then suddenly its a problem. We also have 6 separate jobs running to 1 bucket but it is random on what backup job doesn't work!

[jkolson]

Adding to this topic, I've had this exact same issue ever since v12 was deployed. Now I'm on 12.2, and although intermittent, this random issue of getting stuck on "storage initialization" seems to be worse. I have one job currently running, an incremental, and its been on initializing storage for 10 hours now. I had a very long running open case with Veeam on this topic and Wasabi engineering was brought in as well. Ultimately the finding on the case was that the issue is related to some kind of retention process Veeam is doing, and last I heard from Veeam support was supposed to escalate the issue internally but closed my case. #06296327. Is there any acknowledgement from Veeam that this is a noted problem that is being addressed?

[MaartenA]

Same here since 12.2 Case # 07497412

[bluj83]

This issue has popped up again. Often the only way for me to fix it is to stop the job forcefully and reboot the Veeam server.