Backup Stuck On Storage Initialization - Wasabi

[TDG]

Backup Stuck On Storage Initialization - Wasabi
Case #06152757

It's been working all this time & bucket size is 48TB

Any ideas pls

[Andreas Neufert]

Thanks for sharing. I can not remember seeing this somewhere.

I would maybe check general things in addition to the support case:
1) Are the Veeam servers at correct time (use NTP) so that certificates for SSL encryption work correctly.
2) Does DNS work with for external URLs
3) Check firewall settings regarding the used port
4) Try to ping around in the internet and do a tracert. Do you go through the correct path to the internet in general and to the wasabi server?
5) Try another Wasabi region for a test.

[TDG]

Hi this morning all are fine
But a nice gentleman Jeremy Wolfe from Veeam US advise the below - Keep up the good work & a big thanks to all

Have a good day

Here is a link that outlines the patch notes for v12 and improvements for working with Object Storage repositories that may improve the issues you are currently facing:
https://www.veeam.com/kb4420
Object Storage
- Poor retention policy processing and backup deletion performance due to the loss of multi-threading in the backup removal process.
- Upgrading the metadata of backup chains with immutable copies stored in the Capacity Tier takes an extremely long time.

As for the "xxx_4_hour_to_wasabi" job, it's currently waiting on resource availability. To note, by default, Veeam only allows 4 VMware snapshots per datastore at any one time to prevent causing issues with the storage. This is a built-in limitation that can be increased via the registry key:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
MaxSnapshotsPerDatastore
• Type: REG_DWORD
• Default value: 4
• Description: specifies the number of snapshots that can be simultaneously created on the same datastore

This key will likely not exist unless it's been added manually previously. Once you create the registry key, you can set the desired value. The Veeam Backup Service will scan the registry nearly every 15 minutes and find the change, so no services or server reboots are required for this to take effect.

[Andreas Neufert]

Thanks, yes, updating to latest version / Patch level makes total sense.

[Maddin21]

The stuck on storage initialization with Wasabi is topic in this thread: veeam-backup-replication-f2/backup-copy ... 86650.html
Me as other customers have this issue not every day but in the last few days this is getting worse and worse. Running V12 since March and it was stable and fast with Wasabi so I don´t think this issue will be gone after the update but I will give it a try.

[TDG]

Thanks Maddin21. I killed the job & reboot Veeam & all are back to normal. I may open a case with Wasabi. I have a feeling V12 + Wasabi = Still issues which may be sorted over the time ?

[TDG]

in order to reduce the number of http requests to Wasabi I was asked to do this regedit

S3ConcurrentTaskLimit
1. Type: REG_DWORD
2. Default value: 32
3. Description: Amount of parallel HTTP requests for data upload (tasks) to the S3-compatible object storage (archive tier). Applied either on VBR server and applied automatically to all extents/gates, or can be set specifically on the extent/gate. Key on VBR server has a higher priority. (NOT APPLICABLE TO AZURE)

[Andreas Neufert]

With version v11 and newer no reg key modification is needed. You can control the load based on "maximum number of task slots used" setting for the bucket.

[cgsm]

This is the same issue I am having, I started the thread at veeam-backup-replication-f2/backup-copy ... 86650.html.

Setting the S3ConcurrentTaskLimit did not resolve the issue for me.

Setting the repo "Limit concurrent tasks to" checkbox/input to 4 (it was previously unchecked) seems to have alleviated my problem. I am not sure what the default value is when this setting us not checked/unused but maybe it is just way too high for Wasabi.

Still investigating with support...

[Andreas Neufert]

If this checkbox is not checked we use up to the number of task slots configures at the Repository (or SOBR extents). Each task slot that transport data does 64 parallel S3 operations.

[cgsm]

How do I determine "the number of task slots configure[d] at the Repository"? The "Limit concurrent tasks to" checkbox/field is at the repo level. Are we talking about the same thing here?