Veeam Failing "Offload to Archive" Tasks on SoBR

[bg.ranken]

Veeam support case ID: 05593210

So I wanted to post here because we are having an issue getting the Archiving tier to work in our SoBR. I haven't been able to find reference to these specific errors anywhere from Veeam or Microsoft, so am hoping someone else might have some insight, or if we end up finding a solution I can update this for others that might have the same issue.

When performing the "Offload to Archive" tier the job is failing for each VM with the following error:

8/26/2022 11:59:47 AM :: Task failed. Error: One of the HTTP headers specified in the request is not supported.
RequestId:94fe3cc2-****-****-****-************
Time:2022-08-26T18:59:42.2134472Z, Details: 'HeaderName':'x-ms-access-tier'
Agent failed to process method {LongTerm.Upload}.

I have tickets opened with both Veeam and Microsoft. Currently Veeam is saying the issue may be related to something altering the header within Azure, but is not sure and is asking for an Azure tech to review the requestID. From the Azure side, the Microsoft tech is saying he detected we are using Veeam version 10 (though we are currently on the latest, 11.0.1.1261) and that the issue may be related to the version we are using or that Veeam is not using the correct header settings.

Does anyone possibly have any insight into this? We are trying to utilize the Azure West-US2 region, so am wondering if possibly this is a restriction on this specific region.

[HannesK]

Hello,
the error seems to be very rare (I found an one year old support case for UK region, and the customer closed the case without solution, because he realized that he is using by far less space than he expected. so he cancelled the whole archive topic).

The only similarity I found between your case an the other is, that both seem to have permission / network restrictions in place. So my guess would be something around networking or permissions.

Let's see, whether any other customer can comment on the error itself.

Best regards,
Hannes

[veremin]

Cannot it be a West-US3 by any chance cause West-US3 does not support Archive Tier? Also, can you tell us what the storage account type is (using outdated or improperly upgraded accounts might result in the same issue)? Thanks!

[bg.ranken]

So after working with a Microsoft rep they discovered that there is official documentation from them stating that Archive Tier is not supported on ZRS, GZRS, or RA-GZRS.

https://docs.microsoft.com/en-us/azure/ ... ccess-tier

Only storage accounts that are configured for LRS, GRS, or RA-GRS support moving blobs to the Archive tier. The Archive tier isn't supported for ZRS, GZRS, or RA-GZRS accounts.

We had chosen ZRS for our storage account. Even the tech we were working with was surprised that this is the case, it only seems to be mentioned this one location in Microsoft's documentation; I haven't been able to find it anywhere else.

On Veeam's documentation side, I do see it listed under the documentation for Veeam Backup for Azure (https://helpcenter.veeam.com/docs/vbazu ... =40#backup and https://helpcenter.veeam.com/docs/vbazu ... positories) but don't see that limitation listed anywhere under the Veeam Backup and Replication documentation. The only references to limitations surrounding ZRS are related to restoring to Azure, or the helper appliance locations.

If it is there then I must have missed it, but I wonder if it might be possible to request that there is a warning popup somewhere in the console when either configuring the capacity or archive extents of a SoBR, similar to how the ReFS popup happens when creating a repository that is not formatted with ReFS 64KB cluster size. Luckily we don't have all our datasets uploaded, but we do have upwards of 40TB that we now have to move to a new storage account, so it is going to cost us quite a bit of time and money to reconfigure things.

[veremin]

Correct, Archive Tier does not work with ZRS-like accounts. We cannot guarantee that warning will make it into release, but we have already reached our technical writers to add this information to our User Guide. Thanks for raising this!

[bg.ranken]

Thank you for that.

In terms of where it would help for the documentation, I would definitely say under "Limitations for Microsoft Azure Object Storage" https://helpcenter.veeam.com/docs/backu ... ct-storage

But you also might want to add it somewhere as a warning under one or more of the sections regarding Azure storage https://helpcenter.veeam.com/docs/backu ... ml?ver=110. Currently if you are just searching the documentation for the process it takes you straight to this section, and unless you look around you may not even know the "Considerations and Limitations" section exists, so having a warning there would be helpful. Or alternatively you could have a link in a warning box under these sections that redirects to the Azure section of the Considerations and Limitations page.

[veremin]

Yes, these are the places we were thinking about adding a note to. Also, we might add a warning (like you have suggested) later on in post-v12 releases. Thanks!