Error when offloading to azure

[andre.simard]

Hi,

We have some issue with our SOBR offloading to Azure. It was working fine until last week when we start getting this error: Error: Failed to retrieve certificate from DefaultEndpointsProtocol=https;AccountName=xxxxxxxx

Anybody had this issue before or have an idea what can cause this?

We have check and all linux repo still have acces to internet.

Case number is 05010436. case is open since last thursday

Thank you

[Gostev]

Hi, yes we've seen this before. It's a firewall issue on your end. Thanks!

[andre.simard]

Hi Gostev,

I will double check to be sure that nothing has been changed, but it was working fine before and i'm not aware of any change in our firewall.

When i edit the configuration of the Azure repo and do next next next there is no error.

Thank you

[Mildur]

When i edit the configuration of the Azure repo and do next next next there is no error.

Configuration is done by the vbr management server. It's possible that this is working, but not from the backup repos if they are different machines.
Offload is done by Backup Repos or the vbr server, if you have linux hardened Backup Repos.

[andre.simard]

Offload is done by Backup Repos or the vbr server, if you have linux hardened Backup Repos.

Yes it's hardened Linux repo. In the BloB configuration i did not select any gateway

[Mildur]

Then you need to make sure, that your vbr server can access port 443 and all URLs from Azure as documented in this guide:
https://helpcenter.veeam.com/docs/backu ... onnections

Cloud endpoints:
xxx.blob.core.windows.net (for Global region)
xxx.blob.core.chinacloudapi.cn (for China region)
xxx.blob.core.cloudapi.de (for Germany region)
xxx.blob.core.usgovcloudapi.net (for Government region)

Certificate verification endpoints:
ocsp.digicert.com
ocsp.msocsp.com
*.d-trust.net

___________

Consider the following:
The <xxx> part of the address must be replaced with your actual storage account URL, which can be found in the Azure management portal.

Certificate verification endpoints (CRL URLs and OCSP servers) are subject to change. The actual list of addresses can be found in the certificate itself.

The *.d-trust.net endpoint is used for the Germany region only.

[andre.simard]

Thank you Mildur for your help!

the case engineer ask me to verify this also, but i have double check just to be very sure and yes i can access the endpoint xxx.blob.core.windows.net (for Global region) and all certificate verification endpoints on port 443 from VBR and Linux Repo

[pls]

I am sure our issues are not the same but just in case. Our backup server has 2 NIC cards. One for internal backup traffic and one for regular sever network. Backup vlan is not permitted external access at the firewall. When sending data to AWS about 75% of the time the backup traffic vlan was used and we would get a certificate error. We changed the preference so the regular server network is used to send data to AWS and have not had an issue since.

[andre.simard]

Hi pls,

Thank you for your suggestion, but it's not our case. we only have one NIC.

[andre.simard]

Hi,

I just find that i get the error: "Unable to use a Linux server registered with single-use credentials" when i tried to use the linux repo has gateway server for the offload.

is it normal? I didn't see this limitiation in the documentation

[veremin]

Correct

If you use single-use credentials, the host where the repository resides cannot have any other role: you cannot add it as a proxy or as a file server.

It does not specifically mention gateway server, but you get the idea, right.

Thanks!

[andre.simard]

thank you for the quick reply!

What is the recommendation in this case? we have multiple repo servers that need to offload backup. if we select only one gateway there is a bottleneck when they all tried to send data to that gateway.

Is is a good thing to change single-use credentials to standard credential?

Maybe if we can select multiple gateway instead of just one in a future release could be a good idea.

[Gostev]

Actually it's hard for me to imagine one gateway server being a bottleneck (as opposed to your Internet bandwidth or Azure blob storage performance). Gateway servers don't do any data processing or transformation, so they don't really require much compute resources. Can you share what specific resource is a constraint on that single gateway server, which makes it a bottleneck?

[veremin]

Is is a good thing to change single-use credentials to standard credential?

It's not a bad or good thing - it all comes down to your needs, requirements and security considerations. Andreas's provided some fine thoughts regarding it here; might be worth taking a look.

Maybe if we can select multiple gateway instead of just one in a future release could be a good idea.

This feature is on our radar.

Thanks!

[andre.simard]

Thank you for your answer!

[tmjwyo]

Hello,

We have been experiencing the same issue. We have support case #05132973 open with Veeam. My team informs me that they are seeing this with an increased frequency.

12/07/2021 00:30:33 :: Failed to offload backup. Error: Failed to retrieve certificate from DefaultEndpointsProtocol=https;AccountName=xxxxxxxxxx

I've seen a post referencing a firewall issue, @gostev can you please be specific?

We are on 11.0 and planning an upgrade to 11a. However, I'd prefer to hear a more specific reason to upgrade rather than "After looking over your logs I'd like to recommend you update your Veeam Version 11 to the latest 11a <https://www.veeam.com/kb4215> update that has a lot of fixes in place for capacity tier"

Looking for specific, root-cause, targeted input.0

Thank you.

[veremin]

Typically there is a certain connection block that prevents receiving or validating certificate, let support team analyze the debug logs and find what exactly is denied by firewall and I agree with you that you should not necessarily accept "update will fix everything" as an answer. But here on the forums we don't have your debug logs and environment overview, so cannot suggest reliably what might be wrong with your network setup.

So kindly keep working with the support team on addressing this.

Thanks!

[Priit]

So we upgraded to the latest version 11a about a month ago, and all backups and offloads have been working fine for a month, there have been no changes to any firewalls or network config and all backup related servers have a bypass on https inspection on the firewall. Now last night I got this same message from the offloading task:

Code: Select all

22/12/2021 8:52:10 PM :: Failed to retrieve certificate from DefaultEndpointsProtocol=https;AccountName=xxxxxxxxxxxxxxx  

Since it has been working with no issues and there have been no changes to the network, we are on latest version, so I dont really understand why it suddenly failed. The stranger fact is that we replicate about 20 odd servers to the same azure blob and we got this message for 2 of the servers while the other 18 replicated fine. Might be something on the Azure side that had a hickup, so will keep an eye on it, but if it starts doing that again, will have to open a ticket also.

[Gostev]

No Internet service can deliver 100% availability and reliability, and Azure is not an exception... so in general, sudden failures should not come as a surprise. As long as they are rare, it's normal and not something to worry about.

[mpm@aramark]

Has anyone fixed this?
Case #07360380