Using object storage as a backup target
Post Reply
andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Error when offloading to azure

Post by andre.simard »

Hi,

We have some issue with our SOBR offloading to Azure. It was working fine until last week when we start getting this error: Error: Failed to retrieve certificate from DefaultEndpointsProtocol=https;AccountName=xxxxxxxx

Anybody had this issue before or have an idea what can cause this?

We have check and all linux repo still have acces to internet.

Case number is 05010436. case is open since last thursday

Thank you

Gostev
SVP, Product Management
Posts: 30029
Liked: 5936 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error when offloading to azure

Post by Gostev »

Hi, yes we've seen this before. It's a firewall issue on your end. Thanks!

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

Hi Gostev,

I will double check to be sure that nothing has been changed, but it was working fine before and i'm not aware of any change in our firewall.

When i edit the configuration of the Azure repo and do next next next there is no error.

Thank you

Mildur
Veeam Software
Posts: 4469
Liked: 1368 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: St. Gallen, Switzerland
Contact:

Re: Error when offloading to azure

Post by Mildur »

When i edit the configuration of the Azure repo and do next next next there is no error.
Configuration is done by the vbr management server. It's possible that this is working, but not from the backup repos if they are different machines.
Offload is done by Backup Repos or the vbr server, if you have linux hardened Backup Repos.
Product Management Analyst @ Veeam Software

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

Offload is done by Backup Repos or the vbr server, if you have linux hardened Backup Repos.
Yes it's hardened Linux repo. In the BloB configuration i did not select any gateway

Mildur
Veeam Software
Posts: 4469
Liked: 1368 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: St. Gallen, Switzerland
Contact:

Re: Error when offloading to azure

Post by Mildur »

Then you need to make sure, that your vbr server can access port 443 and all URLs from Azure as documented in this guide:
https://helpcenter.veeam.com/docs/backu ... onnections
Cloud endpoints:
xxx.blob.core.windows.net (for Global region)
xxx.blob.core.chinacloudapi.cn (for China region)
xxx.blob.core.cloudapi.de (for Germany region)
xxx.blob.core.usgovcloudapi.net (for Government region)

Certificate verification endpoints:
ocsp.digicert.com
ocsp.msocsp.com
*.d-trust.net

___________

Consider the following:
The <xxx> part of the address must be replaced with your actual storage account URL, which can be found in the Azure management portal.

Certificate verification endpoints (CRL URLs and OCSP servers) are subject to change. The actual list of addresses can be found in the certificate itself.

The *.d-trust.net endpoint is used for the Germany region only.
Product Management Analyst @ Veeam Software

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

Thank you Mildur for your help!

the case engineer ask me to verify this also, but i have double check just to be very sure and yes i can access the endpoint xxx.blob.core.windows.net (for Global region) and all certificate verification endpoints on port 443 from VBR and Linux Repo

pls
Enthusiast
Posts: 30
Liked: 7 times
Joined: May 17, 2019 5:44 pm
Full Name: Phil Shields
Contact:

Re: Error when offloading to azure

Post by pls »

I am sure our issues are not the same but just in case. Our backup server has 2 NIC cards. One for internal backup traffic and one for regular sever network. Backup vlan is not permitted external access at the firewall. When sending data to AWS about 75% of the time the backup traffic vlan was used and we would get a certificate error. We changed the preference so the regular server network is used to send data to AWS and have not had an issue since.

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

Hi pls,

Thank you for your suggestion, but it's not our case. we only have one NIC.

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

Hi,

I just find that i get the error: "Unable to use a Linux server registered with single-use credentials" when i tried to use the linux repo has gateway server for the offload.

is it normal? I didn't see this limitiation in the documentation

veremin
Product Manager
Posts: 19820
Liked: 2125 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Error when offloading to azure

Post by veremin »

Correct
If you use single-use credentials, the host where the repository resides cannot have any other role: you cannot add it as a proxy or as a file server.
It does not specifically mention gateway server, but you get the idea, right.

Thanks!

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

thank you for the quick reply!

What is the recommendation in this case? we have multiple repo servers that need to offload backup. if we select only one gateway there is a bottleneck when they all tried to send data to that gateway.

Is is a good thing to change single-use credentials to standard credential?

Maybe if we can select multiple gateway instead of just one in a future release could be a good idea.

Gostev
SVP, Product Management
Posts: 30029
Liked: 5936 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error when offloading to azure

Post by Gostev »

Actually it's hard for me to imagine one gateway server being a bottleneck (as opposed to your Internet bandwidth or Azure blob storage performance). Gateway servers don't do any data processing or transformation, so they don't really require much compute resources. Can you share what specific resource is a constraint on that single gateway server, which makes it a bottleneck?

veremin
Product Manager
Posts: 19820
Liked: 2125 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Error when offloading to azure

Post by veremin »

Is is a good thing to change single-use credentials to standard credential?
It's not a bad or good thing - it all comes down to your needs, requirements and security considerations. Andreas's provided some fine thoughts regarding it here; might be worth taking a look.
Maybe if we can select multiple gateway instead of just one in a future release could be a good idea.
This feature is on our radar.

Thanks!

andre.simard
Service Provider
Posts: 155
Liked: 13 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Error when offloading to azure

Post by andre.simard »

Thank you for your answer!

tmjwyo
Novice
Posts: 3
Liked: never
Joined: Dec 02, 2021 8:57 pm
Full Name: Thomas Moore
Contact:

Re: Error when offloading to azure

Post by tmjwyo »

Hello,

We have been experiencing the same issue. We have support case #05132973 open with Veeam. My team informs me that they are seeing this with an increased frequency.

12/07/2021 00:30:33 :: Failed to offload backup. Error: Failed to retrieve certificate from DefaultEndpointsProtocol=https;AccountName=xxxxxxxxxx

I've seen a post referencing a firewall issue, @gostev can you please be specific?

We are on 11.0 and planning an upgrade to 11a. However, I'd prefer to hear a more specific reason to upgrade rather than "After looking over your logs I'd like to recommend you update your Veeam Version 11 to the latest 11a <https://www.veeam.com/kb4215> update that has a lot of fixes in place for capacity tier"

Looking for specific, root-cause, targeted input.0

Thank you.

veremin
Product Manager
Posts: 19820
Liked: 2125 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Error when offloading to azure

Post by veremin »

Typically there is a certain connection block that prevents receiving or validating certificate, let support team analyze the debug logs and find what exactly is denied by firewall and I agree with you that you should not necessarily accept "update will fix everything" as an answer. But here on the forums we don't have your debug logs and environment overview, so cannot suggest reliably what might be wrong with your network setup.

So kindly keep working with the support team on addressing this.

Thanks!

Priit
Novice
Posts: 8
Liked: never
Joined: Sep 07, 2020 11:10 pm
Contact:

Re: Error when offloading to azure

Post by Priit »

So we upgraded to the latest version 11a about a month ago, and all backups and offloads have been working fine for a month, there have been no changes to any firewalls or network config and all backup related servers have a bypass on https inspection on the firewall. Now last night I got this same message from the offloading task:

Code: Select all

22/12/2021 8:52:10 PM :: Failed to retrieve certificate from DefaultEndpointsProtocol=https;AccountName=xxxxxxxxxxxxxxx  
Since it has been working with no issues and there have been no changes to the network, we are on latest version, so I dont really understand why it suddenly failed. The stranger fact is that we replicate about 20 odd servers to the same azure blob and we got this message for 2 of the servers while the other 18 replicated fine. Might be something on the Azure side that had a hickup, so will keep an eye on it, but if it starts doing that again, will have to open a ticket also.

Gostev
SVP, Product Management
Posts: 30029
Liked: 5936 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error when offloading to azure

Post by Gostev »

No Internet service can deliver 100% availability and reliability, and Azure is not an exception... so in general, sudden failures should not come as a surprise. As long as they are rare, it's normal and not something to worry about.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests