Failed to retreive certificate - V11

[dhayes16]

Hello All. I know V11 is no longer supported but we have a V11 install since we have a VM running Windows 2008 R2 that still needs protection until they upgrade in October (I know I am rolling the dice here). We have a linux hardened repo on prem and a SOBR pointing to s3.wasabisys.com which has been working great. On 3/24/2024 we started getting "failed to retrieve certificate from https://s3.wasabisys.com and all SOBR offloads were failing. We have had this issue pop up from time to time and went through the usual troubleshooting but we can not get this to work this time. Now if I create a new Object storage repo in V11 using another service point (s3.us-central-1.wasabisys.com) everything works fine and I do not get the certificate error which leads me to believe it is something with Wasabi itself at the s3.wasabisys.com service point. We did reach out to Wasabi and they said they are looking into it. Note that this is not firewall related since all URL's are whitelisted and packets are being allowed out based on the logs. The firewall is a Sophos XG125.

Question: Is there any way to change the service point of an existing Wasabi Object Repo? It is greyed out in properties. If not then can I add another object repo with a different service point to hit the existing bucket without having to reseed a new bucket? I

Please see below the entries from the Satellites file.?

Code: Select all

[PublicCloudCertificateLoader] Loading certificate for 'https://s3.wasabisys.com/'
[09.04.2024 22:29:21] <19> Info                 [AP] (1c90) command: 'Invoke: Network.RetrieveSslCertificate { (EString) HostName = s3.wasabisys.com; (EInt32) Port = 443; }'
[09.04.2024 22:29:25] <10> Info                   [AP] (1c90) output: <VCPCommandResult result="false" exception="Failed to retrieve SSL certificate. Underlying error: An existing connection was forcibly closed by the remote host&#x0A;Agent failed to process method {Network.RetrieveSslCertificate}." />
[09.04.2024 22:29:25] <10> Info                   [AP] (1c90) output: >
[09.04.2024 22:29:25] <19> Error        Failed to retrieve SSL certificate. Underlying error: An existing connection was forcibly closed by the remote host (Veeam.Backup.Common.CCppComponentException)
[09.04.2024 22:29:25] <19> Error        Agent failed to process method {Network.RetrieveSslCertificate}. (Veeam.Backup.Common.CCppComponentException)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[09.04.2024 22:29:25] <19> Error        --- End of stack trace from previous location where exception was thrown ---
[09.04.2024 22:29:25] <19> Error           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.RetrieveSslCertificate(String hostName, Int32 port, String& blob, Boolean& isHostnameVerified)
[09.04.2024 22:29:25] <19> Error           at Veeam.Backup.PublicCloud.CPublicCloudCertificateLoader.LoadCertificate(IPublicCloudConnectionApiUri connectionUri, String hostName, Boolean ignoreAgentTrustStatus)
[09.04.2024 22:29:25] <19> Error        Failed to check S3 service point (ServicePoint='https://s3.wasabisys.com', TrustedCertificate='18f9a652-d037-440c-9d31-a52619696087')

[chrisWasabi]

I would first ask if this something that occurs randomly, or is constant and has prevented backup/offload to take place?

FWIW: The certificates for us-east-1:s3.wasabisys.com, and us-central-1:s3.us-central-1.wasabisys.com have been in place since Sept 2023-ish, and both of them will be up for annual renewal soon.

If it's randomly...

More troubleshooting will need to be done.
It is common to see this error when a network connection becomes saturated, and packets start dropping. The TLS setup process fails because it cannot connect and download the certificate.
There are other things that could be affecting the TLS setup process as well. System settings for CRL checking could be one of them. Depending on your settings, you may checking and downloading the CRL list on every connection to Wasabi. This could be using up all your bandwidth because the CRL is ~8MB in size. Veeam will open 64 connections per task, and all of a sudden you have 64 threads downloading the 8MB CRL, assuming you only have 1 task going. Multiply this by your configured tasks in the repo / proxy.

Things to look for, I'm not sure the commands on windows to double check but this is what I would do.

Figure out your settings for CRL verification.
Figure out CRL caching, is it enabled or disabled.
Check network bandwidth of crl3/crl4.digicert.com, wireshark, fiddler, router trace, CRL is HTTP so its in plain text.

If this is a constant, then something like this will resolve the issue.

Could you ensure your firewall is not blocking the CRL endpoints? You can probably open your browser and verify the URL of both CRL endpoints on the Veeam system itself.

Full Name:
URI:http://crl3.digicert.com/DigiCertGlobal ... 0CA1-1.crl

Full Name:
URI:http://crl4.digicert.com/DigiCertGlobal ... 0CA1-1.crl

If those work, I would go on a hunch that the system is not updating the standard built in root certificates periodically.

You could probably import them manually. However, it is not the "proper" way to solve the issue.

DigiCert Global G2 TLS RSA SHA256 2020 CA1 - https://cacerts.digicert.com/DigiCertGl ... -1.crt.pem
DigiCert Global Root G2 - https://cacerts.digicert.com/DigiCertGl ... G2.crt.pem

Here is a guide how to import them from DigiCert: https://knowledge.digicert.com/solution ... -using-mmc

Heres some information from Microsoft, even though its about a different product, its directly related in speaking about working in a "disconnected" environment where CTLs are not updated, or the digicert endpoints are blocked.
https://learn.microsoft.com/en-us/troub ... -installed

If you want to go experimental, you can use a PowerShell script to update the Root CTLs: https://github.com/asheroto/Root-Certificate-Updater

[dhayes16]

Thank you very much for the detailed response to solve this issue...I really appreciate it. The issue is constant and not random. When I say constant it always fails to retrieve from "s3.wasabisys.com" no matter if I create a new repo account or not. It NEVER fails if I use a different service point at Wasabi such as "s3.us-central-1.wasabisys.com". I can login using s3.us-central-1.wasabisys.com and see all my buckets, etc.

I went through all your steps under the "constant" section and everything checked out. certutil via powershell from the VBR server properly retrieves the CRL's. I know it is not the "proper" way but I manually imported the Digicert root certificates using your digicert instructions and it still does help.

I suppose what I do not understand is why it works with a different wasabi service point. And the issue started out of the blue on 3/24/2024 with no changes to the on-prem environment. This leads me to believe that something happened upstream at Wasabi for the s3.wasabisys.com service point. I will say that we have 20+ other customers using s3.wasabisys.com as the service point with the same on-prem Sophos firewall with no issues at all. However, they are all running Veeam 12.1.

Review of the firewall raw traffic logs shows all traffic being allowed to s3.wasabisys.com.

I am really at a loss here. I have not heard back from Wasabi on this issue. I can likely open a ticket with Veeam but it is V11 so....

Again, thanks for your attempts to help.
Dave

[mjr.epicfail]

maybe its a CRL problem.
Did you check if the CRL URLis reachable from the server?
Could also be some firewall IDP/IDS that is meddling with the traffic.

[dhayes16]

Thanks for the reply. We did test the CRL retrieval process via certutil from the server and they retrieve OK. We also review the packet logs going out to s3.wasabisys.com and all traffic is allowed out. IP's associated with that address are not getting tagged by the firewall in any of the logs.

Much appreciate the reply

[mjr.epicfail]

sorry, I skimmed the solution provided earlier *busted*.

Did you check with Veeam support?

[dhayes16]

Thanks...I can reach out to Veeam on this but it is V11 and I know it is technically no longer supported..

I may try to just use another Wasabi service point since the texas one works fine but I do not want to reseed 2TB of data into a different repo. One of my initial questions was is it possible to edit the service point of an existing repo some other way than via the standard GUI? That option is greyed out.

[chrisWasabi]

Can you log into the Wasabi Console (https://console.wasabisys.com) and browse the buckets in question in us-east-1?
You should not have to switch regions.

The Wasabi Console uses a JS AWS SDK to access the buckets in the same manner as you would from an application. If you can browse the bucket and view data on us-east-1, then the system itself will work for connectivity to the bucket, and all the certificate stuff will be good.

If you view the network traffic, for example, in Chrome DevTools, you will see the specific API calls while browsing the bucket.

PM me your Wasabi Ticket, please.

[mjr.epicfail]

Hi Dave,

What do you mean by V11 is no longer supported?
Veeam support still support V11, its always N -1

[karsten123]

v11 EOS end of february. check the lifecycle policy

[dhayes16]

Can you log into the Wasabi Console (https://console.wasabisys.com) and browse the buckets in question in us-east-1?
You should not have to switch regions.

The Wasabi Console uses a JS AWS SDK to access the buckets in the same manner as you would from an application. If you can browse the bucket and view data on us-east-1, then the system itself will work for connectivity to the bucket, and all the certificate stuff will be good.

If you view the network traffic, for example, in Chrome DevTools, you will see the specific API calls while browsing the bucket.

PM me your Wasabi Ticket, please.

Thank you...I logged into the console as you indicated and was greeting with a "Network Error". Then when we try to access the buckets we receive the following:

Access Denied

If this error is unexpected, please contact your administrator regarding:

Network Failure

So we are going to dig in further this weekend on this but you definitely gave me something to go on...

So very much appreciated!!

[dhayes16]

Can you log into the Wasabi Console (https://console.wasabisys.com) and browse the buckets in question in us-east-1?
You should not have to switch regions.

The Wasabi Console uses a JS AWS SDK to access the buckets in the same manner as you would from an application. If you can browse the bucket and view data on us-east-1, then the system itself will work for connectivity to the bucket, and all the certificate stuff will be good.

If you view the network traffic, for example, in Chrome DevTools, you will see the specific API calls while browsing the bucket.

PM me your Wasabi Ticket, please.

HUGE THANKS to chrisWasabi!! I hope this can help someone else in the future. @chrisWasabi suggested that I try to go to https://console.wasabisys.com from the on-prem server and it failed as per my previous post so we started looking internally again. The firewall we have is a Sophos XG125 (v20 firmware). We checked out the firewall access rules for default outbound traffic and noticed under the section "Filtering common web ports" the following:

The "Use Proxy instead of DPI Engine" was ENABLED. We disabled it and our problem disappeared.

Again THANK YOU for your help in resolving this issue. Made my weekend.
Dave

[chrisWasabi]

This is great news, thanks for keeping us updated!

Can you tell us if this is something that happened automatically like part of an update, or automated trigger?

I would expect in proxy mode that the traffic is also successful. Was it a misplaced firewall rule on the Proxy Mode?

Very interesting!

[dhayes16]

Hello...Thanks...
So this was working all along for 15 months with no issues at all. Then on 3/24/2024 it started happening. So the only thing I can think of is that a pattern update from Sophos hit the firewall as part of their subscription for the web proxy scanning that caused the issue. Technically Sophos wants people to use the DPI engine anyway.

I would be happy to provide any more information that might be helpful for others as well.

[mjr.epicfail]

Its always something with DPI if you get strange connection errors out of the blue (ok almost always )

[RubinCompServ]

v11 EOS end of february. check the lifecycle policy

So they're no longer supporting N-1, or are they considering 12.1 to be a new major version, making v12 the N-1?

[karsten123]

I think the n-1 thing is backwards compatibility, but not sure.
100% sure is, that a VBR version is eos 1 year after the release of the next GA.

[RubinCompServ]

Looks like EoS is 3 years after release. I'm still gun-shy about moving from 11a to 12 because, when I tried it in a small environment, it lost 25% of my backup data. The files were still present, the jobs were still present, but Veeam refused to see them and also refused to continue running the backup jobs (even an Active Full wouldn't work). The only solution was to recreate the affected backup jobs and kick off new Fulls. That was an inconvenience in my internal environment where there were 40 jobs, but would be a catastrophe in my customer environment where there are more than 500 jobs.

[karsten123]

3 years is a better policy. you are right