Comprehensive data protection for all workloads
Post Reply
rmehta
Service Provider
Posts: 73
Liked: 7 times
Joined: Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta
Contact:

SSL/TLS error since VEEAM upgrade to version 9.0

Post by rmehta » Apr 19, 2016 8:25 pm

Veeam support ID: 01764020:- we started getting this error on both backup and replication job since the upgrade to VEEAM version 9.0; update 1. The job eventually succeeds after few tries, however this is concerning as at times we have to manually intervene and the backup triggers during the production hours rather than when it is scheduled. I have logged a case with VEEAM support and honestly not very impressed with the response time.
"Creating snapshot
Error: The request was aborted: Could not create SSL/TLS secure channel."

Any insights would be valuable

Gostev
SVP, Product Management
Posts: 24797
Liked: 3557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Gostev » Apr 19, 2016 10:31 pm

The observed behavior can only be caused by intermittent certificate validation issues (which is why retries always help - eventually). Do you have a CA server on-prem (for example, a server with Active Directory Certificate Services role enabled).

rmehta
Service Provider
Posts: 73
Liked: 7 times
Joined: Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by rmehta » Apr 20, 2016 12:58 am

Yes, we have an internal CA on prem.

rmehta
Service Provider
Posts: 73
Liked: 7 times
Joined: Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by rmehta » Apr 20, 2016 2:07 am

we are using the default vmware certificate

rmehta
Service Provider
Posts: 73
Liked: 7 times
Joined: Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by rmehta » Apr 21, 2016 9:20 pm

this is the response I have got from VEEAM; it is a known issue with Windows 2008 R2, VEEAM and VMWARE...and they want me to upgrade to windows server 2012 R2

Hi Rajeev,

I've been assigned this case from escalation.

I can see that your Veeam server is a Windows 2008 R2 server. Please correct me if I'm wrong.

The issue you are seeing in related to Win 2008 R2, Veeam v9 and VMware 5.x. Veeam and Vmware are working on this issue, but currently there is no resolution.

Only workaround available is to migrate or upgrade the Veeam server to Windows 2012 or 2012 R2 as the issue is not present in those Operating Systems.

I would strongly advice at-least looking into setting up a test Veeam server on Win 2012 or 2012 R2 to confirm as this issue is related to Vmware APIs so a fix might take a long time to be available.

Regards,
Vindika Dissanayake
Veeam Software

hvdhelm
Lurker
Posts: 1
Liked: never
Joined: Apr 23, 2016 8:51 pm
Full Name: Henk van der Helm
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by hvdhelm » Apr 23, 2016 9:03 pm

More or less the same error here, but on a little different configuration. Veeam 9.0u1, running on Windows 7. VMware 5.5u2, Guest to backup Windows 7.
I only get this error on one specific guest vm.

Code: Select all

23-4-2016 22:11:30 :: Inventorying guest system 
23-4-2016 22:12:46 :: Preparing guest for hot backup 
23-4-2016 22:12:56 :: Creating snapshot 
23-4-2016 22:13:43 :: Releasing guest 
[b]23-4-2016 22:14:05 :: Error: The request was aborted: Could not create SSL/TLS secure channel. [/b]
23-4-2016 22:14:05 :: Network traffic verification detected no corrupted blocks 
23-4-2016 22:14:05 :: Processing finished with errors at 23-4-2016 22:14:05

cmaier
Enthusiast
Posts: 30
Liked: 3 times
Joined: Feb 24, 2014 4:01 pm
Full Name: Christian Maier
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by cmaier » Apr 25, 2016 9:00 am

I have an open ticket #01592206 since a long time, too. An upgrade to 2012 R2 is not an option because we would have to buy a complete new set of 2012 server CALs.

What I don't understand: We never had this issue in v8, it started instantly after upgrading to v9.

Gostev
SVP, Product Management
Posts: 24797
Liked: 3557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Gostev » Apr 25, 2016 11:57 pm

The issue was present in v8 as well, but we managed to find some workarounds and included them in v8 U1. But these were mere workarounds, and they no longer help with v9. We've had a support case open with VMware for a very long time, where able to reproduce and collect all the required debug logs for ESXi host from them. They do see the issue on their side, but there does not seem to be much progress towards the resolution.

cowhow
Enthusiast
Posts: 26
Liked: 1 time
Joined: May 17, 2013 5:01 pm
Full Name: Tony Price
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by cowhow » Apr 26, 2016 7:15 pm

Gostev wrote:The observed behavior can only be caused by intermittent certificate validation issues (which is why retries always help - eventually). Do you have a CA server on-prem (for example, a server with Active Directory Certificate Services role enabled).
Gostev, could this be caused by a misconfigured on-prem CA? I'm getting sporadic SSL errors but they are usually resolved on a subsequent retry.
MCITP/EA, VCP6.5-DCV

rmehta
Service Provider
Posts: 73
Liked: 7 times
Joined: Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by rmehta » Apr 26, 2016 8:16 pm

we use the default certificate installed with VEEAM, and yes after the job is retried the job completes, however at times it just exceeds auto-tries and we then manually retry the job which is not what we want

Gostev
SVP, Product Management
Posts: 24797
Liked: 3557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Gostev » Apr 26, 2016 10:19 pm

cowhow wrote:Gostev, could this be caused by a misconfigured on-prem CA? I'm getting sporadic SSL errors but they are usually resolved on a subsequent retry.
Not necessarily. While the issue is indeed with certificate validation, based on what I know at this time I am not inclined to blame misconfigured on-prem CA... there's still a chance of course, but most likely it is a bug in vSphere.

isolated_1
Enthusiast
Posts: 30
Liked: 5 times
Joined: Apr 09, 2015 8:33 pm
Full Name: Simon Chan
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by isolated_1 » May 16, 2016 8:25 pm

Hello All,

I am also experiencing this issue ever since upgrading to Veeam 9. We are also using vSphere ESXi 5.1 My case number is: 01773918.

After weeks of troubleshooting, I am now asked by the Veeam engineer to see if it is possible to upgrade to Windows Server 2012 R2.

Here is a snippet of what the technician found in the log files:

Code: Select all

[12.05.2016 20:38:30] <42> Warning  [Ssl] Custom certificate validation callback for vcenter.local:10443 is not defined. Accepting certificate [Subject]
[12.05.2016 20:38:30] <42> Warning    E=support@vmware.com, CN=VMware default certificate, OU=InventoryService_2012.09.18_104100, O="VMware, Inc."
[12.05.2016 20:38:30] <42> Warning  [Issuer]
[12.05.2016 20:38:30] <42> Warning    E=support@vmware.com, CN=vcenter.local, OU=InventoryService_2012.09.18_104100, O="VMware, Inc."
[12.05.2016 20:38:30] <42> Warning  [Serial Number]
[12.05.2016 20:38:30] <42> Warning    100002
[12.05.2016 20:38:30] <42> Warning  [Not Before]
[12.05.2016 20:38:30] <42> Warning    9/17/2012 10:41:36 AM
[12.05.2016 20:38:30] <42> Warning  [Not After]
[12.05.2016 20:38:30] <42> Warning    9/16/2022 10:41:47 AM
[12.05.2016 20:38:30] <42> Warning  [Thumbprint]
[12.05.2016 20:38:30] <42> Warning    
[12.05.2016 20:38:30] <42> Info     [InvSvc] Successfully logout from inventory service. StatusCode: 'OK', Status Description: 'OK'
[12.05.2016 20:38:30] <42> Error    The request was aborted: Could not create SSL/TLS secure channel. 
The VM would error out but would get processed again once the job finishes and in almost every instance, the second try would be successful. This also happens to random VMs but what I do notice is that once it does effect a given VM(s), the error would persist on mainly those VMs only.

Chris-DE
Novice
Posts: 4
Liked: 1 time
Joined: Jul 06, 2016 7:08 am
Full Name: Christian Scherwinsky
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Chris-DE » Jul 06, 2016 7:10 am

Is there any Update on this beside upgrading to Win 2012 R2?

I´m having this issue with one VM in a vSphere 5.5 Cluster Using Veeam V9 U1

SeektoKnow
Veeam ProPartner
Posts: 1
Liked: never
Joined: Apr 22, 2015 1:46 pm
Full Name: Seydou Kompaore
Location: Atlanta, Ga
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by SeektoKnow » Jul 22, 2016 1:37 pm

Not necessarily. While the issue is indeed with certificate validation, based on what I know at this time I am not inclined to blame misconfigured on-prem CA... there's still a chance of course, but most likely it is a bug in vSphere.
Well I am not sure if the issue was there or it is more related to other users that probably implemented a third party CA with vSphere/Veeam, but we never experienced the issue when using 7, 8 until we upgraded to 9. I also don't get the point when you said ...bug in vSphere. WHAT vSphere version are you referring to, and if you could be more specific here that would help. I appreciate your effort to address this matter but it is obvious that Veeam 9 did not consider certain aspect of vSphere version or so. We used the same version of vSphere 5.1 when we had Veeam 7 and 8 and not we stated to use Veeam 9 and we have the Certification issue.
Our schedule backup would not retry a failed VM but the next day backup will work. We sure have retry 3 time set but still it won't retry if the failure is SSL related. Not Sure why still.

neilmacneil
Service Provider
Posts: 56
Liked: 2 times
Joined: Mar 05, 2015 2:17 pm
Full Name: Neil MacNeil
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by neilmacneil » Aug 18, 2016 12:49 pm

Hi,

We've just started getting these failures after 9 U2. They are only occurring on our veeam server that is running 2k8R2. Also they are happening if the job is using hot-add or direct san. The 2nd retry of the backup has been working.

-Neil

modenet
Lurker
Posts: 2
Liked: 2 times
Joined: Sep 19, 2016 5:36 pm
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by modenet » Sep 19, 2016 5:52 pm 2 people like this post

RESOLVED removing last windows update:
KB3177186
KB3175024
KB3172605
KB3184122
KB3185911

sam.lowry
Novice
Posts: 6
Liked: 2 times
Joined: Dec 01, 2015 4:59 am
Full Name: Guillaume REMBRY
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by sam.lowry » Sep 21, 2016 1:20 pm 2 people like this post

Same error with new windows update KB3185278 : SSL/TLS error
Uninstall resolves the problem.
Those KB must not be installed :
KB3175024
KB3172605
KB3185278

Just installed B&R 9 updt 2 and it's ok

Seve CH
Influencer
Posts: 19
Liked: 9 times
Joined: May 09, 2016 2:34 pm
Full Name: JM Severino
Location: Switzerland
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Seve CH » Sep 27, 2016 12:42 pm

Hi.

Some info:
Since I installed KB3174644 in W2012R2, Veeam 9U2 was unable to connect to vCenter 5.5 and old servers. This patch has a different KB number for other versions of Windows.

Fired by event: VeeamNoHostConnectionEvent
Event description: Unable to connect to XXXXXXXX. Failed to download clients.xml file from https://XXXXXXXX:443/client/clients.xml. The request was aborted: Could not create SSL/TLS secure channel.
Initiated by: Veeam ONE Monitor (ZZZZZZZ)

And there were a lot of schannel errors in System event log. Source: Schannel, Event ID 36888:
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 813.

It seems that the new Diffie-Hellman patch from Microsoft changed the minimum bits supported by Windows to a higher value which blocks connecting to old servers (we still have some ESX 4.1 and 5.1). I've uninstalled the patch and everything is now working fine. It will last until weak certificates get blocked again (SHA-1 anyone?).

Regards.


joechay
Novice
Posts: 3
Liked: 1 time
Joined: Sep 15, 2016 1:09 am
Full Name: Joe Chay
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by joechay » Oct 02, 2016 9:23 pm

Not helping as I do not have the Microsoft KB update on my Veeam Server running WIndow 2008R2.

I did however look at me vCenter to regenerate certificate and so far no TLS error.

Frosty
Expert
Posts: 174
Liked: 36 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Frosty » Mar 15, 2017 10:40 pm

I'm getting this problem as well.
Our environment has to be PCI DSS compliant, so I'm not really in a position to be uninstalling security patches.
Presents in the Windows event logs as Event ID 36888 in SCHANNEL in the System log, every time a backup is run.
Can confirm we have internal PKI and server's certificate is 2048 bits, but with SHA-1.
Backup server is Windows Server 2008 R2 running VBR v9.0 U2 (build 1715).
Our ESXi hosts are v5.1 ... although I plan to upgrade them to v5.5 shortly.
Any suggestions? Can it easily be fixed, or do I just live with it?

Frosty
Expert
Posts: 174
Liked: 36 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by Frosty » Mar 21, 2017 10:13 pm 3 people like this post

Completed an upgrade to vCenter from v5.1 to v5.5 Update 3e yesterday and the SCHANNEL Event ID 36888 errors are gone. Am assuming that the vCenter self-signed certificate was updated in the process (though not sure of this) and this fixed the issue.

jim3cantos
Enthusiast
Posts: 52
Liked: 10 times
Joined: Jan 08, 2013 6:14 pm
Full Name: José Ignacio Martín Jiménez
Location: Madrid, Spain
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by jim3cantos » Apr 17, 2017 7:50 am

In our case, upgrading from vcenter v.5.5 update 3b to v5.5 update 3e didn't seem to regenerate the certificate so we forced regeneration from vcenter appliance console. After that, we got "The remote certificate is invalid" error in backup jobs. Resolved following steps from this thread but still getting SSL/TSL errors in replication jobs. Support case 02129771 opened.

jim3cantos
Enthusiast
Posts: 52
Liked: 10 times
Joined: Jan 08, 2013 6:14 pm
Full Name: José Ignacio Martín Jiménez
Location: Madrid, Spain
Contact:

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Post by jim3cantos » Apr 21, 2017 6:38 am 1 person likes this post

jim3cantos wrote:In our case, upgrading from vcenter v.5.5 update 3b to v5.5 update 3e didn't seem to regenerate the certificate so we forced regeneration from vcenter appliance console. After that, we got "The remote certificate is invalid" error in backup jobs. Resolved following steps from this thread but still getting SSL/TSL errors in replication jobs. Support case 02129771 opened.
Update: Problem solved with this last step from Veeam Support:
If your Veeam Server is on Windows 2008R2, apply the following registry value and reboot

Registry path: HKLM SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
Parameter: ClientCacheTime
Type: REG_DWORD
Value: 0

Ensure no jobs are running, restart the Veeam Backup server, and try the jobs again.

This is an old issue with Microsoft where the secure connection caching has unexpected consequences.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], oleg.feoktistov, perrylee and 53 guests