SSL/TLS error since VEEAM upgrade to version 9.0

[rmehta]

Veeam support ID: 01764020:- we started getting this error on both backup and replication job since the upgrade to VEEAM version 9.0; update 1. The job eventually succeeds after few tries, however this is concerning as at times we have to manually intervene and the backup triggers during the production hours rather than when it is scheduled. I have logged a case with VEEAM support and honestly not very impressed with the response time.
"Creating snapshot
Error: The request was aborted: Could not create SSL/TLS secure channel."

Any insights would be valuable

[Gostev]

The observed behavior can only be caused by intermittent certificate validation issues (which is why retries always help - eventually). Do you have a CA server on-prem (for example, a server with Active Directory Certificate Services role enabled).

[rmehta]

Yes, we have an internal CA on prem.

[rmehta]

we are using the default vmware certificate

[rmehta]

this is the response I have got from VEEAM; it is a known issue with Windows 2008 R2, VEEAM and VMWARE...and they want me to upgrade to windows server 2012 R2

Hi Rajeev,

I've been assigned this case from escalation.

I can see that your Veeam server is a Windows 2008 R2 server. Please correct me if I'm wrong.

The issue you are seeing in related to Win 2008 R2, Veeam v9 and VMware 5.x. Veeam and Vmware are working on this issue, but currently there is no resolution.

Only workaround available is to migrate or upgrade the Veeam server to Windows 2012 or 2012 R2 as the issue is not present in those Operating Systems.

I would strongly advice at-least looking into setting up a test Veeam server on Win 2012 or 2012 R2 to confirm as this issue is related to Vmware APIs so a fix might take a long time to be available.

Regards,
Vindika Dissanayake
Veeam Software

[hvdhelm]

More or less the same error here, but on a little different configuration. Veeam 9.0u1, running on Windows 7. VMware 5.5u2, Guest to backup Windows 7.
I only get this error on one specific guest vm.

Code: Select all

23-4-2016 22:11:30 :: Inventorying guest system 
23-4-2016 22:12:46 :: Preparing guest for hot backup 
23-4-2016 22:12:56 :: Creating snapshot 
23-4-2016 22:13:43 :: Releasing guest 
[b]23-4-2016 22:14:05 :: Error: The request was aborted: Could not create SSL/TLS secure channel. [/b]
23-4-2016 22:14:05 :: Network traffic verification detected no corrupted blocks 
23-4-2016 22:14:05 :: Processing finished with errors at 23-4-2016 22:14:05

[cmaier]

I have an open ticket #01592206 since a long time, too. An upgrade to 2012 R2 is not an option because we would have to buy a complete new set of 2012 server CALs.

What I don't understand: We never had this issue in v8, it started instantly after upgrading to v9.

[Gostev]

The issue was present in v8 as well, but we managed to find some workarounds and included them in v8 U1. But these were mere workarounds, and they no longer help with v9. We've had a support case open with VMware for a very long time, where able to reproduce and collect all the required debug logs for ESXi host from them. They do see the issue on their side, but there does not seem to be much progress towards the resolution.

[cowhow]

The observed behavior can only be caused by intermittent certificate validation issues (which is why retries always help - eventually). Do you have a CA server on-prem (for example, a server with Active Directory Certificate Services role enabled).

Gostev, could this be caused by a misconfigured on-prem CA? I'm getting sporadic SSL errors but they are usually resolved on a subsequent retry.

[rmehta]

we use the default certificate installed with VEEAM, and yes after the job is retried the job completes, however at times it just exceeds auto-tries and we then manually retry the job which is not what we want

[Gostev]

Gostev, could this be caused by a misconfigured on-prem CA? I'm getting sporadic SSL errors but they are usually resolved on a subsequent retry.

Not necessarily. While the issue is indeed with certificate validation, based on what I know at this time I am not inclined to blame misconfigured on-prem CA... there's still a chance of course, but most likely it is a bug in vSphere.

[isolated_1]

Hello All,

I am also experiencing this issue ever since upgrading to Veeam 9. We are also using vSphere ESXi 5.1 My case number is: 01773918.

After weeks of troubleshooting, I am now asked by the Veeam engineer to see if it is possible to upgrade to Windows Server 2012 R2.

Here is a snippet of what the technician found in the log files:

Code: Select all

[12.05.2016 20:38:30] <42> Warning  [Ssl] Custom certificate validation callback for vcenter.local:10443 is not defined. Accepting certificate [Subject]
[12.05.2016 20:38:30] <42> Warning    E=support@vmware.com, CN=VMware default certificate, OU=InventoryService_2012.09.18_104100, O="VMware, Inc."
[12.05.2016 20:38:30] <42> Warning  [Issuer]
[12.05.2016 20:38:30] <42> Warning    E=support@vmware.com, CN=vcenter.local, OU=InventoryService_2012.09.18_104100, O="VMware, Inc."
[12.05.2016 20:38:30] <42> Warning  [Serial Number]
[12.05.2016 20:38:30] <42> Warning    100002
[12.05.2016 20:38:30] <42> Warning  [Not Before]
[12.05.2016 20:38:30] <42> Warning    9/17/2012 10:41:36 AM
[12.05.2016 20:38:30] <42> Warning  [Not After]
[12.05.2016 20:38:30] <42> Warning    9/16/2022 10:41:47 AM
[12.05.2016 20:38:30] <42> Warning  [Thumbprint]
[12.05.2016 20:38:30] <42> Warning    
[12.05.2016 20:38:30] <42> Info     [InvSvc] Successfully logout from inventory service. StatusCode: 'OK', Status Description: 'OK'
[12.05.2016 20:38:30] <42> Error    The request was aborted: Could not create SSL/TLS secure channel. 

The VM would error out but would get processed again once the job finishes and in almost every instance, the second try would be successful. This also happens to random VMs but what I do notice is that once it does effect a given VM(s), the error would persist on mainly those VMs only.

[Chris-DE]

Is there any Update on this beside upgrading to Win 2012 R2?

I´m having this issue with one VM in a vSphere 5.5 Cluster Using Veeam V9 U1

[SeektoKnow]

Not necessarily. While the issue is indeed with certificate validation, based on what I know at this time I am not inclined to blame misconfigured on-prem CA... there's still a chance of course, but most likely it is a bug in vSphere.

Well I am not sure if the issue was there or it is more related to other users that probably implemented a third party CA with vSphere/Veeam, but we never experienced the issue when using 7, 8 until we upgraded to 9. I also don't get the point when you said ...bug in vSphere. WHAT vSphere version are you referring to, and if you could be more specific here that would help. I appreciate your effort to address this matter but it is obvious that Veeam 9 did not consider certain aspect of vSphere version or so. We used the same version of vSphere 5.1 when we had Veeam 7 and 8 and not we stated to use Veeam 9 and we have the Certification issue.
Our schedule backup would not retry a failed VM but the next day backup will work. We sure have retry 3 time set but still it won't retry if the failure is SSL related. Not Sure why still.

[neilmacneil]

Hi,

We've just started getting these failures after 9 U2. They are only occurring on our veeam server that is running 2k8R2. Also they are happening if the job is using hot-add or direct san. The 2nd retry of the backup has been working.

-Neil

[modenet]

RESOLVED removing last windows update:
KB3177186
KB3175024
KB3172605
KB3184122
KB3185911

[sam.lowry]

Same error with new windows update KB3185278 : SSL/TLS error
Uninstall resolves the problem.
Those KB must not be installed :
KB3175024
KB3172605
KB3185278

Just installed B&R 9 updt 2 and it's ok

[Seve CH]

Hi.

Some info:
Since I installed KB3174644 in W2012R2, Veeam 9U2 was unable to connect to vCenter 5.5 and old servers. This patch has a different KB number for other versions of Windows.

Fired by event: VeeamNoHostConnectionEvent
Event description: Unable to connect to XXXXXXXX. Failed to download clients.xml file from https://XXXXXXXX:443/client/clients.xml. The request was aborted: Could not create SSL/TLS secure channel.
Initiated by: Veeam ONE Monitor (ZZZZZZZ)

And there were a lot of schannel errors in System event log. Source: Schannel, Event ID 36888:
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 813.

It seems that the new Diffie-Hellman patch from Microsoft changed the minimum bits supported by Windows to a higher value which blocks connecting to old servers (we still have some ESX 4.1 and 5.1). I've uninstalled the patch and everything is now working fine. It will last until weak certificates get blocked again (SHA-1 anyone?).

Regards.

[alanbolte]

KB: "The request was aborted: Could not create SSL/TLS secure channel" after installing September 2016 Windows Updates

[joechay]

Not helping as I do not have the Microsoft KB update on my Veeam Server running WIndow 2008R2.

I did however look at me vCenter to regenerate certificate and so far no TLS error.

[Frosty]

I'm getting this problem as well.
Our environment has to be PCI DSS compliant, so I'm not really in a position to be uninstalling security patches.
Presents in the Windows event logs as Event ID 36888 in SCHANNEL in the System log, every time a backup is run.
Can confirm we have internal PKI and server's certificate is 2048 bits, but with SHA-1.
Backup server is Windows Server 2008 R2 running VBR v9.0 U2 (build 1715).
Our ESXi hosts are v5.1 ... although I plan to upgrade them to v5.5 shortly.
Any suggestions? Can it easily be fixed, or do I just live with it?

[Frosty]

Completed an upgrade to vCenter from v5.1 to v5.5 Update 3e yesterday and the SCHANNEL Event ID 36888 errors are gone. Am assuming that the vCenter self-signed certificate was updated in the process (though not sure of this) and this fixed the issue.

[jim3cantos]

In our case, upgrading from vcenter v.5.5 update 3b to v5.5 update 3e didn't seem to regenerate the certificate so we forced regeneration from vcenter appliance console. After that, we got "The remote certificate is invalid" error in backup jobs. Resolved following steps from this thread but still getting SSL/TSL errors in replication jobs. Support case 02129771 opened.

[jim3cantos]

In our case, upgrading from vcenter v.5.5 update 3b to v5.5 update 3e didn't seem to regenerate the certificate so we forced regeneration from vcenter appliance console. After that, we got "The remote certificate is invalid" error in backup jobs. Resolved following steps from this thread but still getting SSL/TSL errors in replication jobs. Support case 02129771 opened.

Update: Problem solved with this last step from Veeam Support:

If your Veeam Server is on Windows 2008R2, apply the following registry value and reboot

Registry path: HKLM SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
Parameter: ClientCacheTime
Type: REG_DWORD
Value: 0

Ensure no jobs are running, restart the Veeam Backup server, and try the jobs again.

This is an old issue with Microsoft where the secure connection caching has unexpected consequences.

[Moebius]

I, too, was getting this intermittent error:
"Processing <VM name> Error: The request was aborted: Could not create SSL/TLS secure channel."
The first retry always succeeded.

I followed the previous post advise and created the reg key HKLM SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\ClientCacheTime (DWORD, Value= 0), which solved the problem.