SSL/TLS error since VEEAM upgrade to version 9.0

Availability for the Always-On Enterprise

SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby rmehta » Tue Apr 19, 2016 8:25 pm

Veeam support ID: 01764020:- we started getting this error on both backup and replication job since the upgrade to VEEAM version 9.0; update 1. The job eventually succeeds after few tries, however this is concerning as at times we have to manually intervene and the backup triggers during the production hours rather than when it is scheduled. I have logged a case with VEEAM support and honestly not very impressed with the response time.
"Creating snapshot
Error: The request was aborted: Could not create SSL/TLS secure channel."

Any insights would be valuable
rmehta
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Wed Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby Gostev » Tue Apr 19, 2016 10:31 pm

The observed behavior can only be caused by intermittent certificate validation issues (which is why retries always help - eventually). Do you have a CA server on-prem (for example, a server with Active Directory Certificate Services role enabled).
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby rmehta » Wed Apr 20, 2016 12:58 am

Yes, we have an internal CA on prem.
rmehta
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Wed Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby rmehta » Wed Apr 20, 2016 2:07 am

we are using the default vmware certificate
rmehta
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Wed Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby rmehta » Thu Apr 21, 2016 9:20 pm

this is the response I have got from VEEAM; it is a known issue with Windows 2008 R2, VEEAM and VMWARE...and they want me to upgrade to windows server 2012 R2

Hi Rajeev,

I've been assigned this case from escalation.

I can see that your Veeam server is a Windows 2008 R2 server. Please correct me if I'm wrong.

The issue you are seeing in related to Win 2008 R2, Veeam v9 and VMware 5.x. Veeam and Vmware are working on this issue, but currently there is no resolution.

Only workaround available is to migrate or upgrade the Veeam server to Windows 2012 or 2012 R2 as the issue is not present in those Operating Systems.

I would strongly advice at-least looking into setting up a test Veeam server on Win 2012 or 2012 R2 to confirm as this issue is related to Vmware APIs so a fix might take a long time to be available.

Regards,
Vindika Dissanayake
Veeam Software
rmehta
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Wed Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby hvdhelm » Sat Apr 23, 2016 9:03 pm

More or less the same error here, but on a little different configuration. Veeam 9.0u1, running on Windows 7. VMware 5.5u2, Guest to backup Windows 7.
I only get this error on one specific guest vm.

Code: Select all
23-4-2016 22:11:30 :: Inventorying guest system
23-4-2016 22:12:46 :: Preparing guest for hot backup
23-4-2016 22:12:56 :: Creating snapshot
23-4-2016 22:13:43 :: Releasing guest
[b]23-4-2016 22:14:05 :: Error: The request was aborted: Could not create SSL/TLS secure channel. [/b]
23-4-2016 22:14:05 :: Network traffic verification detected no corrupted blocks
23-4-2016 22:14:05 :: Processing finished with errors at 23-4-2016 22:14:05
hvdhelm
Lurker
 
Posts: 1
Liked: never
Joined: Sat Apr 23, 2016 8:51 pm
Full Name: Henk van der Helm

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby cmaier » Mon Apr 25, 2016 9:00 am

I have an open ticket #01592206 since a long time, too. An upgrade to 2012 R2 is not an option because we would have to buy a complete new set of 2012 server CALs.

What I don't understand: We never had this issue in v8, it started instantly after upgrading to v9.
cmaier
Influencer
 
Posts: 23
Liked: 1 time
Joined: Mon Feb 24, 2014 4:01 pm
Full Name: Christian Maier

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby Gostev » Mon Apr 25, 2016 11:57 pm

The issue was present in v8 as well, but we managed to find some workarounds and included them in v8 U1. But these were mere workarounds, and they no longer help with v9. We've had a support case open with VMware for a very long time, where able to reproduce and collect all the required debug logs for ESXi host from them. They do see the issue on their side, but there does not seem to be much progress towards the resolution.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby cowhow » Tue Apr 26, 2016 7:15 pm

Gostev wrote:The observed behavior can only be caused by intermittent certificate validation issues (which is why retries always help - eventually). Do you have a CA server on-prem (for example, a server with Active Directory Certificate Services role enabled).

Gostev, could this be caused by a misconfigured on-prem CA? I'm getting sporadic SSL errors but they are usually resolved on a subsequent retry.
MCITP/EA, VCP5-DCV
cowhow
Enthusiast
 
Posts: 26
Liked: 1 time
Joined: Fri May 17, 2013 5:01 pm
Full Name: Tony Price

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby rmehta » Tue Apr 26, 2016 8:16 pm

we use the default certificate installed with VEEAM, and yes after the job is retried the job completes, however at times it just exceeds auto-tries and we then manually retry the job which is not what we want
rmehta
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Wed Mar 16, 2016 8:15 pm
Full Name: Rajeev Mehta

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby Gostev » Tue Apr 26, 2016 10:19 pm

cowhow wrote:Gostev, could this be caused by a misconfigured on-prem CA? I'm getting sporadic SSL errors but they are usually resolved on a subsequent retry.

Not necessarily. While the issue is indeed with certificate validation, based on what I know at this time I am not inclined to blame misconfigured on-prem CA... there's still a chance of course, but most likely it is a bug in vSphere.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby isolated_1 » Mon May 16, 2016 8:25 pm

Hello All,

I am also experiencing this issue ever since upgrading to Veeam 9. We are also using vSphere ESXi 5.1 My case number is: 01773918.

After weeks of troubleshooting, I am now asked by the Veeam engineer to see if it is possible to upgrade to Windows Server 2012 R2.

Here is a snippet of what the technician found in the log files:

Code: Select all
[12.05.2016 20:38:30] <42> Warning  [Ssl] Custom certificate validation callback for vcenter.local:10443 is not defined. Accepting certificate [Subject]
[12.05.2016 20:38:30] <42> Warning    E=support@vmware.com, CN=VMware default certificate, OU=InventoryService_2012.09.18_104100, O="VMware, Inc."
[12.05.2016 20:38:30] <42> Warning  [Issuer]
[12.05.2016 20:38:30] <42> Warning    E=support@vmware.com, CN=vcenter.local, OU=InventoryService_2012.09.18_104100, O="VMware, Inc."
[12.05.2016 20:38:30] <42> Warning  [Serial Number]
[12.05.2016 20:38:30] <42> Warning    100002
[12.05.2016 20:38:30] <42> Warning  [Not Before]
[12.05.2016 20:38:30] <42> Warning    9/17/2012 10:41:36 AM
[12.05.2016 20:38:30] <42> Warning  [Not After]
[12.05.2016 20:38:30] <42> Warning    9/16/2022 10:41:47 AM
[12.05.2016 20:38:30] <42> Warning  [Thumbprint]
[12.05.2016 20:38:30] <42> Warning   
[12.05.2016 20:38:30] <42> Info     [InvSvc] Successfully logout from inventory service. StatusCode: 'OK', Status Description: 'OK'
[12.05.2016 20:38:30] <42> Error    The request was aborted: Could not create SSL/TLS secure channel.


The VM would error out but would get processed again once the job finishes and in almost every instance, the second try would be successful. This also happens to random VMs but what I do notice is that once it does effect a given VM(s), the error would persist on mainly those VMs only.
isolated_1
Novice
 
Posts: 8
Liked: 2 times
Joined: Thu Apr 09, 2015 8:33 pm
Full Name: Simon Chan

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby Chris-DE » Wed Jul 06, 2016 7:10 am

Is there any Update on this beside upgrading to Win 2012 R2?

I´m having this issue with one VM in a vSphere 5.5 Cluster Using Veeam V9 U1
Chris-DE
Lurker
 
Posts: 2
Liked: never
Joined: Wed Jul 06, 2016 7:08 am
Full Name: Christian Scherwinsky

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby SeektoKnow » Fri Jul 22, 2016 1:37 pm

Not necessarily. While the issue is indeed with certificate validation, based on what I know at this time I am not inclined to blame misconfigured on-prem CA... there's still a chance of course, but most likely it is a bug in vSphere.

Well I am not sure if the issue was there or it is more related to other users that probably implemented a third party CA with vSphere/Veeam, but we never experienced the issue when using 7, 8 until we upgraded to 9. I also don't get the point when you said ...bug in vSphere. WHAT vSphere version are you referring to, and if you could be more specific here that would help. I appreciate your effort to address this matter but it is obvious that Veeam 9 did not consider certain aspect of vSphere version or so. We used the same version of vSphere 5.1 when we had Veeam 7 and 8 and not we stated to use Veeam 9 and we have the Certification issue.
Our schedule backup would not retry a failed VM but the next day backup will work. We sure have retry 3 time set but still it won't retry if the failure is SSL related. Not Sure why still.
SeektoKnow
Veeam ProPartner
 
Posts: 1
Liked: never
Joined: Wed Apr 22, 2015 1:46 pm
Location: Atlanta, Ga
Full Name: Seydou Kompaore

Re: SSL/TLS error since VEEAM upgrade to version 9.0

Veeam Logoby neilmacneil » Thu Aug 18, 2016 12:49 pm

Hi,

We've just started getting these failures after 9 U2. They are only occurring on our veeam server that is running 2k8R2. Also they are happening if the job is using hot-add or direct san. The 2nd retry of the backup has been working.

-Neil
neilmacneil
Service Provider
 
Posts: 39
Liked: 1 time
Joined: Thu Mar 05, 2015 2:17 pm
Full Name: Neil MacNeil

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: jslic, maiki and 47 guests