Discussions specific to the VMware vSphere hypervisor
Post Reply
tkeith
Service Provider
Posts: 26
Liked: 10 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » Mar 25, 2020 3:56 pm

When we run through properties on our VCD server we get:

Checking connection...
Checking Certificate...
“A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

No previous issues prior to Veeam 10. Our second VCD server works fine. (although this one is 9.7)

In our test environment i tried removing and adding VCD server again and it fails with same error. So removing it isnt an option as right now jobs are still working but if we remove it and try to add it again we will break it since it wont add again.

I haven't found any firewall issues and I can browse VCD directly from the B&R server on 443 and I can also SSH from the B&R server as a test so there should be nothing blocking access between them.

We have a few tickets referencing this issue:
04073621 - My ticket
04067097 - other ticket

Does Veeam cache the connections somewhere? The certificate is valid until 2021 across all VCD cells. Not sure why the failure.
I tried adding a single cell instead of the VIP for the cluster but all cells give same error...
In one of my tickets recently i had to increase a default timeout setting on App-aware backups.
Is there a timeout that's happening before Veeam can validate all the cells?

Keith

Andreas Neufert
Veeam Software
Posts: 4149
Liked: 764 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Andreas Neufert » Mar 25, 2020 7:30 pm

Does Veeam has access breside the vCD load balancer to the cell servers?

tkeith
Service Provider
Posts: 26
Liked: 10 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » Mar 25, 2020 8:47 pm

i tried adding vcd using a single cell and it results in the same error:

Code: Select all

[25.03.2020 10:40:04] <746> Error    Failed to get certificate from: https://<cell_ip>:443 (System.Exception)
[25.03.2020 10:40:04] <746> Error    Server stack trace: 
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve(String url)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Satellite.CSatelliteInfrastructureManagementBaseServiceStub.ResolveCertificate(String url)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[25.03.2020 10:40:04] <746> Error    Exception rethrown at [0]: 
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Interaction.Management.IInfrastructureManagementBaseService.ResolveCertificate(String resourceUrl)
[25.03.2020 10:40:04] <746> Error       at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Service.CInfrastructureManagementService.ResolveCertificate(String url)
[25.03.2020 10:40:04] <746> Error    Unable to connect to the remote server (System.Net.WebException)
[25.03.2020 10:40:04] <746> Error       at System.Net.HttpWebRequest.GetResponse()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.ResolveImpl()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve()
[25.03.2020 10:40:04] <746> Error    A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond (System.Net.Sockets.SocketException)
[25.03.2020 10:40:04] <746> Error       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[25.03.2020 10:40:04] <746> Error       at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
I assume its a cert issue but trying to figure it out

Andreas Neufert
Veeam Software
Posts: 4149
Liked: 764 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Andreas Neufert » Mar 26, 2020 12:54 pm

Beside what support says, check time and date of the servers (NTP setting) as well on the Veeam side.

tkeith
Service Provider
Posts: 26
Liked: 10 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » Mar 26, 2020 6:43 pm

UPDATE:

Ok I think this is a bug in Veeam.

I setup a new Veeam 9.5 environment….
Adding vCloud Director 9.5. During this process it adds all the vCenters connected to this vCloud Director.

Setup goes as expected and everything is working good.

I then upgrade the Veeam 9.5 to Veeam 10.
vCloud director integration still works but you cant run through properties without getting the errors I have been sending. Same as our current production environment.
SO I figured something is changing during the Veeam 10 upgrade with regards to the vCD integration.

Next I remove vCD and try to add it again. Results in the same error and I cannot add vCD again…. I try individual nodes instead of the VIP…. Same result cannot add vCD eith the error provided.

Next I remove all the vCenters that were attached to the vCD to setup the entire infrastructure again…
With no vsphere infrastructure I proceed to add vCD again but results in the same error….

No matter what I try I cannot re-add vCD I had previously integrated prior to the upgrade.

Please let customers know that the Veeam 10 upgrade breaks vCD integration to some capacity. I’m going to assume my previous instinct that something is holding this vCD info and its preventing it from setting up new…
Is the Veeam database not cleaning up the details?

My next test is to setup a new fresh Veeam 10 instance and see if I can add vCD 9.5 and all works

Keith

veremin
Product Manager
Posts: 17454
Liked: 1553 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin » Mar 26, 2020 7:52 pm

Seems you've come across know issue (mentioned here). Due to changes introduced version 10 tries to reach individual vCD-cells to validate certificate - this behaviour was not there before. So, this might explain why you do not have this problem on 9.5 version.

We're currently working on a fix that will be a part of Update 1.

Kindly, keep working with the support team on this, should they confirm the issue, they will provide you with the remedy actions.

Thanks!

tkeith
Service Provider
Posts: 26
Liked: 10 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » Mar 26, 2020 9:44 pm

We dont use cloud connect. I don't see my specific issue mentioned here in the known issues?

Keith

tkeith
Service Provider
Posts: 26
Liked: 10 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » Mar 26, 2020 9:51 pm

I setup a new Veeam 10 server. Still cant add vCD 9.5. Definitely something is not right.
I can add my 9.7 vCD so somethings different between 9.5 and 9.7
Keith

veremin
Product Manager
Posts: 17454
Liked: 1553 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin » Mar 27, 2020 6:29 pm

I understand that you're not using Cloud Connect, but most of the known issues specific to SPs are mentioned there (including the one you seem to be affected with).

Anyway, for now it's more like speculation from our side, so, please, continue working with our support team on finding the root cause.

Thanks!

tkeith
Service Provider
Posts: 26
Liked: 10 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » Mar 27, 2020 8:26 pm 1 person likes this post

Workaround performed with support assistance. It will now bypass the cert check when adding vCD and allows me to add vCloud Director again.
Contact support if you run into similar issues and they can provide the fix.

veremin
Product Manager
Posts: 17454
Liked: 1553 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin » Mar 28, 2020 11:37 am

Yep, support team is aware of the regkey that allows you to skip cell verification.

We're also working on more subtle solution and planning to include it in the first product update.

Thanks!

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 19 guests