Host-based backup of VMware vSphere VMs.
Post Reply
tkeith
Enthusiast
Posts: 32
Liked: 17 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Veeam 10 and vCloud Director 9.5 issue

Post by tkeith »

When we run through properties on our VCD server we get:

Checking connection...
Checking Certificate...
“A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

No previous issues prior to Veeam 10. Our second VCD server works fine. (although this one is 9.7)

In our test environment i tried removing and adding VCD server again and it fails with same error. So removing it isnt an option as right now jobs are still working but if we remove it and try to add it again we will break it since it wont add again.

I haven't found any firewall issues and I can browse VCD directly from the B&R server on 443 and I can also SSH from the B&R server as a test so there should be nothing blocking access between them.

We have a few tickets referencing this issue:
04073621 - My ticket
04067097 - other ticket

Does Veeam cache the connections somewhere? The certificate is valid until 2021 across all VCD cells. Not sure why the failure.
I tried adding a single cell instead of the VIP for the cluster but all cells give same error...
In one of my tickets recently i had to increase a default timeout setting on App-aware backups.
Is there a timeout that's happening before Veeam can validate all the cells?

Keith
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Andreas Neufert »

Does Veeam has access breside the vCD load balancer to the cell servers?
tkeith
Enthusiast
Posts: 32
Liked: 17 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith »

i tried adding vcd using a single cell and it results in the same error:

Code: Select all

[25.03.2020 10:40:04] <746> Error    Failed to get certificate from: https://<cell_ip>:443 (System.Exception)
[25.03.2020 10:40:04] <746> Error    Server stack trace: 
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve(String url)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Satellite.CSatelliteInfrastructureManagementBaseServiceStub.ResolveCertificate(String url)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[25.03.2020 10:40:04] <746> Error    Exception rethrown at [0]: 
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Interaction.Management.IInfrastructureManagementBaseService.ResolveCertificate(String resourceUrl)
[25.03.2020 10:40:04] <746> Error       at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Service.CInfrastructureManagementService.ResolveCertificate(String url)
[25.03.2020 10:40:04] <746> Error    Unable to connect to the remote server (System.Net.WebException)
[25.03.2020 10:40:04] <746> Error       at System.Net.HttpWebRequest.GetResponse()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.ResolveImpl()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve()
[25.03.2020 10:40:04] <746> Error    A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond (System.Net.Sockets.SocketException)
[25.03.2020 10:40:04] <746> Error       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[25.03.2020 10:40:04] <746> Error       at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
I assume its a cert issue but trying to figure it out
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Andreas Neufert »

Beside what support says, check time and date of the servers (NTP setting) as well on the Veeam side.
tkeith
Enthusiast
Posts: 32
Liked: 17 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith »

UPDATE:

Ok I think this is a bug in Veeam.

I setup a new Veeam 9.5 environment….
Adding vCloud Director 9.5. During this process it adds all the vCenters connected to this vCloud Director.

Setup goes as expected and everything is working good.

I then upgrade the Veeam 9.5 to Veeam 10.
vCloud director integration still works but you cant run through properties without getting the errors I have been sending. Same as our current production environment.
SO I figured something is changing during the Veeam 10 upgrade with regards to the vCD integration.

Next I remove vCD and try to add it again. Results in the same error and I cannot add vCD again…. I try individual nodes instead of the VIP…. Same result cannot add vCD eith the error provided.

Next I remove all the vCenters that were attached to the vCD to setup the entire infrastructure again…
With no vsphere infrastructure I proceed to add vCD again but results in the same error….

No matter what I try I cannot re-add vCD I had previously integrated prior to the upgrade.

Please let customers know that the Veeam 10 upgrade breaks vCD integration to some capacity. I’m going to assume my previous instinct that something is holding this vCD info and its preventing it from setting up new…
Is the Veeam database not cleaning up the details?

My next test is to setup a new fresh Veeam 10 instance and see if I can add vCD 9.5 and all works

Keith
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin »

Seems you've come across know issue (mentioned here). Due to changes introduced version 10 tries to reach individual vCD-cells to validate certificate - this behaviour was not there before. So, this might explain why you do not have this problem on 9.5 version.

We're currently working on a fix that will be a part of Update 1.

Kindly, keep working with the support team on this, should they confirm the issue, they will provide you with the remedy actions.

Thanks!
tkeith
Enthusiast
Posts: 32
Liked: 17 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith »

We dont use cloud connect. I don't see my specific issue mentioned here in the known issues?

Keith
tkeith
Enthusiast
Posts: 32
Liked: 17 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith »

I setup a new Veeam 10 server. Still cant add vCD 9.5. Definitely something is not right.
I can add my 9.7 vCD so somethings different between 9.5 and 9.7
Keith
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin »

I understand that you're not using Cloud Connect, but most of the known issues specific to SPs are mentioned there (including the one you seem to be affected with).

Anyway, for now it's more like speculation from our side, so, please, continue working with our support team on finding the root cause.

Thanks!
tkeith
Enthusiast
Posts: 32
Liked: 17 times
Joined: Jan 09, 2015 4:49 pm
Full Name: Keith Thiessen
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by tkeith » 1 person likes this post

Workaround performed with support assistance. It will now bypass the cert check when adding vCD and allows me to add vCloud Director again.
Contact support if you run into similar issues and they can provide the fix.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin »

Yep, support team is aware of the regkey that allows you to skip cell verification.

We're also working on more subtle solution and planning to include it in the first product update.

Thanks!
lothargr
Service Provider
Posts: 10
Liked: 3 times
Joined: Dec 06, 2010 1:23 pm
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by lothargr »

Hello,

We have the same issue with Veeam 10 build 4461 and vCloud Director 8.20.

Is there a fix available or do we have to contact support and get the regkey?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Andreas Neufert » 2 people like this post

Please upgrade to v10a to address this.
lothargr
Service Provider
Posts: 10
Liked: 3 times
Joined: Dec 06, 2010 1:23 pm
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by lothargr » 2 people like this post

Hello Andreas,

Issue is resolved with Veeam v10a, thanks for the info.
spiritie
Service Provider
Posts: 189
Liked: 40 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert
Location: Denmark
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by spiritie »

Andreas Neufert wrote: Aug 12, 2020 3:42 pm Please upgrade to v10a to address this.
FYI this hasn't been fixed for us, running latest version of Veeam (10.0.1.4854 P20201202 and vCloud 10.1

After our cells was rebooted Veeam throws an error that there is a mismatch when it comes to the cell names as is has the certificate from the load balancer in front.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Andreas Neufert » 1 person likes this post

Can you please open a ticket and upload logs there. This will allow us to check what happened. Please share ticket number here for reference.
spiritie
Service Provider
Posts: 189
Liked: 40 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert
Location: Denmark
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by spiritie »

Hi Andreas

Time flies and completely forgot about this. It happened to use again, currently we are on v11 GA.

I've created a support case: #04705516
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin »

We will investigate the issue internally, I will let you know, should we find something worthwhile. Thanks!
badabing888
Enthusiast
Posts: 83
Liked: 11 times
Joined: Jul 17, 2017 12:37 pm
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by badabing888 »

Hi,

Sorry to highjack the thread

Also getting a very similar error both in v10 and v11 GA.

My support case is 04701436

vCD added prior to v10 is fine until you go properties -> next -> next etc
It then add's the cert thumbprint of the cells to
dbo.hosts and dbo.vcd.organisations

When you then proceed to rescan it will check the public Cert and a mismatch of thumbprints will cause an issue.

Haven't tested latest v10
But is an issue in v11 RTM

Daniel
badabing888
Enthusiast
Posts: 83
Liked: 11 times
Joined: Jul 17, 2017 12:37 pm
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by badabing888 »

Hi,

To also add to that works as expected with v9.5.x builds.
So is specific to v10 and v11.

Looks like as of v10 and additional filed is added to dbo.hosts / dbo.vcd.orgs

CertificateThumbprint=""/>

If we use the SkipCertificateCheck reg key does this cause an issue in v11 as per

#4: vCloud Director jobs fail with a VDDK error
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin » 1 person likes this post

QA team is aware of both of the issues (SkipCertificateCheck and load balancer). We're planning to fix the first one in the next (second) cumulative patch, the timeframe for the for the second fix is still under discussion.

I will keep you updated.

Thanks!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Gostev »

We already released the second cumulative patch last week, so it would be third :)
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin » 1 person likes this post

Discussed with QA team, current plan is to fix the load balancer issue in the second, sorry, third cumulative patch :)
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by Gostev » 1 person likes this post

You mean in the 3rd patch that they call CP2 internally? Sorry, could not resist :D
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 10 and vCloud Director 9.5 issue

Post by veremin »

I should have said "in the next cumulative patch" instead, then the game would be over :)
Post Reply

Who is online

Users browsing this forum: No registered users and 107 guests