Veeam 10 and vCloud Director 9.5 issue

[tkeith]

When we run through properties on our VCD server we get:

Checking connection...
Checking Certificate...
“A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

No previous issues prior to Veeam 10. Our second VCD server works fine. (although this one is 9.7)

In our test environment i tried removing and adding VCD server again and it fails with same error. So removing it isnt an option as right now jobs are still working but if we remove it and try to add it again we will break it since it wont add again.

I haven't found any firewall issues and I can browse VCD directly from the B&R server on 443 and I can also SSH from the B&R server as a test so there should be nothing blocking access between them.

We have a few tickets referencing this issue:
04073621 - My ticket
04067097 - other ticket

Does Veeam cache the connections somewhere? The certificate is valid until 2021 across all VCD cells. Not sure why the failure.
I tried adding a single cell instead of the VIP for the cluster but all cells give same error...
In one of my tickets recently i had to increase a default timeout setting on App-aware backups.
Is there a timeout that's happening before Veeam can validate all the cells?

Keith

[Andreas Neufert]

Does Veeam has access breside the vCD load balancer to the cell servers?

[tkeith]

i tried adding vcd using a single cell and it results in the same error:

Code: Select all

[25.03.2020 10:40:04] <746> Error    Failed to get certificate from: https://<cell_ip>:443 (System.Exception)
[25.03.2020 10:40:04] <746> Error    Server stack trace: 
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve(String url)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Satellite.CSatelliteInfrastructureManagementBaseServiceStub.ResolveCertificate(String url)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[25.03.2020 10:40:04] <746> Error    Exception rethrown at [0]: 
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[25.03.2020 10:40:04] <746> Error       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Interaction.Management.IInfrastructureManagementBaseService.ResolveCertificate(String resourceUrl)
[25.03.2020 10:40:04] <746> Error       at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Service.CInfrastructureManagementService.ResolveCertificate(String url)
[25.03.2020 10:40:04] <746> Error    Unable to connect to the remote server (System.Net.WebException)
[25.03.2020 10:40:04] <746> Error       at System.Net.HttpWebRequest.GetResponse()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.ResolveImpl()
[25.03.2020 10:40:04] <746> Error       at Veeam.Backup.Core.CCertificateResolver.Resolve()
[25.03.2020 10:40:04] <746> Error    A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond (System.Net.Sockets.SocketException)
[25.03.2020 10:40:04] <746> Error       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[25.03.2020 10:40:04] <746> Error       at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)

I assume its a cert issue but trying to figure it out

[Andreas Neufert]

Beside what support says, check time and date of the servers (NTP setting) as well on the Veeam side.

[tkeith]

UPDATE:

Ok I think this is a bug in Veeam.

I setup a new Veeam 9.5 environment….
Adding vCloud Director 9.5. During this process it adds all the vCenters connected to this vCloud Director.

Setup goes as expected and everything is working good.

I then upgrade the Veeam 9.5 to Veeam 10.
vCloud director integration still works but you cant run through properties without getting the errors I have been sending. Same as our current production environment.
SO I figured something is changing during the Veeam 10 upgrade with regards to the vCD integration.

Next I remove vCD and try to add it again. Results in the same error and I cannot add vCD again…. I try individual nodes instead of the VIP…. Same result cannot add vCD eith the error provided.

Next I remove all the vCenters that were attached to the vCD to setup the entire infrastructure again…
With no vsphere infrastructure I proceed to add vCD again but results in the same error….

No matter what I try I cannot re-add vCD I had previously integrated prior to the upgrade.

Please let customers know that the Veeam 10 upgrade breaks vCD integration to some capacity. I’m going to assume my previous instinct that something is holding this vCD info and its preventing it from setting up new…
Is the Veeam database not cleaning up the details?

My next test is to setup a new fresh Veeam 10 instance and see if I can add vCD 9.5 and all works

Keith

[veremin]

Seems you've come across know issue (mentioned here). Due to changes introduced version 10 tries to reach individual vCD-cells to validate certificate - this behaviour was not there before. So, this might explain why you do not have this problem on 9.5 version.

We're currently working on a fix that will be a part of Update 1.

Kindly, keep working with the support team on this, should they confirm the issue, they will provide you with the remedy actions.

Thanks!

[tkeith]

We dont use cloud connect. I don't see my specific issue mentioned here in the known issues?

Keith

[tkeith]

I setup a new Veeam 10 server. Still cant add vCD 9.5. Definitely something is not right.
I can add my 9.7 vCD so somethings different between 9.5 and 9.7
Keith

[veremin]

I understand that you're not using Cloud Connect, but most of the known issues specific to SPs are mentioned there (including the one you seem to be affected with).

Anyway, for now it's more like speculation from our side, so, please, continue working with our support team on finding the root cause.

Thanks!

[tkeith]

Workaround performed with support assistance. It will now bypass the cert check when adding vCD and allows me to add vCloud Director again.
Contact support if you run into similar issues and they can provide the fix.

[veremin]

Yep, support team is aware of the regkey that allows you to skip cell verification.

We're also working on more subtle solution and planning to include it in the first product update.

Thanks!

[lothargr]

Hello,

We have the same issue with Veeam 10 build 4461 and vCloud Director 8.20.

Is there a fix available or do we have to contact support and get the regkey?

[Andreas Neufert]

Please upgrade to v10a to address this.

[lothargr]

Hello Andreas,

Issue is resolved with Veeam v10a, thanks for the info.

[spiritie]

Please upgrade to v10a to address this.

FYI this hasn't been fixed for us, running latest version of Veeam (10.0.1.4854 P20201202 and vCloud 10.1

After our cells was rebooted Veeam throws an error that there is a mismatch when it comes to the cell names as is has the certificate from the load balancer in front.

[Andreas Neufert]

Can you please open a ticket and upload logs there. This will allow us to check what happened. Please share ticket number here for reference.

[spiritie]

Hi Andreas

Time flies and completely forgot about this. It happened to use again, currently we are on v11 GA.

I've created a support case: #04705516

[veremin]

We will investigate the issue internally, I will let you know, should we find something worthwhile. Thanks!

[badabing888]

Hi,

Sorry to highjack the thread

Also getting a very similar error both in v10 and v11 GA.

My support case is 04701436

vCD added prior to v10 is fine until you go properties -> next -> next etc
It then add's the cert thumbprint of the cells to
dbo.hosts and dbo.vcd.organisations

When you then proceed to rescan it will check the public Cert and a mismatch of thumbprints will cause an issue.

Haven't tested latest v10
But is an issue in v11 RTM

Daniel

[badabing888]

Hi,

To also add to that works as expected with v9.5.x builds.
So is specific to v10 and v11.

Looks like as of v10 and additional filed is added to dbo.hosts / dbo.vcd.orgs

CertificateThumbprint=""/>

If we use the SkipCertificateCheck reg key does this cause an issue in v11 as per

#4: vCloud Director jobs fail with a VDDK error

[veremin]

QA team is aware of both of the issues (SkipCertificateCheck and load balancer). We're planning to fix the first one in the next (second) cumulative patch, the timeframe for the for the second fix is still under discussion.

I will keep you updated.

Thanks!

[Gostev]

We already released the second cumulative patch last week, so it would be third

[veremin]

Discussed with QA team, current plan is to fix the load balancer issue in the second, sorry, third cumulative patch

[Gostev]

You mean in the 3rd patch that they call CP2 internally? Sorry, could not resist

[veremin]

I should have said "in the next cumulative patch" instead, then the game would be over