Cannot connect vCenter and vCloud to VBS

[PikesMilan]

This is in relation to Case #05444289

We recently had an ESXi host disconnection that hosted the VBS VM.
After recovering the host, it was noted that we cannot re-connect the VBS to both vCSA and vCD.
No network modifications were done in the environment and a snippet of the logs we found after the error prompts are shown below:

Code: Select all

[19.05.2022 02:25:17] <19> Error        [ViHostHierarchyManager:VCENTER NAME] System.Exception: Failed to login to "VCENTER NAME", port 443, user "administrator@vsphere.local", proxy srv: port:0 ---> System.Net.WebException: The operation has timed out
[19.05.2022 02:25:17] <19> Error           at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
[19.05.2022 02:25:17] <19> Error           at System.Net.WebClient.DownloadString(Uri address)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CClientConfig.DownloadXmlWithRetries(WebClient webClient, String address)
[19.05.2022 02:25:17] <19> Error        --- End of stack trace from previous location where exception was thrown ---
[19.05.2022 02:25:17] <19> Error           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CClientConfig.DownloadXmlWithRetries(WebClient webClient, String address)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CClientConfig.CreateFromHost(CServiceConnSpec connSpec)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CServiceSession.Create(CServiceConnSpec connSpec, CSpsServiceHolder spsServiceHolder, CPbmServiceHolder pbmServiceHolder, CInventoryServiceHolder invServiceHolder, CRawSoapInvokerHolder rawSoapInvokerHolder)
[19.05.2022 02:25:17] <19> Error           --- End of inner exception stack trace ---
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CServiceSession.Create(CServiceConnSpec connSpec, CSpsServiceHolder spsServiceHolder, CPbmServiceHolder pbmServiceHolder, CInventoryServiceHolder invServiceHolder, CRawSoapInvokerHolder rawSoapInvokerHolder)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CSoapService..ctor(Guid hostId, String host, CServiceConnSpec connSpec, IHierarchyContentLoader contentLoader, RemoteCertificateValidationCallback validationCallback)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CSoapConnection.Connect(Guid dbHostId, String host, Int32 port, String user, String password, Boolean useProxy, String proxyHost, Int32 proxyPort, IHierarchyContentLoader contentLoader, RemoteCertificateValidationCallback validationCallback)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.ViSoap.CSoapConnection..ctor(Guid dbHostId, String host, Int32 port, String user, String password, Boolean useProxy, String proxyHost, Int32 proxyPort, IHierarchyContentLoader contentLoader, RemoteCertificateValidationCallback validationCallback, Boolean useCache)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.Core.SSoapConnection.Create(Guid dbHostId, String srvName, CDBSoapCreds adminSoapCreds, Boolean useCache, CRemoteHierarchyContentLoader contentLoader, CPlainVssCredentials userCredsOrNull)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.Core.SSoapConnection.CreateFromBrokerService(CDBHost host, CPlainVssCredentials userCreds)
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.BrokerService.CViHostNonContinuousHierarchyManager.DoHierarchyCollectCycle()
[19.05.2022 02:25:17] <19> Error           at Veeam.Backup.BrokerService.CViHostNonContinuousHierarchyManager.DoHierarchyCollectCycleSafe()

&

Code: Select all

[19.05.2022 02:57:28] <206> Error        Failed to get certificate from: https://VCENTERNAME:443 (System.Exception)
[19.05.2022 02:57:28] <206> Error        Server stack trace:
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.Core.CCertificateResolver.Resolve()
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.Core.CCertificateResolver.Resolve(String url)
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.Satellite.CSatelliteInfrastructureManagementBaseServiceStub.ResolveCertificate(String url)
[19.05.2022 02:57:28] <206> Error           at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[19.05.2022 02:57:28] <206> Error           at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[19.05.2022 02:57:28] <206> Error        Exception rethrown at [0]:
[19.05.2022 02:57:28] <206> Error           at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[19.05.2022 02:57:28] <206> Error           at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.Interaction.Management.IInfrastructureManagementBaseService.ResolveCertificate(String resourceUrl)
[19.05.2022 02:57:28] <206> Error           at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.ServiceLib.CInfrastructureManagementService.ResolveCertificate(String url)
[19.05.2022 02:57:28] <206> Error        The operation has timed out (System.Net.WebException)
[19.05.2022 02:57:28] <206> Error           at System.Net.HttpWebRequest.GetResponse()
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.Core.CCertificateResolver.ResolveImpl()
[19.05.2022 02:57:28] <206> Error           at Veeam.Backup.Core.CCertificateResolver.Resolve()

Access to vCenter on port 443 from VBS work and it is listening on that port as well.
The cert was not changed on the server either. Both machines can ping each other and the account used to add the vCenter and vCloud to VBS is enabled and has all the required permissions.

Any ideas? I know that the logs are quite self explanatory but as already stated, no network modification was recently applied to the environment.

[Mildur]

Hi Pikes

Thanks for the case. I see, that you have opened it 20 minutes ago.
Please give our support engineer some time to analyze the logs.

And please provide the entire debug logs to the case. It will be the first question of the support engineer
https://www.veeam.com/kb1832

[jorgedlcruz]

Hello,
As Fabian mentioned, give our Engineers some time. Things to check, well the basics really:

• Can the Veeam Server, and proxies, ping over FQDN to the VMware Environment?
• Can the VMware Environment, meaning vCenter and ESXi ping over FQDN the Veeam Infrastructure?
• Host was down, perhaps some NTP issues happened. Can you make sure that both the Veeam Server, and the ESXi that failed are in NTP sync with the rest of the environment?
• On Veeam itself, under Infrastructure, have you tried to go to Properties under your VCSA, introduce user/pass, etc?
• If nothing has changed in SSL on any of the VMware environments you will save big time on not doing the lsdoctor procedure

Hope our Engineers find the issue soon.

Thanks!