Failed to login to vCenter Server by SOAP, port 443

[roberi]

Hi

I have a Veeam 6.5 that's been working very good for a few weeks since I upgraded to VMware 5.1.
But this night I have an error...
"Failed to login to vCenter Server by SOAP, port 443, user " Domain\administrator", proxy srv; port 0. Cannot complete login due to an incorrect user name or password"

Half of the jobs ran OK but the other half failed. I cannot edit the jobs... same error. I cannot edit the server... same error.
I tried to add the server with DNS-name this time but I get the same error.

Patch 3 is installed.
I can login with the same user on the vCenter and the Veeam server so account is OK.

Anyone have any idea??

Regards
Robban

[Dima P.]

Hello Robban,

Possibly, your vCenter database is on the SQL Server Express Edition, which has a 4 GB of data limitation. If the vCenter database is full, every new transaction is written to cache, as a result when the cache is full, the database service will just stop. The cache gets cleared after a reboot of the vCenter Server, so vCenter starts to work until the cache is exhausted again.

Could you please double check that your vCenter database is not over 4 Gigs?
Thank you.

[roberi]

Hi

Its only on 1.8GB, and its a Express 2008R2 so the limit is 10GB I think?!

Thank you.
br
Robban

[Dima P.]

Robban, thank you for clarifying!

Any chance you changed account permissions/or user account group? Any changes with SSO server itself?

Also please double check this solution:

I'm adding this here as it may help someone else in the forum. In our case, we found that one ESXi host Time config was 7 minutes behind the other 2 hosts even though we have all 3 hosts synching with the NTP Server. Probable cause was the the Software upgrade a couple days earlier. Any way, after 3 nights of backup failures on the 1st backup run, I synched all the hosts to the NTP server, on the next sheduled nightly backup run we finally got successful backups on the scheduled time and not on the retries.
Hope this helps someone!

[roberi]

Well... last night replication was OK...
hmmm I don't know why.. but it seems to be okey now... I will see if it happens again tonight.

Thank you.

//Robban

[veremin]

Its only on 1.8GB, and its a Express 2008R2 so the limit is 10GB I think?!

Yep, as far as I know, the database size limit has been increased for SQL Server 2008 R2 Express. So, the Maximum database size in SQL Server 2008 R2 is 10 GB.

As to the issue itself, I glad to hear that it went away. Though, keep us updated how things will be going in your environment. Thanks.

[roberi]

Hi again...

Now my second Cluster starts acting almost the same. Also a 5.1.
Veeam reports bad user or password... I can only login localy to vCenter on the vCenter server... with remote vCenter clients I get the same error. Its been working OK for a few weeks also.
It seems to be vCenter related then...
I will investigate further...

Regards
Robban

[Dima P.]

Robban, please check time sync between servers, as it was recommended above. Thank you - keep us posted!

[roberi]

Hi

Sorry I missed that. However the time is OK! NTP server is set and OK.

Maybe tis problem will go away over night like the last one... dosent feel good this is happening tho.
There is a lot of time/login-time "problem" with VMware 5.1 and SSO.. maybe that is in play here to?!

regards
Robban

[roberi]

Problem remains...
I have two Veeam installations that are working agains this VMware 5.1 cluster with the same error, "Failed to login to vCenter Server by SOAP, port 443"

Anyone have any idea?
I will open a case with VMware and Veeam.

Regards
Robban

[veremin]

I will open a case with VMware and Veeam.

The best idea, indeed, since sometimes it’s rather hard to make speculations regarding particular case without seeing logs and/or virtual environment. Thanks.

[roberi]

I think I fixed it....
We had a trusted domain removed a few days ago… I thought that could not be tribute to my problem because Veeam was not involved with any of the related servers. But the SSO on VMware 5.1 had that domain listed in its list of domains and reported it dead. I removed it from the list and now Veeam seems to run OK.

I used this page... http://kb.vmware.com/selfservice/micros ... Id=2033880
(http://www.virtuallyghetto.com/2012/10/ ... ntity.html)

<SSO Server Directory>\utils\ssocli configure-riat -a discover-is -u admin
Then
List all to get the ID
rsautil manage-identity-sources -a list
Then
rsautil manage-identity-sources -a delete

Thank you for your help!

regards
Robban

[dries.vergote]

Hi

I have a similare problem. We are on v9.5 update3.
Sincce we upgraded our vcenter to version 6.5 appliance we get this error.

Code: Select all

25/01/2018 16:59:54 :: Getting VM info from vSphere  
25/01/2018 17:21:38 :: Error: Cannot get service content.
Soap fault. TimeoutDetail: 'connect failed in tcp_connect()', endpoint: 'https://vcenter:443/sdk'
SOAP connection is not available. Connection ID: [vcenter].
Failed to create NFC download stream. NFC path: [nfc://conn:vcenter,nfchost:host-828998,stg:server.vmx].

Anyone has any idea?
If you go to infrastructure and go through the vcenter connection there is no problem.
Only when running a job(s). We get this error on several different job to the same vcenter.

Other jobs to the same vcenter doesn't have a problem.
browse to the vcenter from veeam server is also no problem.

thx in advance

[IHeartCats]

We have just hit this after we veeam 9.5 u3 after a vsphere upgrade earlier in the week.... It seems to have affected one of our vcentre environments but not the other..

[dirch]

We have almost the same error, with the difference being "upload stream" instead of "download stream".
Setup Veeam 9.5 u3 with the target vCenter on 6.0 update 3b.
We want to use it for replication purposes and connections seem ok, but replication jobs fail:

Code: Select all

Error    Processing <source virtual server>
Error    Text2File failed (...)
(9 lines of Veeam.backup.core stuff)
Error    Cannot get service content. (...)
Error    Soap fault. TimeoutDetail: 'connect failed in tcp_connect()', endpoint: 'https://target.vCenter.IP:433/sdk'
Error    SOAP connection is not available. Connection ID: [target.vCenter.IP].
Error    Failed to create NFC upload stream. NFC path: [nfc://conn:target.vCenter.IP,nfchost:(...)
Error       in c++: Failed to create target file [nfc://conn:c,nfchost:(...)
Error       in c++: Failed to copy local file. (...)
Error       in c++: Unable to set content of the file. File path: [nfc://conn:target.vCenter.IP,(...)
Error       in c++: Client failed to process the command. Command: [text2file].

[dirch]

Issue fixed. It seems the target proxy need to communication with the target vCenter.
After we allowed for this traffic, the error disappeared and we could replicate from source to target vCenter.
The "downstream" text must be describing traffic on the source side and "upstream" on the target side.
Furthermore we added all relevant servers to Veeam Backup and Proxies hosts-files also - vCenters, ESXi-hosts, Backup, Proxies etc and checked traffic on IP and servernames with telnet on port 443.

[sams2]

I had a same issue. It helped me disable the Lockdown Mode (vsphere client: esxi server, configure, security profile, Lockdown Mode, disabled)

[pep]

I have a similare problem. We are on v9.5 update3.
Sincce we upgraded our vcenter to version 6.5 appliance we get this error.

Hey dries.vergote,
we have exactly the same problem. How did you solve it?

thx in advance

[foggy]

The most effective way of addressing this issue is to contact technical support directly with full logs ready for investigation. Thanks!

[zciklacekic]

This problem, if especially contains following lines:
---------------------------------------------------------------------------
Error: Cannot get service content.
Soap fault. TimeoutDetail: 'connect failed in tcp_connect()
---------------------------------------------------------------------------
is %90 percent related with network connection problem between any of following.
* If Job type is backup job:
**Backup Management server(VBR) to vCenter or ESXi
**Backup Proxy to vCenter (if managed server is added as vCenter)
**Backup Proxy to Repository
**Backup Proxy to ESX (if managed server is added as ESXi and not as vCenter)
*If job type is replication job:
**Source Backup Proxy to source vCenter (or ESXi if standalone ESXi is added)
**Source Backup Management server(VBR) to target vCenter (or ESXi if standalone ESXi is added)
**Source Backup Proxy to Target Backup Proxy
**Target Backup Proxy to target vCenter (or ESXi if standalone ESXi is added)
Refer to "Used Ports" part of User Guide. Guide link is added here for VBR v11: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[PetrM]

Hi Zafer,

Speaking about the mentioned error, I would narrow down the scope of possible reasons to a connectivity issue between any proxy or Veeam server and VMware server, for example vCenter or ESXi host. SOAP is used to access vSphere objects or fetch some info from inventory but not used for connections between proxies and repositories.

Thanks!