Host-based backup of VMware vSphere VMs.
Post Reply
bautistalatam2025
Lurker
Posts: 2
Liked: never
Joined: May 14, 2024 8:46 pm
Full Name: José Daniel Bautista Cruz
Location: MÉXICO
Contact:

SOAP connection is not available

Post by bautistalatam2025 »

Hello, Regards to all!! :idea:
I have the following soap connection problem, it is a sporadic message, it does not always appear
I have different remote sites, all data centers point to a vCenter version 7.0.3.01700

Already validate the following tests:

Network is validated: network traffic, origin, destination, open ports
Veeam server performance: Memory, CPU
vCenter performance: cpu memory, disks
DNS resolution is validated from the veeam server and proxies
host file was edited, FQDN and vcenter IP address were added
Antivirus exclusions added
New proxy added to isolate the issue

As a temporary solution, the jobs are pointed directly to the ESXi, this way the backups no longer fail

A new server is installed and the vcenter backups are configured again, the Soap message is still displayed

In a different data center, where there was vCenter version 6.7, a new vCenter is installed, with the latest version 7.0.3.01700, 50 jobs are run, 49 jobs completed and 1 job fails and the message appears
I'm still checking how I can remove that message from the soap, if anyone has an idea


I ADD FULL ERROR MESSAGE:

5/14/2024 12:21:22 AM :: Processing hostnamevm Error: Cannot authenticate user.
Soap fault. Detail: '', endpoint: ''
SOAP connection is not available. Connection ID: [IP-VCENTER].
Failed to reconnect to storage. Storage: [stg:datastore-77994,nfchost:host-78040,conn:IP-VCENTER]. Storage display name: [VOL434].
Failed to create NFC download stream. NFC path: [nfc://conn:IP-VCENTER,nfchost:host-78040,stg:datastore-77994@HOSTNAME-VM/-000001.vmdk].
Agent failed to process method {Transfer.FileToText}.

Case #07240552
david.domask
Veeam Software
Posts: 2367
Liked: 560 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: SOAP connection is not available

Post by david.domask »

Hi bautistalatam2025, welcome to the forums.

Thank you for sharing the case number, and sorry to hear about the challenges here.

I can see recently the case was escalated to Veeam's Advanced Support Team, so please allow the Engineers some time to review the situation. I can see there has been indeed a lot of testing, and your patience is much appreciated as I can imagine this was a bit of work on your side.

At first blush it does look like just a networking issue or maybe something on the vCenter side, but it's best to let the Advanced Support team review the details on the case; I can see you have network captures included which is much appreciated.

(off-topic, but should note that based on your error there, looks like you have some machines with leftover snapshot files:

> Failed to create NFC download stream. NFC path: [nfc://conn:IP-VCENTER,nfchost:host-78040,stg:datastore-77994@HOSTNAME-VM/-000001.vmdk].

I'm not confident it's related to your issue, but just more an FYI that it might be worth checking for leftover snapshot delta files if you don't see anything in the snapshot manager.)
David Domask | Product Management: Principal Analyst
bautistalatam2025
Lurker
Posts: 2
Liked: never
Joined: May 14, 2024 8:46 pm
Full Name: José Daniel Bautista Cruz
Location: MÉXICO
Contact:

Re: SOAP connection is not available

Post by bautistalatam2025 »

after a long list of tests, It's already resolved,
The domain account had the 3 minute and 20 second delay, corresponding to the 200,000 millisecond timeout

the issue pertains specifically to the authentication functionality of the vCenter server.
occurred when using an Active Directory account, due to intermittent delays in the relay of authentication from the vCenter server to the/a domain controller

A local vmware account was configured and the problem was resolved, thanks to the veeam support team
david.domask
Veeam Software
Posts: 2367
Liked: 560 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: SOAP connection is not available

Post by david.domask »

Hi Jose,

Oh that's quite a root cause, thank you very much for sharing it. I hadn't considered that domain authentication would present like this but I can understand it. Appreciate your patience and cooperation while working with the Support Team and again for sharing the results. :)
David Domask | Product Management: Principal Analyst
TWuser
Enthusiast
Posts: 40
Liked: 8 times
Joined: Sep 07, 2021 5:37 pm
Full Name: TW
Contact:

Re: SOAP connection is not available

Post by TWuser »

Similar issue. We were using Integrated Windows Authentication, and are trying to switch to secure LDAP using domain accounts. vcenter 7.0.3, build 24026615. Veeam 12.1.

A backup job with 10 VMs seems to fail on half of them, unable to properly authenticate to vcenter.
Performing a rescan in VBR of the vCenter is similar, sometimes it works fine, often it fails to authenticate.

Have done reboots of VBR Server and vCenter. The OP switching to Local authentication vs Domain seems more like a workaround than a fix.
Did open a ticket 07351607 and will see if we find a resolution.
david.domask
Veeam Software
Posts: 2367
Liked: 560 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: SOAP connection is not available

Post by david.domask »

Hi TWuser,

Thank you for sharing the case number, let's see what Veeam Support finds in their review.

Regarding the local vs AD account, I understand your position and can agree long-term the solution is get it working with the desired account -- using administrator@vsphere.local is usually at least a reasonable isolation step to narrow the scope of issues, so while it can be a long-term solution, usually it's meant as a quick isolation test.

Thanks!
David Domask | Product Management: Principal Analyst
TWuser
Enthusiast
Posts: 40
Liked: 8 times
Joined: Sep 07, 2021 5:37 pm
Full Name: TW
Contact:

Re: SOAP connection is not available

Post by TWuser »

Fyi if anybody else has a similar issue - we tested switching the LDAP settings and pointed directly to a local domain controller rather than our domain. This did fix our issue.
We're assuming LDAP isn't smart enough to query just the local DC, and was randomly hitting DC's across the country, some of which it didn't have a route to. This explains why it would succeed on a portion every time.

Had the ticket closed, not sure if they looked at the logs and found anything obvious or not. I didn't see anything super obvious in the log files I looked at.
Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 49 guests