Discussions specific to the VMware vSphere hypervisor
Post Reply
pirx
Veeam Legend
Posts: 288
Liked: 27 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Error: NFC storage connection is unavailable

Post by pirx »

Hi,

(Case 03389599)

I've a fresh Veeam 9.5 U4 test setup (no license) with one W2K16 VM running all Veeam components. I've added a NetApp CIFS Shares as backup repository which I can access from Veeam VM, creating files and rescannig of Repo is workig too. If I start a VeeamZIP backup of a VM it always fails with:

Code: Select all

30.01.2019 12:49:28 :: Error: NFC storage connection is unavailable. Storage: [stg:datastore-11,nfchost:host-17,conn:192.168.0.20]. Storage display name: [vsan-x00-edgemgmt].
Failed to create NFC download stream. NFC path: [nfc://conn:192.168.0.20,nfchost:host-17,stg:datastore-11@aa8f495c-eeab-5467-802b-48df371c7cd0/x00-jumphost-linux.vmx].
The VM is stored on a 4 node cluster vSAN datastore, but I get the same error backing up a VM on a standalone server with local raid'ed storage. 192.168.0.20 is my vCenter, vCenter and ESXi host are all running latest 6.5U2 version. Veeam settings a more or less defaults. I've found some threads about this error message, but I've problems finding the root of the problem. From Netapp I can resolve the ESXi server names.

HannesK
Veeam Software
Posts: 8069
Liked: 1428 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Error: NFC storage connection is unavailable

Post by HannesK »

Hello,
I have seen it very often and from my customer experience it was always a firewall / networking / (split)DNS issue https://www.veeam.com/kb1198

The backup server cannot connect to the ESX hosts or VCenter.

Best regards,
Hannes

pirx
Veeam Legend
Posts: 288
Liked: 27 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Error: NFC storage connection is unavailable

Post by pirx »

The setup is currently very simple, only one Veeam B&R VM.

vCenter Login is working:
[30.01.2019 16:50:40] < 4788> vim| [SOAP] Successfully logged in ( server: [192.168.0.20], user: [Administrator@vsphere.local], sessionKey: [525a3156-1789-6f95-bb44-5ca496d945d6] )
But DNS lookup of ESXi host seems to be broken.
[30.01.2019 16:50:40] < 4788> nfc| Connecting to NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. Storage: [vsan-x00-edgemgmt]. VI SOAP connection ID: [192.168.0.20].
[30.01.2019 16:50:40] < 4788> nfc| NFC service: [vpxa-nfc]. Port: [902].
[30.01.2019 16:50:40] < 4788> nfc| Opening NFC session with the specified ticket [52 f0 b7 c2 95 a8 b2 df-89 e9 90 6a eb 6d 9c a9]...
[30.01.2019 16:50:40] < 4788> nfc| Establishing connection with the host [x00-esxem-til-024.dhc-dev.internal]. Port: [902].
[30.01.2019 16:50:40] < 4788> nfc| Resolving host name (x00-esxem-til-024.dhc-dev.internal) to IP address...
[30.01.2019 16:50:52] < 4788> nfc| Resolving host name (x00-esxem-til-024.dhc-dev.internal) to IP address... Failed.
[30.01.2019 16:50:52] < 4788> nfc| Establishing connection with the host [x00-esxem-til-024.dhc-dev.internal]. Port: [902]. Failed.
[30.01.2019 16:50:52] < 4788> nfc| Opening NFC session with the specified ticket [52 f0 b7 c2 95 a8 b2 df-89 e9 90 6a eb 6d 9c a9]... Failed.
[30.01.2019 16:50:52] < 4788> nfc| Connecting to NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. Storage: [vsan-x00-edgemgmt]. VI SOAP connection ID: [192.168.0.20]. Failed.
[30.01.2019 16:50:52] < 4788> ERR |Failed to initiate NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. VI connection ID: [192.168.0.20]. Storage MOID: [vsan-x00-edgemgmt].

But nslookup for the host from the Veema VM is working:
nslookup.exe x00-esxem-til-024.dhc-dev.internal
Server: x00-pfsense.dhc-dev.internal
Address: 192.168.0.10

Name: x00-esxem-til-024.dhc-dev.internal
Address: 192.168.1.14
Same from vC:
# nslookup x00-esxem-til-024.dhc-dev.internal.
Server: 127.0.0.1
Address: 127.0.0.1#53

Name: x00-esxem-til-024.dhc-dev.internal
Address: 192.168.1.14
And from NetApp:
vserver services name-service getxxbyyy getaddrinfo -node cluster-prodit-01 -vserver svm1 -hostname x00-esxem-til-024.dhc-dev.internal
Host name: x00-esxem-til-024.dhc-dev.internal
Canonical Name: x00-esxem-til-024.dhc-dev.internal
IPv4: 192.168.1.14

So where is this DNS lookup performed?

pirx
Veeam Legend
Posts: 288
Liked: 27 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Error: NFC storage connection is unavailable

Post by pirx » 2 people like this post

Problem solved. DNS lookup on cli was working but the Veeam VM is multihomed and the interface with the private IP had the wrong DNS server setting...

HannesK
Veeam Software
Posts: 8069
Liked: 1428 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Error: NFC storage connection is unavailable

Post by HannesK »

thanks, good to hear

vmtech123
Enthusiast
Posts: 47
Liked: 24 times
Joined: Mar 28, 2019 2:01 pm
Full Name: SP
Contact:

Re: Error: NFC storage connection is unavailable

Post by vmtech123 » 2 people like this post

Just wanted to bump this and say thanks.

Had the NFC error. scanned this thread. Hostname ping failed. Was a DNS issue.

jbishir
Lurker
Posts: 1
Liked: 1 time
Joined: Nov 20, 2019 9:23 pm
Full Name: jbishir
Contact:

Re: Error: NFC storage connection is unavailable

Post by jbishir » 1 person likes this post

I wanted to add what fixed my issue in case anyone else gets this error and has tried all of the DNS and firewall stuff.

When I got this error in Veeam, I also found out that I could not open my ESXi 6.5u3's web interface in any interner browser from the Veeam server. I found that my management vmk nic's MTU didnt match the vSwitch's MTU that the vmk was assigned to. Once I updated them to match, I was able to open ESXi in my internet browser and the NFC Storage error went away in Veeam instantly. Why did the MTU's not match, not sure, but it all started when I updated my ESXi 6.5u1 server to 6.5u3.

ITGuy
Novice
Posts: 3
Liked: never
Joined: Jan 07, 2020 8:03 pm
Full Name: Eliot Sinclair
Contact:

Re: Error: NFC storage connection is unavailable

Post by ITGuy »

I had another issue which was causing this. All 3 options here were checked and correct but upon logging into vCentre I was presented with:

vmware esx cannot synchronize host "incorrect username and password"
Cannot synchronize host : Cannot complete login due to an incorrect user name or password

I have no idea how this happened but followed this from VMware and all is fixed:
https://kb.vmware.com/s/article/2014275

ottoste
Lurker
Posts: 1
Liked: never
Joined: Oct 07, 2020 3:20 pm
Full Name: Ste Otto
Contact:

Re: Error: NFC storage connection is unavailable

Post by ottoste »

pirx wrote: Jan 30, 2019 3:40 pm Hi,

(Case 03389599)

I've a fresh Veeam 9.5 U4 test setup (no license) with one W2K16 VM running all Veeam components. I've added a NetApp CIFS Shares as backup repository which I can access from Veeam VM, creating files and rescannig of Repo is workig too. If I start a VeeamZIP backup of a VM it always fails with:

Code: Select all

30.01.2019 12:49:28 :: Error: NFC storage connection is unavailable. Storage: [stg:datastore-11,nfchost:host-17,conn:192.168.0.20]. Storage display name: [vsan-x00-edgemgmt].
Failed to create NFC download stream. NFC path: [nfc://conn:192.168.0.20,nfchost:host-17,stg:datastore-11@aa8f495c-eeab-5467-802b-48df371c7cd0/x00-jumphost-linux.vmx].
The VM is stored on a 4 node cluster vSAN datastore, but I get the same error backing up a VM on a standalone server with local raid'ed storage. 192.168.0.20 is my vCenter, vCenter and ESXi host are all running latest 6.5U2 version. Veeam settings a more or less defaults. I've found some threads about this error message, but I've problems finding the root of the problem. From Netapp I can resolve the ESXi server names.
In my case I've configured Veeam 9.5 to the vSphere, but I still get the same error. Analyzing the log file it says "nfc| Establishing connection with the host [esx1]. Port: [902]. Failed.", but the firewall is not blocking this port. I suspect that Veeam needs to communicate also with the ESXI nodes and not only with vSphere. Is that correct?

HannesK
Veeam Software
Posts: 8069
Liked: 1428 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Error: NFC storage connection is unavailable

Post by HannesK »

I suspect that Veeam needs to communicate also with the ESXI nodes and not only with vSphere. Is that correct?
correct. please see https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Zew
Expert
Posts: 291
Liked: 54 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Error: NFC storage connection is unavailable

Post by Zew »

Came across this issue today. Veeam's KB: https://www.veeam.com/kb1198 was an amazing resource to troubleshooting the issue.

As other's here mine was related to DNS. For further details feel free to see my blog post on my findings. https://zewwy.ca/index.php/2021/04/16/v ... ad-stream/

Cheers, stay safe everyone!

Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests