Host-based backup of VMware vSphere VMs.
Post Reply
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Error: NFC storage connection is unavailable

Post by pirx »

Hi,

(Case 03389599)

I've a fresh Veeam 9.5 U4 test setup (no license) with one W2K16 VM running all Veeam components. I've added a NetApp CIFS Shares as backup repository which I can access from Veeam VM, creating files and rescannig of Repo is workig too. If I start a VeeamZIP backup of a VM it always fails with:

Code: Select all

30.01.2019 12:49:28 :: Error: NFC storage connection is unavailable. Storage: [stg:datastore-11,nfchost:host-17,conn:192.168.0.20]. Storage display name: [vsan-x00-edgemgmt].
Failed to create NFC download stream. NFC path: [nfc://conn:192.168.0.20,nfchost:host-17,stg:datastore-11@aa8f495c-eeab-5467-802b-48df371c7cd0/x00-jumphost-linux.vmx].
The VM is stored on a 4 node cluster vSAN datastore, but I get the same error backing up a VM on a standalone server with local raid'ed storage. 192.168.0.20 is my vCenter, vCenter and ESXi host are all running latest 6.5U2 version. Veeam settings a more or less defaults. I've found some threads about this error message, but I've problems finding the root of the problem. From Netapp I can resolve the ESXi server names.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Error: NFC storage connection is unavailable

Post by HannesK »

Hello,
I have seen it very often and from my customer experience it was always a firewall / networking / (split)DNS issue https://www.veeam.com/kb1198

The backup server cannot connect to the ESX hosts or VCenter.

Best regards,
Hannes
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Error: NFC storage connection is unavailable

Post by pirx »

The setup is currently very simple, only one Veeam B&R VM.

vCenter Login is working:
[30.01.2019 16:50:40] < 4788> vim| [SOAP] Successfully logged in ( server: [192.168.0.20], user: [Administrator@vsphere.local], sessionKey: [525a3156-1789-6f95-bb44-5ca496d945d6] )
But DNS lookup of ESXi host seems to be broken.
[30.01.2019 16:50:40] < 4788> nfc| Connecting to NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. Storage: [vsan-x00-edgemgmt]. VI SOAP connection ID: [192.168.0.20].
[30.01.2019 16:50:40] < 4788> nfc| NFC service: [vpxa-nfc]. Port: [902].
[30.01.2019 16:50:40] < 4788> nfc| Opening NFC session with the specified ticket [52 f0 b7 c2 95 a8 b2 df-89 e9 90 6a eb 6d 9c a9]...
[30.01.2019 16:50:40] < 4788> nfc| Establishing connection with the host [x00-esxem-til-024.dhc-dev.internal]. Port: [902].
[30.01.2019 16:50:40] < 4788> nfc| Resolving host name (x00-esxem-til-024.dhc-dev.internal) to IP address...
[30.01.2019 16:50:52] < 4788> nfc| Resolving host name (x00-esxem-til-024.dhc-dev.internal) to IP address... Failed.
[30.01.2019 16:50:52] < 4788> nfc| Establishing connection with the host [x00-esxem-til-024.dhc-dev.internal]. Port: [902]. Failed.
[30.01.2019 16:50:52] < 4788> nfc| Opening NFC session with the specified ticket [52 f0 b7 c2 95 a8 b2 df-89 e9 90 6a eb 6d 9c a9]... Failed.
[30.01.2019 16:50:52] < 4788> nfc| Connecting to NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. Storage: [vsan-x00-edgemgmt]. VI SOAP connection ID: [192.168.0.20]. Failed.
[30.01.2019 16:50:52] < 4788> ERR |Failed to initiate NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. VI connection ID: [192.168.0.20]. Storage MOID: [vsan-x00-edgemgmt].

But nslookup for the host from the Veema VM is working:
nslookup.exe x00-esxem-til-024.dhc-dev.internal
Server: x00-pfsense.dhc-dev.internal
Address: 192.168.0.10

Name: x00-esxem-til-024.dhc-dev.internal
Address: 192.168.1.14
Same from vC:
# nslookup x00-esxem-til-024.dhc-dev.internal.
Server: 127.0.0.1
Address: 127.0.0.1#53

Name: x00-esxem-til-024.dhc-dev.internal
Address: 192.168.1.14
And from NetApp:
vserver services name-service getxxbyyy getaddrinfo -node cluster-prodit-01 -vserver svm1 -hostname x00-esxem-til-024.dhc-dev.internal
Host name: x00-esxem-til-024.dhc-dev.internal
Canonical Name: x00-esxem-til-024.dhc-dev.internal
IPv4: 192.168.1.14

So where is this DNS lookup performed?
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Error: NFC storage connection is unavailable

Post by pirx » 2 people like this post

Problem solved. DNS lookup on cli was working but the Veeam VM is multihomed and the interface with the private IP had the wrong DNS server setting...
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Error: NFC storage connection is unavailable

Post by HannesK »

thanks, good to hear
vmtech123
Veeam Legend
Posts: 235
Liked: 133 times
Joined: Mar 28, 2019 2:01 pm
Full Name: SP
Contact:

Re: Error: NFC storage connection is unavailable

Post by vmtech123 » 2 people like this post

Just wanted to bump this and say thanks.

Had the NFC error. scanned this thread. Hostname ping failed. Was a DNS issue.
jbishir
Lurker
Posts: 1
Liked: 1 time
Joined: Nov 20, 2019 9:23 pm
Full Name: jbishir
Contact:

Re: Error: NFC storage connection is unavailable

Post by jbishir » 1 person likes this post

I wanted to add what fixed my issue in case anyone else gets this error and has tried all of the DNS and firewall stuff.

When I got this error in Veeam, I also found out that I could not open my ESXi 6.5u3's web interface in any interner browser from the Veeam server. I found that my management vmk nic's MTU didnt match the vSwitch's MTU that the vmk was assigned to. Once I updated them to match, I was able to open ESXi in my internet browser and the NFC Storage error went away in Veeam instantly. Why did the MTU's not match, not sure, but it all started when I updated my ESXi 6.5u1 server to 6.5u3.
ITGuy
Novice
Posts: 3
Liked: never
Joined: Jan 07, 2020 8:03 pm
Full Name: Eliot Sinclair
Contact:

Re: Error: NFC storage connection is unavailable

Post by ITGuy »

I had another issue which was causing this. All 3 options here were checked and correct but upon logging into vCentre I was presented with:

vmware esx cannot synchronize host "incorrect username and password"
Cannot synchronize host : Cannot complete login due to an incorrect user name or password

I have no idea how this happened but followed this from VMware and all is fixed:
https://kb.vmware.com/s/article/2014275
ottoste
Lurker
Posts: 1
Liked: never
Joined: Oct 07, 2020 3:20 pm
Full Name: Ste Otto
Contact:

Re: Error: NFC storage connection is unavailable

Post by ottoste »

pirx wrote: Jan 30, 2019 3:40 pm Hi,

(Case 03389599)

I've a fresh Veeam 9.5 U4 test setup (no license) with one W2K16 VM running all Veeam components. I've added a NetApp CIFS Shares as backup repository which I can access from Veeam VM, creating files and rescannig of Repo is workig too. If I start a VeeamZIP backup of a VM it always fails with:

Code: Select all

30.01.2019 12:49:28 :: Error: NFC storage connection is unavailable. Storage: [stg:datastore-11,nfchost:host-17,conn:192.168.0.20]. Storage display name: [vsan-x00-edgemgmt].
Failed to create NFC download stream. NFC path: [nfc://conn:192.168.0.20,nfchost:host-17,stg:datastore-11@aa8f495c-eeab-5467-802b-48df371c7cd0/x00-jumphost-linux.vmx].
The VM is stored on a 4 node cluster vSAN datastore, but I get the same error backing up a VM on a standalone server with local raid'ed storage. 192.168.0.20 is my vCenter, vCenter and ESXi host are all running latest 6.5U2 version. Veeam settings a more or less defaults. I've found some threads about this error message, but I've problems finding the root of the problem. From Netapp I can resolve the ESXi server names.
In my case I've configured Veeam 9.5 to the vSphere, but I still get the same error. Analyzing the log file it says "nfc| Establishing connection with the host [esx1]. Port: [902]. Failed.", but the firewall is not blocking this port. I suspect that Veeam needs to communicate also with the ESXI nodes and not only with vSphere. Is that correct?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Error: NFC storage connection is unavailable

Post by HannesK »

I suspect that Veeam needs to communicate also with the ESXI nodes and not only with vSphere. Is that correct?
correct. please see https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Error: NFC storage connection is unavailable

Post by Zew »

Came across this issue today. Veeam's KB: https://www.veeam.com/kb1198 was an amazing resource to troubleshooting the issue.

As other's here mine was related to DNS. For further details feel free to see my blog post on my findings. https://zewwy.ca/index.php/2021/04/16/v ... ad-stream/

Cheers, stay safe everyone!
da-michael
Lurker
Posts: 1
Liked: never
Joined: Jun 29, 2021 2:21 pm
Full Name: DA Michael
Contact:

Re: Error: NFC storage connection is unavailable

Post by da-michael »

I had the same errors. My issues wasn't DNS related, as my vcenter is connected via an IP address, but I had an unexpected power loss and my main esx box removed itself from the vcenter management. As a result, Veeam was able to reach vcenter, but vcenter wasn't able to reach esxi. Once I readded that server to inventory, the backups worked again.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Error: NFC storage connection is unavailable

Post by Andreas Neufert »

Hi DA Michael,

thanks for sharing. Please know that even if you add the ESXi host by IP address to the vcenter, the VMware APIs always work with the full qualified domain names.
If you register a ESXi host by IP address to the vcenter it will transfer the hostname and as well the set domain to the vcenter which is used later. Of cause Veeam has added some fallbacks for more reliable processing, but ideally DNS should work well in any backup environment if it is Veeam Servers or VMware infrastructure. Protocolls like VMware VAIO which we use for Veeam Continous Data Protection do not work at all without fully configured and working DNS.
LickABrick
Enthusiast
Posts: 58
Liked: 30 times
Joined: Dec 23, 2019 7:26 pm
Full Name: Lick A Brick
Contact:

Re: Error: NFC storage connection is unavailable

Post by LickABrick »

Had issues with this, turns out we had the 'vSphere Backup NFC' disabled on the VM Kernel Adapter used for the restore/surebackup job. After enabling this option on the adapter it worked.

Test-NetConnection and DNS all worked with the option disabled.
Post Reply

Who is online

Users browsing this forum: No registered users and 68 guests