Error: NFC storage connection is unavailable

[pirx]

Hi,

(Case 03389599)

I've a fresh Veeam 9.5 U4 test setup (no license) with one W2K16 VM running all Veeam components. I've added a NetApp CIFS Shares as backup repository which I can access from Veeam VM, creating files and rescannig of Repo is workig too. If I start a VeeamZIP backup of a VM it always fails with:

Code: Select all

30.01.2019 12:49:28 :: Error: NFC storage connection is unavailable. Storage: [stg:datastore-11,nfchost:host-17,conn:192.168.0.20]. Storage display name: [vsan-x00-edgemgmt].
Failed to create NFC download stream. NFC path: [nfc://conn:192.168.0.20,nfchost:host-17,stg:datastore-11@aa8f495c-eeab-5467-802b-48df371c7cd0/x00-jumphost-linux.vmx].

The VM is stored on a 4 node cluster vSAN datastore, but I get the same error backing up a VM on a standalone server with local raid'ed storage. 192.168.0.20 is my vCenter, vCenter and ESXi host are all running latest 6.5U2 version. Veeam settings a more or less defaults. I've found some threads about this error message, but I've problems finding the root of the problem. From Netapp I can resolve the ESXi server names.

[HannesK]

Hello,
I have seen it very often and from my customer experience it was always a firewall / networking / (split)DNS issue https://www.veeam.com/kb1198

The backup server cannot connect to the ESX hosts or VCenter.

Best regards,
Hannes

[pirx]

The setup is currently very simple, only one Veeam B&R VM.

vCenter Login is working:

[30.01.2019 16:50:40] < 4788> vim| [SOAP] Successfully logged in ( server: [192.168.0.20], user: [Administrator@vsphere.local], sessionKey: [525a3156-1789-6f95-bb44-5ca496d945d6] )

But DNS lookup of ESXi host seems to be broken.

[30.01.2019 16:50:40] < 4788> nfc| Connecting to NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. Storage: [vsan-x00-edgemgmt]. VI SOAP connection ID: [192.168.0.20].
[30.01.2019 16:50:40] < 4788> nfc| NFC service: [vpxa-nfc]. Port: [902].
[30.01.2019 16:50:40] < 4788> nfc| Opening NFC session with the specified ticket [52 f0 b7 c2 95 a8 b2 df-89 e9 90 6a eb 6d 9c a9]...
[30.01.2019 16:50:40] < 4788> nfc| Establishing connection with the host [x00-esxem-til-024.dhc-dev.internal]. Port: [902].
[30.01.2019 16:50:40] < 4788> nfc| Resolving host name (x00-esxem-til-024.dhc-dev.internal) to IP address...
[30.01.2019 16:50:52] < 4788> nfc| Resolving host name (x00-esxem-til-024.dhc-dev.internal) to IP address... Failed.
[30.01.2019 16:50:52] < 4788> nfc| Establishing connection with the host [x00-esxem-til-024.dhc-dev.internal]. Port: [902]. Failed.
[30.01.2019 16:50:52] < 4788> nfc| Opening NFC session with the specified ticket [52 f0 b7 c2 95 a8 b2 df-89 e9 90 6a eb 6d 9c a9]... Failed.
[30.01.2019 16:50:52] < 4788> nfc| Connecting to NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. Storage: [vsan-x00-edgemgmt]. VI SOAP connection ID: [192.168.0.20]. Failed.
[30.01.2019 16:50:52] < 4788> ERR |Failed to initiate NFC session. Target host: [x00-esxem-til-024.dhc-dev.internal]. VI connection ID: [192.168.0.20]. Storage MOID: [vsan-x00-edgemgmt].

But nslookup for the host from the Veema VM is working:

nslookup.exe x00-esxem-til-024.dhc-dev.internal
Server: x00-pfsense.dhc-dev.internal
Address: 192.168.0.10

Name: x00-esxem-til-024.dhc-dev.internal
Address: 192.168.1.14

Same from vC:

# nslookup x00-esxem-til-024.dhc-dev.internal.
Server: 127.0.0.1
Address: 127.0.0.1#53

Name: x00-esxem-til-024.dhc-dev.internal
Address: 192.168.1.14

And from NetApp:

vserver services name-service getxxbyyy getaddrinfo -node cluster-prodit-01 -vserver svm1 -hostname x00-esxem-til-024.dhc-dev.internal
Host name: x00-esxem-til-024.dhc-dev.internal
Canonical Name: x00-esxem-til-024.dhc-dev.internal
IPv4: 192.168.1.14

So where is this DNS lookup performed?

[pirx]

Problem solved. DNS lookup on cli was working but the Veeam VM is multihomed and the interface with the private IP had the wrong DNS server setting...

[HannesK]

thanks, good to hear

[vmtech123]

Just wanted to bump this and say thanks.

Had the NFC error. scanned this thread. Hostname ping failed. Was a DNS issue.

[jbishir]

I wanted to add what fixed my issue in case anyone else gets this error and has tried all of the DNS and firewall stuff.

When I got this error in Veeam, I also found out that I could not open my ESXi 6.5u3's web interface in any interner browser from the Veeam server. I found that my management vmk nic's MTU didnt match the vSwitch's MTU that the vmk was assigned to. Once I updated them to match, I was able to open ESXi in my internet browser and the NFC Storage error went away in Veeam instantly. Why did the MTU's not match, not sure, but it all started when I updated my ESXi 6.5u1 server to 6.5u3.

[ITGuy]

I had another issue which was causing this. All 3 options here were checked and correct but upon logging into vCentre I was presented with:

vmware esx cannot synchronize host "incorrect username and password"
Cannot synchronize host : Cannot complete login due to an incorrect user name or password

I have no idea how this happened but followed this from VMware and all is fixed:
https://kb.vmware.com/s/article/2014275

[ottoste]

Hi,

(Case 03389599)

I've a fresh Veeam 9.5 U4 test setup (no license) with one W2K16 VM running all Veeam components. I've added a NetApp CIFS Shares as backup repository which I can access from Veeam VM, creating files and rescannig of Repo is workig too. If I start a VeeamZIP backup of a VM it always fails with:

Code: Select all

30.01.2019 12:49:28 :: Error: NFC storage connection is unavailable. Storage: [stg:datastore-11,nfchost:host-17,conn:192.168.0.20]. Storage display name: [vsan-x00-edgemgmt].
Failed to create NFC download stream. NFC path: [nfc://conn:192.168.0.20,nfchost:host-17,stg:datastore-11@aa8f495c-eeab-5467-802b-48df371c7cd0/x00-jumphost-linux.vmx].

The VM is stored on a 4 node cluster vSAN datastore, but I get the same error backing up a VM on a standalone server with local raid'ed storage. 192.168.0.20 is my vCenter, vCenter and ESXi host are all running latest 6.5U2 version. Veeam settings a more or less defaults. I've found some threads about this error message, but I've problems finding the root of the problem. From Netapp I can resolve the ESXi server names.

In my case I've configured Veeam 9.5 to the vSphere, but I still get the same error. Analyzing the log file it says "nfc| Establishing connection with the host [esx1]. Port: [902]. Failed.", but the firewall is not blocking this port. I suspect that Veeam needs to communicate also with the ESXI nodes and not only with vSphere. Is that correct?

[HannesK]

I suspect that Veeam needs to communicate also with the ESXI nodes and not only with vSphere. Is that correct?

correct. please see https://helpcenter.veeam.com/docs/backu ... ml?ver=100

[Zew]

Came across this issue today. Veeam's KB: https://www.veeam.com/kb1198 was an amazing resource to troubleshooting the issue.

As other's here mine was related to DNS. For further details feel free to see my blog post on my findings. https://zewwy.ca/index.php/2021/04/16/v ... ad-stream/

Cheers, stay safe everyone!

[da-michael]

I had the same errors. My issues wasn't DNS related, as my vcenter is connected via an IP address, but I had an unexpected power loss and my main esx box removed itself from the vcenter management. As a result, Veeam was able to reach vcenter, but vcenter wasn't able to reach esxi. Once I readded that server to inventory, the backups worked again.

[Andreas Neufert]

Hi DA Michael,

thanks for sharing. Please know that even if you add the ESXi host by IP address to the vcenter, the VMware APIs always work with the full qualified domain names.
If you register a ESXi host by IP address to the vcenter it will transfer the hostname and as well the set domain to the vcenter which is used later. Of cause Veeam has added some fallbacks for more reliable processing, but ideally DNS should work well in any backup environment if it is Veeam Servers or VMware infrastructure. Protocolls like VMware VAIO which we use for Veeam Continous Data Protection do not work at all without fully configured and working DNS.

[LickABrick]

Had issues with this, turns out we had the 'vSphere Backup NFC' disabled on the VM Kernel Adapter used for the restore/surebackup job. After enabling this option on the adapter it worked.

Test-NetConnection and DNS all worked with the option disabled.

[Andreas Neufert]

Good point, thanks for sharing. @Sturniolo FYI