vPower NFS Export Mount Issue after 9.5 Update 4 > FIX

[mjurado]

When I updated to 9.5 Update 4 a month ago I noticed the NFS export which gives us the Instant VM Recovery datastore in vSphere went offline. Support told me (2 different techs), oh no that's not us its a VMware problem. Spoke with VMware, ran multiple ping/vmkping tests, multiple failed attempts to recreate the datastore, firewall changes in vSphere, attempts to mount natively in ESXi (esxcfg-nas), trying different ports in my backup repository... ALL FAILED.

Yesterday I had a Veeam rep reach out to me so I revisited the issue. 3 hours later I noticed that I can mount the export from the Veeam system itself, if I use the Windows NFS client.
mount 127.0.0.1:/VeeamBackup_hostname x:

But ONLY if I use loopback, if I swap in the LAN IP it fails. Discovered registry key

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam NFS]
AllowedAddresses had a value of 127.0.0.1

Editing this and inserting LAN IPs separated by semicolons, and no spaces, AND restarting the vPower NFS service, now allows my ESXi hosts to mount the export for Instant VM Recovery.

I'm not able to find any documentation on this registry value. This should be documented in an article, but if not or until then, maybe this will help someone else.

[mjurado]

I should add above when I said ALL FAILED, I meant those steps all failed to yield the point of failure. All my ping tests worked OK

I'd love to know if this AllowedAddresses value is new in Update 4, or was something the update just reset/wiped out...

[foggy]

Hi Matthew, this looks like a side effect of the new security model that Update 4 brings to Veeam vPower NFS datastore. Starting from U4 this datastore validates the permissions of the hosts trying to connect to it. By default, access is allowed for the host which provisioned the datastore and looks like in your case Veeam B&R was unable to resolve the server address for some reason. I'd appreciate if you could share the case ID for this issue for us to investigate. Thanks!

[mjurado]

03450263, PM sent too.

[foggy]

Any chance you've initially mounted the datastore manually, not via SureBackup/IR/Linux FLR operation? This might be the reason why address resolution doesn't work.

[mjurado]

No its pretty straight forward. That datastore was mounted prior to update 4. Then it became 'inaccessible.' Because the IP of the hosts using the datastore were not in that 'AllowedAddresses' value, they could not connect. Its not a resolution issue of any kind, its just those hosts were not permitted to connect because they did not exist in that registry value.

[foggy]

What I mean is if it allows loopback, then it should allow the network address for the same server as well, and this works if the datastore was mounted by Veeam B&R itself.

[mjurado]

It should allow, but it doesn't... Until I specified the B&R server IP manually in the registry, the vPower NFS would not accept any attempts to mount the export.

[foggy]

I've already mentioned possible reason for that - the datastore was initially mounted manually before the upgrade.

[perjonsson1960]

Folks,

I have a similar, or even identical issue now... I am running B&R Update 4a, and this morning I tried to do an Instant VM Recovery, which failed miserably with the following error:

-----
2019-05-02 08:47:31 Error Failed to publish VM mivavs2 Error: Failed to mount NFS volume (172.23.8.20:/VeeamBackup_nt35.dir1.local). 172.23.8.20: Fault "PlatformConfigFaultFault", detail "NFS mount 172.23.8.20:/VeeamBackup_nt35.dir1.local failed: Unable to connect to NFS server."
An error occurred during host configuration.
-----

This is the first attempt to do a VM recovery since we upgraded to Update 4.

When I look at the registry value "AllowedAddresses" there is only 127.0.0.1 in the main Veeam server (from where I run the console and attempted the restore). In the other backup server (nt35.dir1.local in the error above), where the backup in question happens to reside, there is 127.0.0.1 plus the addresses to all our Esxi hosts separated by semicolons and no blanks. In the third server, which is our archive server for backup copies and tape backups, there is only 127.0.0.1. I don't know if B&R added the Esxi addresses now, as I attempted the restore, since I don't know how the value looked before I did my first restore attempt.

Bottom line: Is there a way to get this to work as it should, or is there a temporary workaround that we can use?

Regards,
PJ

[Dima P.]

Hello PJ,

Can you please do a brief review of the vPower NFS Troubleshooting Mounting Issues KB article and let us if it fixes your issue or not? Thank you in advance!

[perjonsson1960]

I'm going through the KB, and hit a bit of a snag... I cannot be able to find the "vPower NFS" tab, neither in the repository extent properties, nor in the scale-out properties...

[foggy]

It's the 'Mount Server' step of the repository wizard.

[foggy]

Bottom line: Is there a way to get this to work as it should, or is there a temporary workaround that we can use?

As an immediate workaround you can request a registry value that disables the new behavior introduced in U4 via support. To investigate the reasons of the error message, please also work with them directly. Thanks!

[perjonsson1960]

Okay, I will do that.
About the "vPower NFS tab", as you call it; In the Mount server step there is a check box named "Enable vPower NFS service on the mount server (recommended)". That box is checked. And there is a folder path for the NFS write cache. Apart from that, there is only the config of the listener ports.

[andreas.raden]

Hi!
Upgraded to Veeam Backup & Replication v9.5 Update 4a and our ESXi environment lost all NFS mounts. I did find a solution by editing the registry value named AllowedAddresses under HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Veeam\Veeam NFS\

Syntax is

Code: Select all

IP1;IP2;IP3;127.0.0.1

ex.

Code: Select all

10.10.10.2;10.10.10.3;10.10.10.4;10.10.10.5;127.0.0.1

Restart the service named Veeam vPower NFS Service after the change.

[perjonsson1960]

Is it important that localhost is last? Veeam itself has put it first in our list of IP addresses...

[andreas.raden]

Did Veeam sort the values you mean? If so, just place 127.0.0.1 at the very end.

After restarting the service, you can have a look in the logfile C:\ProgramData\Veeam\Backup\Svc.VeeamNFS.log and verify that the service is processing all the IP's you told it to.

[perjonsson1960]

All the IPs seem to be processed. However, if I put the localhost address 127.0.0.1 last in the AllowedAddresses value, it is moved first in the list when the service is restarted. So it seems that the list is sorted. I don't know if that if significant?

[foggy]

This KB article contains the registry value to disable access control.