Guest File Restore Usually Fails When Restoring From NetApp Snapshots

[tthomas1@ebsco.com]

Background: we want to start backing up several VMs hosted on NetApp storage with Veeam rather than NetApp's SnapCenter tool. During testing, we discovered guest file restores using Veeam failed about 2/3 of the time.

I've had a case open with Veeam (04997061) exactly five months and have been informed that Veeam developers now agree this is a bug, but there's no way to track when the bug is patched. I was told the best solution was to post details here since Veeam Product Management regularly peruses this board.

Problem Details: NetApp lets you carve up their systems into ‘storage virtual machines’ – think of them as Virtual Machines without the machine. It’s just shared storage as opposed to storage running under a VM, but it’s the same concept. Each SVM has a root volume containing permissions (or “exports”) for access to the volumes. Because a NetApp system has at least two nodes, there is a copy of that root volume on the second node. Any time there’s a permission change, that root volume has to be replicated (or copied) to the other node before the permission change takes effect.

Here's the issue: when running the guest file restore process, Veeam successfully clones the NetApp volume needed to restore the guest files, but then immediately tries to mount the clone to the ESX hosts. The problem is that NetApp has just *started* replicating the root volume as it tries to mount the clone, so the permissions aren’t there yet for it to mount successfully. While the replication job usually finishes within Veeam’s timeout, the issue appears to be that the attempt begins before permissions are there, so it fails almost every time. On the rare occasion when it succeeds, the root volume replication ran extremely fast, like 4 seconds, and it happens to finish before the mount attempt begins. Usually the root volume replication takes 30 – 45 seconds. NetApp has run into this issue with their own backup tool, SnapCenter, and had to work with their own developers to ensure they included a timeout before making the attempt so permissions could be replicated first.

Both I and the Veeam agent wondered why this hasn’t come up before. It’s plausible that many NetApp customers simply don’t follow best practices and aren’t doing the root volume replication. Or, those that do may not also be Veeam customers. Or, some of those that do may not have tested the guest file restore process and discovered that this is a problem.

Hopefully someone in Veeam Product Management can provide updates on when this can be patched. Thank you .

[rennerstefan]

Hi Thomas,

thanks for sharing this information with us here.
Let me look into it and talk to the folks on our team about this.
My guess is that "...many NetApp customers simply don’t follow best practices and aren’t doing the root volume replication..." may be the cause why this is not coming up too often.

Let me check and update here later.
Thanks

[tthomas1@ebsco.com]

Sounds good, thank you @rennerstefan!

[foggy]

Hi Tim, we've checked with the eng team and this is indeed the first report of an issue like that so far (for whatever reason). Sounds interesting and we will try to implement similar setup to check things internally. Thanks for the heads up!

[tthomas1@ebsco.com]

Thank you @foggy that's good to hear! I hope there's an easy resolution as we'd like to implement Veeam for some of our VMs on NetApp storage, but are held up until this is resolved.

[tthomas1@ebsco.com]

Hi @foggy, I was curious if there are any updates? Has development been able to test this issue?

[foggy]

Hi Tim, no updates so far, QA have this on the list but with a background priority as this is the only report about such behavior.

[tthomas1@ebsco.com]

Okay thanks. That's unfortunate but understandable as we were hoping to use Veeam more extensively and this will prevent us from being able to do that . Hopefully someday this will be fixed and we can revisit.

[tthomas1@ebsco.com]

@foggy, for reference, here is the NetApp documentation for setting up load sharing protection of the SVM root volume.

https://docs.netapp.com/us-en/ontap/dat ... -task.html

[foggy]

Hi @tthomas1@ebsco.com, we're currently looking into this and would highly appreciate your assistance. We reproduced the situation where replication lasts longer than the restore and the restore didn't fail, likely due to the fact that the clone is mounted via /.admin folder which makes it possible to access the root volume:

When LS mirrors are available for the vsroot volume, NFSv3 operations are able to leverage the LS mirror destination volumes to traverse the filesystem. When LS mirrors are in use, it is possible to access the source volume through the .admin folder within the NFS mount.

But we have only a 2-node cluster (single ha-pair) available atm. In the case of 4- or 6-node clusters though, root volume is not available via /.admin from the second ha-pair and it might be a failure reason.

While we're trying to get access to the required configuration, it would help us a lot if we could review the logs from your environment that report the failure situation. The logs from your case have been already deleted according to retention, so I'd appreciate it if you could open a new case and provide the logs for investigation. Thanks!

[tthomas1@ebsco.com]

Hi @foggy,

I'm afraid our configuration won't be much help as we are also a two node cluster.

[foggy]

Hmmm. That's even more interesting then and your configuration can still help. Could you please check if you have the IsNeedCheckLsMirror registry value set to zero by any chance (this disables the behavior I mentioned above and could explain the failures)? Otherwise, we'd still like to review the restore and rescan logs if possible.

[tthomas1@ebsco.com]

How would I check for the registry value?

[foggy]

Just check if it is in the standard HKLM\SOFTWARE\Veeam\Veeam Backup and Replication hive. If not - the default (no value or value = 1) is to mount via /.admin.

[tthomas1@ebsco.com]

Okay. I should have thought of this when you asked initially, but I unfortunately don't even have the infrastructure for this set up anymore. No connections to NetApp storage, etc. Just doing a standard agent-based backup of the few VMs we are backing up with Veeam. So I'm afraid I won't be able to test this.

[foggy]

Btw, if you still remember the details of your setup, any chance you had the storage added to Veeam B&R via SVM (directly via SVM mgmt IP)?

[tthomas1@ebsco.com]

Hi foggy,

I'm afraid I don't definitively recall. I lean toward thinking we did SVM connection to limit connectivity to the whole cluster, but again I can't say for sure.