NFS Snapshot Limitation also with native Snapshots using VAAI?

[MatzeB]

Hello,

we generally use Netapp Storage in our solutions and the storage is almost always connected via NFS.
From a Veeam perspective, we always use storage integration with backup from storage snapshots.
Unfortunately, there is an annoying limitation here (on the VMware part) as soon as the VM has one or more snapshots, Backup from Storage Snapshots no longer works and a failover to Network NBD will occur.

"The Direct NFS access mode cannot be used for VMs that have at least one snapshot"

What is not so important for individual VMs has an effect when it happens for a large number of VMs.
Recently, I have noticed that our customers are starting to use solutions that automatically trigger VMware snapshots when appling Windows patches, for example. This means, that there are constantly many VMware snapshots on a lot of VMs.

In my opinion, this is also pain due to the fact that the NBD transport mode has simply not been "fast" since some VMware versions....


For some time now, it is possible to use "native snapshots" in a Netapp environment.
This requires the VAAI Plugin 2.0 for Netapp NFS installed on the ESX host and a parameter in the VMX file or the Advanced Options of the VM "snapshot.alwaysAllowNative =true".

This change creates a file clone on netapp instead of the classic delta file for the snapshot on the storage. This is performance neutral and can also be resolved/deleted instantly.

VM Snapshot without enabled native snapshot


VM Snapshot with enabled native snapshot


So I wondered whether this mechanism would also be useful for bypassing the limitations of NFS with existing snapshots. In my test setup in the lab, I configured native snapshots, created a snapshot on the VM and made a backup. As usual, a failover to NBD takes place. Now I ask myself the question:
- Does this procedure simply not help for this scenario and the VMware limitation is still there?
OR
- Does Veeam simply only check if snapshot >= 1 then never use storage integration, failover to NBD. Does this mean that Veeam does not know about a "better" type of snapshot?


I would be very happy to receive feedback here
Matze

[rennerstefan]

Hi Matze,
I would expect that we are calling the VMware API and ask if there is a VMware Snapshot existent on the VM we try to backup. As in your case I assume there is still a Snapshot shown in the VMware snapshot manager it will fail as Veeam can't know what backend technology was used for that snapshot (VMware native or VAAI offloaded).
With that it will still not work.
@foggy please also add a comment if my thinking is right here.
Thanks
Stefan

[Andreas Neufert]

It is a limitation of the VMware developer documentation. There is no reliable/documented way to read outside of an ESXi host from a redo log snapshot. We would need to read partly from the snapshot file and partly from the base disk. So it would be very risky for us to just assume things and bring the data together. For that reason with Direct NFS and Backup from Storage Snapshot (NFS) we need to create the first VM snapshot that is present, so that we can read from the base vmdk.

The VAAI-NFS snapshot part is I think compatible with the DirectNFS part. Beside other vendors, our experience with DirectNFS and Cisco HyperFlex that uses VAAI-NFS, we learned that each of these technologies are tricky and the VAAI-NFS implementations often diverts from the standard to accommodate the storage technology, which sometimes breaks things on our side (Storage vendor only checks VMware side, not anyone else reading the data).

But the good news is, that we are listen to our customers and like 8 years ago we developed together with one customer QuickBackup and VeeamZIP + integration into vCenter for it.
QuickBackup starts an incremental individual VM backup based on the last backup. And it ages out with the backup chain over time. It uses very little space on the backup target storage and the processing takes just some minutes. It is way better than a snapshot as it is a real backup, the only downside is that it takes some minutes instead of seconds.
Veeam-ZIP with retention. Idea is if you have VMs to decommission, you can back them up once and then after x days (like a year for example), Veeam will automatically remove the backup from the backup target. This give you the chance to delete the VM from production.
The best thing is that you can use our Enterprise Manager based vCenter plug-in to make these 2 functions available to your vCenter users and remove for them the rights to create snapshots. That way you will not end up in full datastores or performance issues, because someone forgot to remove the snapshots.

[MatzeB]

@Stefan / @Andreas thanks for the fast reply.

I understand the concerns - better a slow backup than a corrupt backup. Nevertheless, the impact is not trivial, especially in large environments.

Storage vendor only checks VMware side, not anyone else reading the data

Agree with this, but I still find the approach extremely exciting and maybe this would simply be an opportunity to talk to Netapp about this, maybe this will change

Even if I may repeat things, it's simply a pain point for us because we almost always use NFS. And unfortunately, what I'm seeing in the field is that conventional VMware snapshots are becoming more rather than less. I think for different reasons, better RBAC and external patch management which triggeres VM snaps are only two reasons.

I also know and like the QuickBackup approach. Whenever possible I push it but the reality is that the vCenter plugin is not very common. In addition, there are many external applications that interact with VMware snapshots, so I can't simply use Quick Backup. And at the end of the day I'm the storage and backup guy trying to "solve" all the problems in my layer with my tools

That way you will not end up in full datastores or performance issues, because someone forgot to remove the snapshots.

Also a true statement - but with NFS VAAI integration i mentioned above this is also solved


Would be nice to hear some statement from @foggy

Regards
Matze

[Andreas Neufert]

I checked the documentation and it looks like that VAAI-NFS in general use the same snapshot redolog file structure on the VMware side. Meaning the changes go into the snapshot file. (even on IO level it is different). So when you have already an existing snapshot and we would create an additional snapshot for backup, we would need to read the data from the original vmdk and the snapshot 1 and bring them together. As I wrote before, there is no official documented way from VMware for this to just read from the file share the data. => HotAdd or NBD mode required as VMware would deliver us the needed information at a higher abstraction level that went through the VMware storage backend (not on hardware level).

[rennerstefan]

Hi all,

we did some tests with NFS that has native Snapshots enabled and the limitations still apply. As soon as there is an existing VM Snapshot, storage integrations won't work. Mentioning again that i talk about storage integrations and DirectNFS, not about VADP (NBD, Hot-add).

VM with no existing snapshot
BfSS - OK
DirectNFS - OK

VM with existing snapshots (any count of flat ones)
BfSS - not detected by our limitation
DirectNFS - not detected by our limitation

We will further research if there is way to optimize this but there is no ETA at this stage.

Thanks

[aengel]

Hi Stefan,

thank you for your insights. Yes when traditional VMware snapshots are in place then we also saw technical problems when Native Snaps come in addition.
But the idea is to have only native snaps. This should not end up in problems you pointed out!?

Of course when you have a mixed environment (example: ISCSI with VMFS and NFS with enabled native snaps) you have to take care as admin that
VMs on ISCSI with traditional snaps will not move to NFS with the snap backpack. When you take care of this szenario I dont see a good reson not to use
Native Snaps

Andreas

[Andreas Neufert]

There is a general answer to this.
If VMware writes changes into the specific snapshot files and we create an additional snapshot we would need to bring the data together beween the first snapshot file and the main VMDK.
This is a VMware undocumented situation and we do not proceed with DirectNFS backups here. (Hotadd/NBD mode still possible).

If you create snapshots on the storage that does not influence the VMware seen files, then it is independent and we would create a redolog snapshot and perform backup from the VMDK base disk with Direct NFS.

For VAAI-NFS integrated storages (where you see the storage snapshots as VMware VM snapshots, it is the same, for direct NFS there can not be any VMware Snapshots in the VMware Snapshot manager visible.

There is an integration of Veeam into Cisco HyperFlex that allows you to handle additional snapshots (we mount then the snapshots directly for backup and so we can handle them accordingly).

[aengel]

Hi Andreas,

we dont use snapshots on storage site when triggering a VMware snapshot. We create an object clone on ONTAP site. We simply clone the flat.vmdk file and
VMware is fully aware of this. (Snapshot Manager)

What kind of integration is needed that VEEAM is able to handle this.

Andreas

[Andreas Neufert]

Andreas can you please reach out to Stefan R. to discuss in a call. Thanks.

[MatzeB]

Thanks Stefan Renner / Andreas Neufert for checking this - i also played arround in our labs with this. The good thing is from my point of view, with traditional VMware snapshots OR native VAAI snapshots from storage vendor, the behaviour is the same. Integration got skipped on existing snaps, but backup data is "valid" with an alternative transport mode.
For me it would bei nice if there would be a different detection and handling for VAAI native snapshots where the limitation for additional snapshots isn't there.

[rennerstefan]

thanks for your feedback.
As mentioned above we will further research it and see if we will support it with DirectNFS and BfSS as well.

[chrisr]

There is an integration of Veeam into Cisco HyperFlex that allows you to handle additional snapshots (we mount then the snapshots directly for backup and so we can handle them accordingly).

The problem with this is that it appears to be undocumented by Cisco that a native hx snapshot needs to be present at all times to enable the the storage integrated snapshots to work, as otherwise the alwaysAllowNative attribute is changed back to false at the end of every backup job, meaning that any vcenter snaps created for patching etc are redo ones and therefore any subsequent backups fail as non-native snaps exist. The obvious solution would be to use HX Connect to take the patching snaps, but rather unhelpfully, Cisco left out the ability to schedule monthly snaps from their interface.