V11 Storage Snapshot issue because Netapp qtree

[Fiqsz]

Hello,

First of all Case # 04746707.

We have upgraded our Veeam environment from Veeam ver.10.0.1.4854 P20201202 to ver11.0.0.837 P20210319.
There was no issue during the upgrade.

Unfortunately after the upgrade the storage snapshot started to fail with the following error message:

Code: Select all

23:30:13	Failed to create snapshot for NFS export NETAPP_VOLUME0x Details: An item with the same key has already been added. 	
23:30:25	Failed to create snapshot for NFS export NETAPP_VOLUME0x Details: An item with the same key has already been added. 	
23:30:32	Failed to create snapshot for NFS export NETAPP_VOLUME0x Details: An item with the same key has already been added. 	

The VMware Vcenter has 3 different VM Cluster configured and each VMCluster is backed up with the different job.
The NFS datastores are located on a Netapp storage, where one Netapp volume can have multiple qtree's and these qtree's are attached to the Vmware Cluster as datastores.
A specific qtree can not be attached to two different VMClusters at the same time.

The environment would be like:

Vcenter01

• VMClust01
  • Volume01 qtree01
  • Volume01 qtree02

• VMCLust02
  • Volume02 qtree01

• VMClust03
  • Volume03 qtree01
  • Volume03 qtree02
  • Volume03 qtree03
  • Volume03 qtree04

VMClust01 Backup Job
Backup Job VMClust01 starts with the storage snapshot on VMClust01 of Volume01 and it also gives an error:

Code: Select all

Failed to create snapshot for NFS export Volume01 Details: An item with the same key has already been added.

All the VM's can be backed up with Storage Snapshot if they are located on the "Volume01 qtree02" and all the VM's which are located on the "Volume01 qtree01" will be backed up with nbd, because "Failover to standard backup" option is enabled.

If "Failover to standard backup" is _not_ enabled, backup job will fail and during the 1st retry the storage snapshot will back up the VM's on the "Volume01 qtree01" without any issue and error message.

VMClust02 Backup Job
For VMClust02 and "Volume02 qtree01" the backup job will be created without any issue as there is only one qtree.

VMClust03 Backup Job:
For VMClust03 the same process as for VMClust01 and all the VM's will be backed up with standard backup, which are not located on the "Volume03 qtree04" The VM's on the "Volume03 qtree04" can be backed up with storage snapshot.
At the beginning of the job I will get 3 times this error message:

Code: Select all

23:30:13	Failed to create snapshot for NFS export Volume03 Details: An item with the same key has already been added. 	
23:30:25	Failed to create snapshot for NFS export Volume03 Details: An item with the same key has already been added. 	
23:30:32	Failed to create snapshot for NFS export Volume03 Details: An item with the same key has already been added. 	

So it looks like qtree's are the source of the problem. The job crceated a volume snapshot but somehow the job can not realize that qtree01 and qtree02 are on the same volume and no additional storage volume snapshot is required.
This behaviour is quite strange, because V10 was totally fine with this configuration.

Do you have any idea, how we could get rid of the problem? VeeamBackup DB looks like OK.
We have upgraded our environment only after the first Cumulative Patch was out, we didn't face with the issue #5 described on veeam-backup-replication-f2/top-issues- ... 72418.html.

Any help would be much appreciated.


Thanks,
Gabor

[foggy]

Hi Gabor, thanks for opening a case. There have been some changes in the volume matching logic on NetApp in v11 - if there's an issue, the case will be escalated to R&D. I will also check with the team internally and share the findings, if any.

[Fiqsz]

Hi,

Looks like the latest update Veeam V11.0.0.837 P20210507 has solved the issue. https://www.veeam.com/kb4126

Thank you,
Gabor

[foggy]

Hi Gabor, thanks for the update. Right, the issue in your case was caused by similar datastore names and this has been addressed in the latest patch.