Replication error : Snapshot hierarchy is too deep

[DaFresh]

Hello,

I randomly (but regulary) got this error with replications jobs, on different Veeam instance / different site. For example, one site has 16 vm that are replicated every night, everything works fine/no error are reported by Veeam, until half of them fail with this error cause. All the replications job are done one NFS NAS, same vendor on all site.

The only way I have found to solve this at the moment is to remove the corrupted replica and let Veeam redo a full replication. I got to try this one too :

http://kb.vmware.com/selfservice/micros ... Id=1004545

But it will not solve the root cause, which I begin to think is related to the NFS NAS itself.

Anyone have any idea about that ? It will be greatly appreciated !

Thx,

Cédric

[foggy]

Cedric, could you please check how many snapshots do replica VMs have at the moment of failure? Do you see any snapshots that are not Veeam replica restore points?

The fact is that VMware allows 32 snapshots per VM maximum, while Veeam B&R replication job has maximum retention of 28 restore points. So probably some snapshots are not properly deleted or something.

[DaFresh]

Hi Foggy,

Yes I have one snapshot called "Veeam Replica Working Snapshot" at the end of the chain, then 2 restores point and finally the base image of the VM at the top.
As you guess, the job is configured to keep 3 restore points.

[foggy]

Could you also check your datastore for orphaned snapshot files that are not visible in the vSphere Client?

[DaFresh]

Yes already done :

Code: Select all

-rw-------    1 root     root        208896 Dec  7 00:57 TSEFR.ixblue.corp-000001-delta.vmdk
-rw-------    1 root     root           345 Dec  7 00:57 TSEFR.ixblue.corp-000001.vmdk
-rw-------    1 root     root         33169 Dec  3 05:20 TSEFR.ixblue.corp-Snapshot2.vmsn
-rw-------    1 root     root         33775 Dec  5 02:41 TSEFR.ixblue.corp-Snapshot4.vmsn
-rw-------    1 root     root         33777 Dec  7 00:57 TSEFR.ixblue.corp-Snapshot5.vmsn
-rw-------    1 root     root     107374182400 Dec  7 01:25 TSEFR.ixblue.corp-flat.vmdk
-rw-------    1 root     root          8684 Dec  7 00:53 TSEFR.ixblue.corp.nvram
-rw-------    1 root     root           529 Dec  7 01:25 TSEFR.ixblue.corp.vmdk
-rw-------    1 root     root          1619 Dec  7 00:57 TSEFR.ixblue.corp.vmsd
-rw-------    1 root     root          5469 Dec  7 00:57 TSEFR.ixblue.corp.vmx
-rw-------    1 root     root          3049 Dec  7 00:57 TSEFR.ixblue.corp.vmxf
-rw-------    1 root     root        106496 Dec  7 00:57 TSEFR.ixblue.corp_1-000001-delta.vmdk
-rw-------    1 root     root           349 Dec  7 00:57 TSEFR.ixblue.corp_1-000001.vmdk
-rw-------    1 root     root     53687091200 Dec  7 01:25 TSEFR.ixblue.corp_1-flat.vmdk
-rw-------    1 root     root           530 Dec  7 01:25 TSEFR.ixblue.corp_1.vmdk

[foggy]

Are there probably any related events in vSphere Client? This error comes from vSphere itself and most likely can be reproduced by creating snapshot manually, irrespective of the backup performed by Veeam B&R.

[jkaw328]

Hello,

I randomly (but regulary) got this error with replications jobs, on different Veeam instance / different site. For example, one site has 16 vm that are replicated every night, everything works fine/no error are reported by Veeam, until half of them fail with this error cause. All the replications job are done one NFS NAS, same vendor on all site.

The only way I have found to solve this at the moment is to remove the corrupted replica and let Veeam redo a full replication. I got to try this one too :

http://kb.vmware.com/selfservice/micros ... Id=1004545

But it will not solve the root cause, which I begin to think is related to the NFS NAS itself.

Anyone have any idea about that ? It will be greatly appreciated !

Thx,

Cédric

I have deleted all the snapshot but still getting the same error?

[Vitaliy S.]

Can you manually create the snapshot via vSphere Client for this VM? Did you remove these snapshot via backup console by reducing retention policy or cleared them out using vSphere Client?

[jkaw328]

Hi Vitality
I cleared them out using vsphere client. I will try creating it manually via vsphere client later

[Vitaliy S.]

Johnny, also I would recommend taking a look at related messages in VMware vSphere for this VM. Keep us updated on how it goes.

[DaFresh]

Hello,

As a side note, I still got those errors sometimes, it happens only on my replicated VM stored on NFS, and I think it is directly related to the NFS and/or NAS.

@jkaw328: what is the backend storage, NFS ?

[jkaw328]

Hello,

As a side note, I still got those errors sometimes, it happens only on my replicated VM stored on NFS, and I think it is directly related to the NFS and/or NAS.

@jkaw328: what is the backend storage, NFS ?

Hi DaFresh,
Yes vm stored on NFS but happens on both my backup and replication job.

[Vitaliy S.]

Hi Johnny,

You say you have the same issue for backup jobs? Do you have any orphaned snapshots for the source VMs? In this case I would recommend to VMware support team for further investigation.

Thanks!

[DaFresh]

As Vitaliy said, I am pretty sure too that the issue is snapshot(vMWare) and NFS related and that Veeam does not play any role in that.
I already had opened an issue at vMWare, but they gently redirect me the to my NFS vendor, which is Synology. I didn't take time to investigate more because it happens once every 3 months, and as it only concern the replicated VM I just have to clean the replica and let Veeam regenerate them. I am in the hope that the issue will be solved by itself after a upgrade of the Synology and/or the vMWare NFS stack.

[jkaw328]

Hi Johnny,

You say you have the same issue for backup jobs? Do you have any orphaned snapshots for the source VMs? In this case I would recommend to VMware support team for further investigation.

Thanks!

Hi Vitality,
how do you know if you have any orphaned snapshots. Also I try doing a snapshot in Vcentre as suggested but also got the same error message

[Vitaliy S.]

Can you please browse VM folder located on the datastore and confirm you don't have any snapshots files in this directory. Alternatively, you can use Veeam ONE and custom infrastructure report. To build the report you need to select Virtual Machine as the desired object type, and then add the required columns (such as Snapshot name, Size, Directory, VM name etc.) to the report.

If you don't have any snapshots present, then please contact VMware support team.

[jkaw328]

Hi Vitality,
how do you know if you have any orphaned snapshots. Also I try doing a snapshot in Vcentre as suggested but also got the same error message

Further to the above I just explore the datastore where the problem VM is stored and notice at least 30 vmdk files. I guest this is the problem. How to I safely delete these files.

[Vitaliy S.]

You can try to follow these instructions, should help:
http://kb.vmware.com/selfservice/micros ... Id=1002310
http://kb.vmware.com/selfservice/micros ... Id=2003638
http://kb.vmware.com/selfservice/micros ... Id=1023657

[DaFresh]

If removing orphaned snapshots didn't work (it sometimes happens whatever you do), you can try cloning the VM or do a V2V.

Anyway this will not solve the root cause of this problem which I think is related to snapshot corruption due to NFS (or any related protocol like IP and Ethernet ...) issues. I think mine was at a time caused by a misconfiguration of the STP at the ethernet layer, which make topology changes and then network micro disconnections. (Yes, my NFS NAS are on the user's LAN, I need them to be reachable by other sites for replication .... )
You should try investigate your network, it could maybe help.

Cheers