Comprehensive data protection for all workloads
Post Reply
DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Replication error : Snapshot hierarchy is too deep

Post by DaFresh »

Hello,

I randomly (but regulary) got this error with replications jobs, on different Veeam instance / different site. For example, one site has 16 vm that are replicated every night, everything works fine/no error are reported by Veeam, until half of them fail with this error cause. All the replications job are done one NFS NAS, same vendor on all site.

The only way I have found to solve this at the moment is to remove the corrupted replica and let Veeam redo a full replication. I got to try this one too :

http://kb.vmware.com/selfservice/micros ... Id=1004545

But it will not solve the root cause, which I begin to think is related to the NFS NAS itself.

Anyone have any idea about that ? It will be greatly appreciated !

Thx,

Cédric
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by foggy »

Cedric, could you please check how many snapshots do replica VMs have at the moment of failure? Do you see any snapshots that are not Veeam replica restore points?

The fact is that VMware allows 32 snapshots per VM maximum, while Veeam B&R replication job has maximum retention of 28 restore points. So probably some snapshots are not properly deleted or something.
DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by DaFresh »

Hi Foggy,

Yes I have one snapshot called "Veeam Replica Working Snapshot" at the end of the chain, then 2 restores point and finally the base image of the VM at the top.
As you guess, the job is configured to keep 3 restore points.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by foggy »

Could you also check your datastore for orphaned snapshot files that are not visible in the vSphere Client?
DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by DaFresh »

Yes already done :

Code: Select all


-rw-------    1 root     root        208896 Dec  7 00:57 TSEFR.ixblue.corp-000001-delta.vmdk
-rw-------    1 root     root           345 Dec  7 00:57 TSEFR.ixblue.corp-000001.vmdk
-rw-------    1 root     root         33169 Dec  3 05:20 TSEFR.ixblue.corp-Snapshot2.vmsn
-rw-------    1 root     root         33775 Dec  5 02:41 TSEFR.ixblue.corp-Snapshot4.vmsn
-rw-------    1 root     root         33777 Dec  7 00:57 TSEFR.ixblue.corp-Snapshot5.vmsn
-rw-------    1 root     root     107374182400 Dec  7 01:25 TSEFR.ixblue.corp-flat.vmdk
-rw-------    1 root     root          8684 Dec  7 00:53 TSEFR.ixblue.corp.nvram
-rw-------    1 root     root           529 Dec  7 01:25 TSEFR.ixblue.corp.vmdk
-rw-------    1 root     root          1619 Dec  7 00:57 TSEFR.ixblue.corp.vmsd
-rw-------    1 root     root          5469 Dec  7 00:57 TSEFR.ixblue.corp.vmx
-rw-------    1 root     root          3049 Dec  7 00:57 TSEFR.ixblue.corp.vmxf
-rw-------    1 root     root        106496 Dec  7 00:57 TSEFR.ixblue.corp_1-000001-delta.vmdk
-rw-------    1 root     root           349 Dec  7 00:57 TSEFR.ixblue.corp_1-000001.vmdk
-rw-------    1 root     root     53687091200 Dec  7 01:25 TSEFR.ixblue.corp_1-flat.vmdk
-rw-------    1 root     root           530 Dec  7 01:25 TSEFR.ixblue.corp_1.vmdk
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by foggy »

Are there probably any related events in vSphere Client? This error comes from vSphere itself and most likely can be reproduced by creating snapshot manually, irrespective of the backup performed by Veeam B&R.
jkaw328
Novice
Posts: 5
Liked: never
Joined: Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by jkaw328 »

DaFresh wrote:Hello,

I randomly (but regulary) got this error with replications jobs, on different Veeam instance / different site. For example, one site has 16 vm that are replicated every night, everything works fine/no error are reported by Veeam, until half of them fail with this error cause. All the replications job are done one NFS NAS, same vendor on all site.

The only way I have found to solve this at the moment is to remove the corrupted replica and let Veeam redo a full replication. I got to try this one too :

http://kb.vmware.com/selfservice/micros ... Id=1004545

But it will not solve the root cause, which I begin to think is related to the NFS NAS itself.

Anyone have any idea about that ? It will be greatly appreciated !

Thx,

Cédric
I have deleted all the snapshot but still getting the same error?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by Vitaliy S. »

Can you manually create the snapshot via vSphere Client for this VM? Did you remove these snapshot via backup console by reducing retention policy or cleared them out using vSphere Client?
jkaw328
Novice
Posts: 5
Liked: never
Joined: Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by jkaw328 »

Hi Vitality
I cleared them out using vsphere client. I will try creating it manually via vsphere client later
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by Vitaliy S. »

Johnny, also I would recommend taking a look at related messages in VMware vSphere for this VM. Keep us updated on how it goes.
DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by DaFresh »

Hello,

As a side note, I still got those errors sometimes, it happens only on my replicated VM stored on NFS, and I think it is directly related to the NFS and/or NAS.

@jkaw328: what is the backend storage, NFS ?
jkaw328
Novice
Posts: 5
Liked: never
Joined: Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by jkaw328 »

DaFresh wrote:Hello,

As a side note, I still got those errors sometimes, it happens only on my replicated VM stored on NFS, and I think it is directly related to the NFS and/or NAS.

@jkaw328: what is the backend storage, NFS ?
Hi DaFresh,
Yes vm stored on NFS but happens on both my backup and replication job.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by Vitaliy S. »

Hi Johnny,

You say you have the same issue for backup jobs? Do you have any orphaned snapshots for the source VMs? In this case I would recommend to VMware support team for further investigation.

Thanks!
DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by DaFresh »

As Vitaliy said, I am pretty sure too that the issue is snapshot(vMWare) and NFS related and that Veeam does not play any role in that.
I already had opened an issue at vMWare, but they gently redirect me the to my NFS vendor, which is Synology. I didn't take time to investigate more because it happens once every 3 months, and as it only concern the replicated VM I just have to clean the replica and let Veeam regenerate them. I am in the hope that the issue will be solved by itself after a upgrade of the Synology and/or the vMWare NFS stack.
jkaw328
Novice
Posts: 5
Liked: never
Joined: Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by jkaw328 »

Vitaliy S. wrote:Hi Johnny,

You say you have the same issue for backup jobs? Do you have any orphaned snapshots for the source VMs? In this case I would recommend to VMware support team for further investigation.

Thanks!
Hi Vitality,
how do you know if you have any orphaned snapshots. Also I try doing a snapshot in Vcentre as suggested but also got the same error message
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by Vitaliy S. »

Can you please browse VM folder located on the datastore and confirm you don't have any snapshots files in this directory. Alternatively, you can use Veeam ONE and custom infrastructure report. To build the report you need to select Virtual Machine as the desired object type, and then add the required columns (such as Snapshot name, Size, Directory, VM name etc.) to the report.

If you don't have any snapshots present, then please contact VMware support team.
jkaw328
Novice
Posts: 5
Liked: never
Joined: Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by jkaw328 »

jkaw328 wrote: Hi Vitality,
how do you know if you have any orphaned snapshots. Also I try doing a snapshot in Vcentre as suggested but also got the same error message
Further to the above I just explore the datastore where the problem VM is stored and notice at least 30 vmdk files. I guest this is the problem. How to I safely delete these files.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by Vitaliy S. »

DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Replication error : Snapshot hierarchy is too deep

Post by DaFresh »

If removing orphaned snapshots didn't work (it sometimes happens whatever you do), you can try cloning the VM or do a V2V.

Anyway this will not solve the root cause of this problem which I think is related to snapshot corruption due to NFS (or any related protocol like IP and Ethernet ...) issues. I think mine was at a time caused by a misconfiguration of the STP at the ethernet layer, which make topology changes and then network micro disconnections. (Yes, my NFS NAS are on the user's LAN, I need them to be reachable by other sites for replication .... )
You should try investigate your network, it could maybe help.

Cheers
Post Reply

Who is online

Users browsing this forum: balseb, Google [Bot], Markush-VE, mbjr and 177 guests