Replication error : Snapshot hierarchy is too deep

Availability for the Always-On Enterprise

Replication error : Snapshot hierarchy is too deep

Veeam Logoby DaFresh » Mon Dec 09, 2013 4:44 pm

Hello,

I randomly (but regulary) got this error with replications jobs, on different Veeam instance / different site. For example, one site has 16 vm that are replicated every night, everything works fine/no error are reported by Veeam, until half of them fail with this error cause. All the replications job are done one NFS NAS, same vendor on all site.

The only way I have found to solve this at the moment is to remove the corrupted replica and let Veeam redo a full replication. I got to try this one too :

http://kb.vmware.com/selfservice/micros ... Id=1004545

But it will not solve the root cause, which I begin to think is related to the NFS NAS itself.

Anyone have any idea about that ? It will be greatly appreciated !

Thx,

Cédric
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby foggy » Tue Dec 10, 2013 10:30 am

Cedric, could you please check how many snapshots do replica VMs have at the moment of failure? Do you see any snapshots that are not Veeam replica restore points?

The fact is that VMware allows 32 snapshots per VM maximum, while Veeam B&R replication job has maximum retention of 28 restore points. So probably some snapshots are not properly deleted or something.
foggy
Veeam Software
 
Posts: 15273
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby DaFresh » Tue Dec 10, 2013 3:27 pm

Hi Foggy,

Yes I have one snapshot called "Veeam Replica Working Snapshot" at the end of the chain, then 2 restores point and finally the base image of the VM at the top.
As you guess, the job is configured to keep 3 restore points.
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby foggy » Tue Dec 10, 2013 4:12 pm

Could you also check your datastore for orphaned snapshot files that are not visible in the vSphere Client?
foggy
Veeam Software
 
Posts: 15273
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby DaFresh » Tue Dec 10, 2013 4:21 pm

Yes already done :
Code: Select all

-rw-------    1 root     root        208896 Dec  7 00:57 TSEFR.ixblue.corp-000001-delta.vmdk
-rw-------    1 root     root           345 Dec  7 00:57 TSEFR.ixblue.corp-000001.vmdk
-rw-------    1 root     root         33169 Dec  3 05:20 TSEFR.ixblue.corp-Snapshot2.vmsn
-rw-------    1 root     root         33775 Dec  5 02:41 TSEFR.ixblue.corp-Snapshot4.vmsn
-rw-------    1 root     root         33777 Dec  7 00:57 TSEFR.ixblue.corp-Snapshot5.vmsn
-rw-------    1 root     root     107374182400 Dec  7 01:25 TSEFR.ixblue.corp-flat.vmdk
-rw-------    1 root     root          8684 Dec  7 00:53 TSEFR.ixblue.corp.nvram
-rw-------    1 root     root           529 Dec  7 01:25 TSEFR.ixblue.corp.vmdk
-rw-------    1 root     root          1619 Dec  7 00:57 TSEFR.ixblue.corp.vmsd
-rw-------    1 root     root          5469 Dec  7 00:57 TSEFR.ixblue.corp.vmx
-rw-------    1 root     root          3049 Dec  7 00:57 TSEFR.ixblue.corp.vmxf
-rw-------    1 root     root        106496 Dec  7 00:57 TSEFR.ixblue.corp_1-000001-delta.vmdk
-rw-------    1 root     root           349 Dec  7 00:57 TSEFR.ixblue.corp_1-000001.vmdk
-rw-------    1 root     root     53687091200 Dec  7 01:25 TSEFR.ixblue.corp_1-flat.vmdk
-rw-------    1 root     root           530 Dec  7 01:25 TSEFR.ixblue.corp_1.vmdk
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby foggy » Tue Dec 10, 2013 4:30 pm

Are there probably any related events in vSphere Client? This error comes from vSphere itself and most likely can be reproduced by creating snapshot manually, irrespective of the backup performed by Veeam B&R.
foggy
Veeam Software
 
Posts: 15273
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby jkaw328 » Wed Jun 04, 2014 2:13 pm

DaFresh wrote:Hello,

I randomly (but regulary) got this error with replications jobs, on different Veeam instance / different site. For example, one site has 16 vm that are replicated every night, everything works fine/no error are reported by Veeam, until half of them fail with this error cause. All the replications job are done one NFS NAS, same vendor on all site.

The only way I have found to solve this at the moment is to remove the corrupted replica and let Veeam redo a full replication. I got to try this one too :

http://kb.vmware.com/selfservice/micros ... Id=1004545

But it will not solve the root cause, which I begin to think is related to the NFS NAS itself.

Anyone have any idea about that ? It will be greatly appreciated !

Thx,

Cédric


I have deleted all the snapshot but still getting the same error?
jkaw328
Novice
 
Posts: 5
Liked: never
Joined: Wed Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby Vitaliy S. » Wed Jun 04, 2014 5:47 pm

Can you manually create the snapshot via vSphere Client for this VM? Did you remove these snapshot via backup console by reducing retention policy or cleared them out using vSphere Client?
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby jkaw328 » Wed Jun 04, 2014 11:04 pm

Hi Vitality
I cleared them out using vsphere client. I will try creating it manually via vsphere client later
jkaw328
Novice
 
Posts: 5
Liked: never
Joined: Wed Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby Vitaliy S. » Thu Jun 05, 2014 9:27 am

Johnny, also I would recommend taking a look at related messages in VMware vSphere for this VM. Keep us updated on how it goes.
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby DaFresh » Thu Jun 05, 2014 1:39 pm

Hello,

As a side note, I still got those errors sometimes, it happens only on my replicated VM stored on NFS, and I think it is directly related to the NFS and/or NAS.

@jkaw328: what is the backend storage, NFS ?
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby jkaw328 » Thu Jun 05, 2014 7:53 pm

DaFresh wrote:Hello,

As a side note, I still got those errors sometimes, it happens only on my replicated VM stored on NFS, and I think it is directly related to the NFS and/or NAS.

@jkaw328: what is the backend storage, NFS ?

Hi DaFresh,
Yes vm stored on NFS but happens on both my backup and replication job.
jkaw328
Novice
 
Posts: 5
Liked: never
Joined: Wed Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby Vitaliy S. » Thu Jun 05, 2014 8:13 pm

Hi Johnny,

You say you have the same issue for backup jobs? Do you have any orphaned snapshots for the source VMs? In this case I would recommend to VMware support team for further investigation.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby DaFresh » Fri Jun 06, 2014 2:43 pm

As Vitaliy said, I am pretty sure too that the issue is snapshot(vMWare) and NFS related and that Veeam does not play any role in that.
I already had opened an issue at vMWare, but they gently redirect me the to my NFS vendor, which is Synology. I didn't take time to investigate more because it happens once every 3 months, and as it only concern the replicated VM I just have to clean the replica and let Veeam regenerate them. I am in the hope that the issue will be solved by itself after a upgrade of the Synology and/or the vMWare NFS stack.
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Replication error : Snapshot hierarchy is too deep

Veeam Logoby jkaw328 » Sun Jun 08, 2014 8:44 am

Vitaliy S. wrote:Hi Johnny,

You say you have the same issue for backup jobs? Do you have any orphaned snapshots for the source VMs? In this case I would recommend to VMware support team for further investigation.

Thanks!


Hi Vitality,
how do you know if you have any orphaned snapshots. Also I try doing a snapshot in Vcentre as suggested but also got the same error message
jkaw328
Novice
 
Posts: 5
Liked: never
Joined: Wed Jun 04, 2014 1:42 pm
Full Name: Johnny Kaw

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google Feedfetcher, Moebius and 1 guest