Unable to delete the snapshot due to locked files

[arsprod]

Is anyone else having this issue with esxi 5.x? I've now had two vm's that build up snapshots that don't show up in snapshot manager and eventually get to the point where vsphere is unable to consolidate snapshots because of this locked file issue. The kb article http://kb.vmware.com/selfservice/micros ... Id=2017072 explains it well. Problem is while cloning the vm is a workaround vmware clearly states, "Currently, there is no resolution." I'm wondering if veeam is working on a resolution since it's directly related to backup snapshots.

[Vitaliy S.]

Hello Aaron,

Not sure there is anything we can do now in this situation, since snapshot creation/commitment is controlled by VMware and Veeam just triggers the API call published in VMware documentation. However, I wonder where these locks came from originally. Do you have a case opened on this?

Can you please tell me if you've tried to create new snapshot and then remove the entire snapshot tree to remove the lock?

Thank you!

[arsprod]

Hello Aaron,

Not sure there is anything we can do now in this situation, since snapshot creation/commitment is controlled by VMware and Veeam just triggers the API call published in VMware documentation. However, I wonder where these locks came from originally. Do you have a case opened on this?

Can you please tell me if you've tried to create new snapshot and then remove the entire snapshot tree to remove the lock?

Thank you!

I did have a case with support and we got as far as the workaround and that it's a vmware issue. Manually creating a snapshot and then deleting all doesn't work in this scenario. It appears that the snapshots are 'hidden' from snapshot manager - the vmdk's are there but don't show up in snapshot manager. Apparently this is happening with other backup programs and since there's a kb article I know I'm not the only one!

For veeam it seems that there's some link between creating these phantom snapshots and the machine being used as a proxy. The two machines I have that have done this are both proxies so that's an assumption.

[Vitaliy S.]

Try to restart ESXi hosts agents, might help to commit these orphaned snapshots. BTW, do you see any Veeam processes (VeeamAgent*) running on the backup server with no running jobs?

What kind of proxy servers are these? HotAdd or network? Do you backup these proxy servers as well?

[arsprod]

Try to restart ESXi hosts agents, might help to commit these orphaned snapshots. BTW, do you see any Veeam processes (VeeamAgent*) running on the backup server with no running jobs?

What kind of proxy servers are these? HotAdd or network? Do you backup these proxy servers as well?

I will next time it happens (already cloned the vm) - no veeam processes running with no running jobs. Both are hotadd proxys and yes, I back them up

[Vitaliy S.]

I'm pretty sure that the issue you've reported before is connected to the current one, and the reason for both issues should be common.

[arsprod]

I'm pretty sure that the issue you've reported before is connected to the current one, and the reason for both issues should be common.

Maybe, but the previous issue you highlighted was just me being stupid! I agree that the reason for both may be common but the result is an annoyance and possibly a potential problem. I also think it's a problem with vmware but they're much less likely to find a solution than veeam as some other backup vendors have done. I'm still curious to see if other veeam users are having this issue and if we can find some common infrastructure that causes it. Thanks!

[Vitaliy S.]

Just a wild guess - since these are hotadd proxy servers and you back them up, there might be some race conditions when one proxy is being backed up and at the same time tries either to mount or remove disk from other VMs. If it is acceptable, I would try to switch one proxy to network mode and monitor if it faces the same issue or not.

[arsprod]

Just a wild guess - since these are hotadd proxy servers and you back them up, there might be some race conditions when one proxy is being backed up and at the same time tries either to mount or remove disk from other VMs. If it is acceptable, I would try to switch one proxy to network mode and monitor if it faces the same issue or not.

Sounds reasonable (and likely) - I'll switch now (though I love the speed of hotadd!!)

[Vitaliy S.]

With CBT enabled the difference should not be that noticeable.

[coolgod]

My system is working fine am just asking for information purpose only.

I was reading this VMWARE KB and decided to use same title for this new topic.

I have quoted part of the KB:

This issue occurs if the backup server or appliance or other virtual machine holds a lock on the underlying base disk or previous snapshot file, which prevents the data being consolidated. It can also occur due to stale processes during snapshot creation.

To access the underlying disk to back up the contents, backup servers and appliances use snapshots through the vStorage API. For more or related information on the vStorage API, see vStorage APIs for Data Protection (VADP) FAQ (1021175).

After the backup completes, the backup server requests the snapshot taken for the backup to be deleted and the data merged into the previous disk. In some cases, the backup server may request the snapshot be deleted too early, while the base disk is still locked by the backup server. This results in:
-The snapshot delete process terminating.
-The snapshot manager having inaccurate or no information.
-The virtual machine continuing to run on snapshots.

According to the KB, the backup server may request the snapshot be deleted too early..Is there any precautions Veeam takes to wait and check whether the base disk is still locked by the backup server. ?

What can be done to avoid this ?
VMWARE says on this same KB that Currently, there is no resolution

Please send me your comments.

[coolgod]

This is not clear to me.

Am quoting again the part which is still confusing:

In some cases, the backup server may request the snapshot be deleted too early, while the base disk is still locked by the backup server.

So, is it not possible for VEEAM to check via the VDDK API whether the base disk is still locked before requesting a snapshot consolidation ?

I hope am not harassing you by asking such a question My motivation for asking is to be proactive and prevent issues.

[Vitaliy S.]

I cannot comment on the dev code, but I assume we do run these checks. Also if you want to be proactive, you can use Veeam ONE for monitoring leftovers and orphaned snapshots in your infrastructure.

[coolgod]

OK thanks. can you send me a link for the documentation on how to do this ?

[Vitaliy S.]

Yes, sure. To keep an eye on VM snapshots, you need to install Veeam ONE and use our predefined alerts and reports for that, please see these links for further reading:

http://helpcenter.veeam.com/one/70/vsph ... tails.html
http://helpcenter.veeam.com/one/70/repo ... shots.html

[coolgod]

thanks man.