Solved error removing snapshot: Unable to access file

[m.novelli]

I just solved an error that can raise when using Veeam as Virtual Machine with HOT ADD backup option, and I would like to share with you the resolution

Sometimes can happen that Veeam Backup fail in the middle of processing a VM, and then you can't remove the snapshot from Vmware. When you try you get this error from VI Client: "Unable to access file <unspecified filename> since it is locked"

Actually I got this error two times in about one year. First time time I had to rebuild from scratch the VM, I didn't find a way to remove the snapshot from Vmware. Today happened the second time, and thanks to this post http://blog.vmpros.nl/2010/06/02/vmware ... t-manager/ I discovered that the disks of the VM witch failed the backup where left attached (hot added) to Veeam Backup virtual machine. Simply powering down the Veeam Backup virtual machine and removing the appropriate disks from VM configuration saved me from rebuilding an Exchange server.

Hope this can help,

Marco

[Gostev]

Hi Marco, thanks for sharing this. There is also a hotfix for version 5.0 that prevents this from happening altogether (each time when Veeam Backup service starts, it detects and unmounts any attached disks automatically). So for example, if your server resets/crashes during backup, disks will be removed immediately upon backup server restart.

Upcoming 5.0.1 release will have this functionality merged into it, so installing hotfix will no longer be required.

[m.novelli]

Can you share with us the supposed release month for 5.0.1?
I'm waiting 5.0.1 to upgrade all my customers, I've updated just one that absolutely needed incremental backup mode

Marco

[Gostev]

This should be out in less than a week.

[virtualwatts]

Hi Marco -

Good post, hope this is not a hijack, just showing additional problems with the Veeam snapshots and possible fixes.

We had similar problems because Veeam kept shutting down the guest in the middle of the backup (Duplicate Server Detected). Before we could resolve the problem we had 7 generations of snapshots from Veeam on three of our core production servers. These were true daisy chained snapshots, the CID each pointing to the other.
The quick fix Option 1 in your link quickly consolidated all of the snapshots except for the original. It was a lifesaver.

However - The original Veeam snapshot was unable to be consolidated -
When I opened the the Veeam snapshot vmdk files in vi the parentCID and CID were the same. That is, the snapshot did not have a parent reference, it thought it was it's own parent.
My theory is that Veeam creates non-standard VMware snapshots and in our case VMware couldn't figure out what to do with it and Veeam couldn't clean it up.

I edited the VMDK file to point to the correct original parent (per this VMware kb http://kb.vmware.com/kb/1007969).
This kb plus your link are truly core toolkit lifesavers.

When we tried to restart the guest was DOA and wouldn't start.
The reason was because all of our VMDKs for a VM are on different datastores (no local to vmx, all have same name).
Unfortunately, Veeam creates the snapshot wherever the VMX file is located - rather than the VMware standard of where the vmdk is located.
The problem was that the Veeam snapshot vmdk couldn't find the parent file reference which was on another datastore.
[Soapbox] I think this is a significant design flaw since the high-performance (and large capacity) disks are where the VMDK is located, not where the VMX file is located.[/soapbox]

Lastly, and this wasn't covered in the VMware kb - I had to edit the VMX file becase Veeam creates the snapshot in the "wrong" datastore and VMware couldn't find the vmdk referenced in the ParentCID, so I a) moved the snapshot to the location of the parent VMDK and then b) edited the VMX file to point to the appropriate snapshot file now on another datastore.

After the VMDK and VMX editing and moving above I again performed Option 1 (create new snapshot, delete all snapshots).

Deep sigh of relief the production financial system came back online.

This exercise was great because I got a first-hand look at how VMware performs file-handle magic and it was disturbing because we realized how badly Veeam could mangle our production servers if not properly monitored.

We have had this happen one other time by inadvertantly shutting down the Veeam guest in the middle of a backup. The right answer in that case is to not panic, restart the Veeam server and restart/continue the backup. Veeam will often clean up it's own snapshot files.

[Gostev]

Unfortunately, Veeam creates the snapshot wherever the VMX file is located - rather than the VMware standard of where the vmdk is located.

Actually, this is not correct. The location where the snapshots are created is specified by workingDir parameter of VMX file. By default, workingDir is the same datastore that contains the .vmx file. For example, this is stated here: http://kb.vmware.com/kb/1012384

Of course, potentially we could override default behavior, and use VMDK datastore as workingDir. However, often VM has more than one VMDK, with VMDKs located on different datastores, as VMware recommendation suggests. How do we choose datastore in this case? To resolve quite common issue mentioned in KB article above, it would be logical to select the datastore containing the biggest VMDK. But then again, this would add quite a bit of "randomness" to default behavior that we want to preserve... plus, consequences of this choice can be bad sometimes as well. For example, biggest VMDK may often be stored on cheaper SATA disks, which will in turn affect snapshot consolidation performance.

Finally, changing workingDir requires changing source VM settings. And last thing we want to do is mess around with source VMs and change whatever settings they have

[virtualwatts]

Hi Anton - Thank you much for that clarification.

Our working dirs are not set to the home datastore, so I should see the Veeam snapshots in that location. Interesting.
Unfortunately that unravels my theory for why the Veeam snapshot CID/ParentCIDs were honked up. Well, it was only a theory after all.

Rick

[Bunce]

Would it be possible to add a job option to configure the snapshot location per VM? When calling a snapshot from Veeam, can the datastore be passed as a parameter or must it be hardcoded in the workingDir parameter first?

We have similiar issues with a few of our larger VM's with VMDK's on various LUN's, and have to be careful to manage space on the LUN with the vmx file, even though the larger VMDK's are on a different LUN.

[Gostev]

When calling a snapshot from Veeam, can the datastore be passed as a parameter or must it be hardcoded in the workingDir parameter first?

No, unfortunately CreateSnapshot() VMware API call does not provide this ability. Modification of source VM settings (workingDir) is absolutely required. I recommend that you set this parameter individually for those few larger VMs you need careful disk space management for.

[Titanmike]

Hiya,

where can I find the hotfix please ? I have about 10 backups which have the exact same error and indeed have the snapshots not being removed. Some do show the error even though there isn't even a snapshot to remove ...

Even when I manually remove the snapshot where it still exist, it won't remove it again the next time the backup is running ...

[Gostev]

Hi, from our technical support, as usual

[Titanmike]

Hi, from our technical support, as usual

Is it worth getting in contact with them or can we expect 5.0.1 early next week anyway ?s

[Gostev]

Good catch, you are right, 5.0.1 should be out early next week, so probably no sense to spend time patching 5.0 install at this time.

[LEE337phx]

sorry - almost seems like a tangent of the thread, but there was direct mention regarding this situation, re: parameter for snapshot locations:

yes, I can manually edit the vmx files to change workingDir...but I shouldn't have to...AND, this also causes the vm's swap file to relocate also, unless I FURTHER add another mod for "sched swap dir"...the combination of these two VM parameters puts the swap file where I want it (on the faster disk where the system volume lives), and the snapshots on the slower, but larger, and larger block size, logical disk I have for this....

BUT - a vMotion (whether done proactively, or, as a result of failover or load balance) of a VM will RESET the values back to the ESXi/Cluster/Datacenter default configurations...meaning I would have to re-modify whenever this event occurred, or, use a failed backup due to block-size-snapshot-failure to alert me...

I don't think re-allocating my VM datastores to a block size compatible with data volumes is a very good solution. The types of drives partitioned on my SAN, the block sizing for datastores, and using multiple datastores for type of application balanced against size of logical drives...were what I THOUGHT were the considerations in carving up disk space...and I of course, like perhaps many others, fell prey to the "yes it runs good, yes you have multiple volumes on different types of disk/blocking, performs quite well, HA/vMotion is working....oops, you can't do VM snapshots, nor can you get backup products that utilize snapshots to run" pit...

I HAVE located a PowerShell/PowerCLI script from Hugo Peeters, which I have not completely analyzed, intent of which is to change one or all VM snapshot locations to a large datastore, and then reset to the vmx orginal location...a possible solution, but, something I'd then have to test, particularly in situations of errors in the script, errors during a backup job (one wonders...change the snapshot location, get an error during snapshot, reset the location...what happens next change/snap run?)....

Again, I don't think I should have to do this - I think Veeam should incorporate something like this, or, assuming Veeam has a bit more...pull...with VMware than I would (!), there should ideally be a configuration item (and exposed, so that Veeam can make use of it), to allow high-level definition of where snapshot datastores live, SEPARATELY from workingDir (main vmx location, main swap file location), and swap file location (I see options in vSphere for pointing swap files elsewhere, as <grin> the "not recommeded" choices)...why not a "SnapShotLocation="<somewhereelse>"" that isn't affected by vm relocation ?

[Gostev]

Thank you for your feedback. We will see what we can do.