problem removing veeam snapshots

[cicero]

Hi,

after a crashed/hung Veeam Backup 5.0.0 Backupjob,
there are (growing) snapshots, not showing up in Vsphere nore are deleteable
though

vmware-cmd FileserverNeu.vmx removesnapshots

Code: Select all

 vmware-cmd FileserverNeu.vmx removesnapshots
Traceback (most recent call last):
  File "/usr/bin/vmware-cmd", line 93, in ?
    main()
  File "/usr/bin/vmware-cmd", line 65, in main
    result = TaskManager().HandleIfTask(result, async=False)
  File "/usr/lib/vmware/vmware-cmd/TaskManager.py", line 17, in HandleIfTask
    else:      return self.WaitForTask(task)
  File "/usr/lib/vmware/vmware-cmd/TaskManager.py", line 28, in WaitForTask
    result = task.WaitForCompletion()
  File "/usr/lib/vmware/vmware-cmd/Task.py", line 20, in WaitForCompletion
    onProgressUpdate=self.OnProgressUpdate)
  File "/usr/lib/python2.4/site-packages/pyVim/task.py", line 120, in WaitForTask
    raise task.info.error
pyVmomi.VmomiSupport.vim.fault.FileLocked: (vim.fault.FileLocked) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Unable to access file <unspecified filename> since it is locked',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   file = '<unspecified filename>'

since i read a early post with similar issue, i upgraded to 5.0.2 -> same errors:

Code: Select all

Removing snapshot
RemoveSnapshot failed, snapshotRef "16-snapshot-37", timeout "3600000" 
Unable to access file <unspecified filename> since it is locked 
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. 
Possible causes for snapshot removal failure: 
- Network connectivity issue, or vCenter Server is too busy to serve the request 
- ESX host was unable to process snapshot removal request in a timely manner 
- Snapshot was already removed by another application

any idea how i can get rid of them, without reinstall or duplicate the 500GB Windows-Guest?

Thanks a lot, cicero

[ThomasMc]

I had something similar but the disk was still mounted to Veeam VM after the crash which was stopping it

1. Unmount any addition drives from the Veeam VM
2. Create a snapshot of the affected VM and then Delete All

Here is a great article on snapshot troubleshooting and removal

[cicero]

1. Unmount any addition drives from the Veeam VM
2. Create a snapshot of the affected VM and then Delete All
Here is a great article on snapshot troubleshooting and removal

Thanks Thomas for the link!

1.) Do you mean my (in my case physical) Windows machine running Veeam Backup or some FLR appliances?
because there are no further drives mounter @Veeam

2.) did that without success...

[ThomasMc]

1.) Do you mean my (in my case physical) Windows machine running Veeam Backup or some FLR appliances?
because there are no further drives mounter @Veeam

Ah sorry something else must be causing your issue, maybe contact Veeam support

[Vitaliy S.]

I would also recommend looking through this topic (pages 3-4) to try more suggestions of how to remove these snapshots.

[cicero]

okay, thanks!
Maybe there is an issue with a veeam zomby process,
becaus of the filelock (?).

Code: Select all

pyVmomi.VmomiSupport.vim.fault.FileLocked: (vim.fault.FileLocked)

I'll try to reboot ESX an then go further

[tsightler]

Are you using virtual appliance mode? It feels like that message almost has to mean that the virtual disk is still attached to the VM running Veeam. I know you said earlier that "there are no further drives mounter @Veeam" but did you verify that it's removed from the VM's configuration in vCenter?

[cicero]

ESX reboot did't it for me an unlocked the file.
After that I deleted the snapshots with vmware-cmd.

like windows in the past ...

Thanks for all your help!

Cicero

[gshaw]

[merged]

Had an error when testing Veeam yesterday backing up a test server and vCenter (just found out now about the vCenter backup workaround) but think I've got stranded snapshot files in the data store

Code: Select all

Removing snapshot
RemoveSnapshot failed, snapshotRef "snapshot-69", timeout "3600000" 
Failed to login to "SERVER" by SOAP, port 443, user "USER", proxy srv: port:0 
Unable to connect to the remote server 
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond IP:443 
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. 
Possible causes for snapshot removal failure: 
- Network connectivity issue, or vCenter Server is too busy to serve the request 
- ESX host was unable to process snapshot removal request in a timely manner 
- Snapshot was already removed by another application

Thing is I can now see files in the data store as SERVER_1-ctk.vmdk that I think are the orphaned files?

Does this mean I could end up with snapshot problems? I've seen a script in another thread that's meant to sort it but want to know what's the Veeam supported way to tidy this up? Would redefining the job using the ESX host as the way to get to vCenter clean up the orphaned files or will it be a manual process?

[Vitaliy S.]

On the subsequent job run, we will trigger an API call to remove that snapshot, if this call fails for some reason, you should delete existing snapshot manually using the recommendations mentioned above. Redefining the backup job through ESX host wouldn't make any difference.

If you continue experiencing snapshot removal problems I would suggest opening the support case with VMware, so they could shed some light on this. Thanks.

[gshaw]

I think it failed because Veeam was using vCenter to find the vCenter VM, which I understand times out once it freezes the VM via VSS so hopefully if I redefine the job as per here...

Back up vCenter VM?

It should remove the old snapshot?

[Vitaliy S.]

If you continue using existing job this snapshot should be removed, otherwise you should delete it manually.

[gshaw]

If you continue using existing job this snapshot should be removed, otherwise you should delete it manually.

Ah, that's a problem then as vCenter will never complete on that job if I understand correctly?

Or could I run the backup job with VSS disabled on the vCenter VM just to remove the snapshot then delete the backup files and start again with a new job directed at the ESX host to get a VSS-enabled vCenter backup

[Vitaliy S.]

Or could I run the backup job with VSS disabled on the vCenter VM just to remove the snapshot then delete the backup files and start again with a new job directed at the ESX host to get a VSS-enabled vCenter backup

Exactly.

[gshaw]

Just ran the backup from the same job and it didn't remove the -ctk.vmdk files... unless they're meant to stay there (are they change block tracking files or orphaned snapshot files?)

Maybe I'm worrying when I don't need to... what's the best tell-tale sign of orphaned snapshot files being present?

[Vitaliy S.]

...or use VMware script to commit all the snapshots.

[gshaw]

If any were orphaned would they be visible when browsing the datastore? Can't see anything in there apart from the ctk files at the moment so that's a good sign

[glennda]

[merged]

I've got a Server 2008 machine which is backing up fine Via Veeam - although its removing any snapshots its creating the disks still need consolidating as otherwise the Luns just fill up and the server runs off snapshots for a day. Being a Large File Server (about 1tb). These snapshots tend to get rather large and the Luns fill up - server halts.

If you manually consolidate the disks it does this but its annoying Veeam isn't doing it.

Any ideas where to track this down?

[joergr]

VMware? If yes, which ESXi Version and how much space is left on this LUN?
Best regards, Joerg

[kpnz]

We use B&R for our 5.0 environment and hoping to be able to use this to offer a DR solution to our clients. However there have been a couple on instances where our 'test' servers have failed a replication for unknown reasons and we've then found that subsequent replications have resulted in additional CTK files being produced with no option to remove the snapshots or consolidate the disks. I thought this was a replication issue however I noticed a couple of weeks ago one of our vCenter servers, which isn't replicated but backed up with Veeam, had the same issue. For that I ended up preforming a restore since the DB is hosted elsewhere.

Has anyone else experienced this before and know what conditions may cause this to occur? The first couple of servers that this happened to were being both rep0licated and backed up, albeit at different times, however the vCenter server issue in this case was only being backed up. I had a search through the forums and only found an thread which applied to B&R 4.1. We're using 6.5

Thanks

[foggy]

Is there a chance you are experiencing the problem similar to the one discussed above? Otherwise, please contact technical support for assistance.