Host-based backup of VMware vSphere VMs.
Post Reply
cicero
Enthusiast
Posts: 39
Liked: never
Joined: Mar 18, 2009 9:48 am
Full Name: David Winkler
Contact:

problem removing veeam snapshots

Post by cicero »

Hi,

after a crashed/hung Veeam Backup 5.0.0 Backupjob,
there are (growing) snapshots, not showing up in Vsphere nore are deleteable
though

vmware-cmd FileserverNeu.vmx removesnapshots

Code: Select all

 vmware-cmd FileserverNeu.vmx removesnapshots
Traceback (most recent call last):
  File "/usr/bin/vmware-cmd", line 93, in ?
    main()
  File "/usr/bin/vmware-cmd", line 65, in main
    result = TaskManager().HandleIfTask(result, async=False)
  File "/usr/lib/vmware/vmware-cmd/TaskManager.py", line 17, in HandleIfTask
    else:      return self.WaitForTask(task)
  File "/usr/lib/vmware/vmware-cmd/TaskManager.py", line 28, in WaitForTask
    result = task.WaitForCompletion()
  File "/usr/lib/vmware/vmware-cmd/Task.py", line 20, in WaitForCompletion
    onProgressUpdate=self.OnProgressUpdate)
  File "/usr/lib/python2.4/site-packages/pyVim/task.py", line 120, in WaitForTask
    raise task.info.error
pyVmomi.VmomiSupport.vim.fault.FileLocked: (vim.fault.FileLocked) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Unable to access file <unspecified filename> since it is locked',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   file = '<unspecified filename>'
since i read a early post with similar issue, i upgraded to 5.0.2 -> same errors:

Code: Select all

Removing snapshot
RemoveSnapshot failed, snapshotRef "16-snapshot-37", timeout "3600000" 
Unable to access file <unspecified filename> since it is locked 
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. 
Possible causes for snapshot removal failure: 
- Network connectivity issue, or vCenter Server is too busy to serve the request 
- ESX host was unable to process snapshot removal request in a timely manner 
- Snapshot was already removed by another application
any idea how i can get rid of them, without reinstall or duplicate the 500GB Windows-Guest?

Thanks a lot, cicero
ThomasMc
Veteran
Posts: 293
Liked: 19 times
Joined: Apr 13, 2011 12:45 pm
Full Name: Thomas McConnell
Contact:

Re: problem removing veeam snapshots

Post by ThomasMc »

I had something similar but the disk was still mounted to Veeam VM after the crash which was stopping it

1. Unmount any addition drives from the Veeam VM
2. Create a snapshot of the affected VM and then Delete All

Here is a great article on snapshot troubleshooting and removal
cicero
Enthusiast
Posts: 39
Liked: never
Joined: Mar 18, 2009 9:48 am
Full Name: David Winkler
Contact:

Re: problem removing veeam snapshots

Post by cicero »

ThomasMc wrote: 1. Unmount any addition drives from the Veeam VM
2. Create a snapshot of the affected VM and then Delete All
Here is a great article on snapshot troubleshooting and removal
Thanks Thomas for the link!

1.) Do you mean my (in my case physical) Windows machine running Veeam Backup or some FLR appliances?
because there are no further drives mounter @Veeam

2.) did that without success...
ThomasMc
Veteran
Posts: 293
Liked: 19 times
Joined: Apr 13, 2011 12:45 pm
Full Name: Thomas McConnell
Contact:

Re: problem removing veeam snapshots

Post by ThomasMc »

1.) Do you mean my (in my case physical) Windows machine running Veeam Backup or some FLR appliances?
because there are no further drives mounter @Veeam
Ah sorry something else must be causing your issue, maybe contact Veeam support :)
Vitaliy S.
VP, Product Management
Posts: 27347
Liked: 2785 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: problem removing veeam snapshots

Post by Vitaliy S. »

I would also recommend looking through this topic (pages 3-4) to try more suggestions of how to remove these snapshots.
cicero
Enthusiast
Posts: 39
Liked: never
Joined: Mar 18, 2009 9:48 am
Full Name: David Winkler
Contact:

Re: problem removing veeam snapshots

Post by cicero »

okay, thanks!
Maybe there is an issue with a veeam zomby process,
becaus of the filelock (?).

Code: Select all

pyVmomi.VmomiSupport.vim.fault.FileLocked: (vim.fault.FileLocked)
I'll try to reboot ESX an then go further ;)
tsightler
VP, Product Management
Posts: 6031
Liked: 2856 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: problem removing veeam snapshots

Post by tsightler »

Are you using virtual appliance mode? It feels like that message almost has to mean that the virtual disk is still attached to the VM running Veeam. I know you said earlier that "there are no further drives mounter @Veeam" but did you verify that it's removed from the VM's configuration in vCenter?
cicero
Enthusiast
Posts: 39
Liked: never
Joined: Mar 18, 2009 9:48 am
Full Name: David Winkler
Contact:

Re: problem removing veeam snapshots

Post by cicero »

ESX reboot did't it for me an unlocked the file.
After that I deleted the snapshots with vmware-cmd.

like windows in the past ... ;)

Thanks for all your help!

Cicero
gshaw
Enthusiast
Posts: 51
Liked: never
Joined: Apr 07, 2011 2:25 pm
Full Name: Gerrard Shaw
Contact:

Failed job snapshots

Post by gshaw »

[merged]

Had an error when testing Veeam yesterday backing up a test server and vCenter (just found out now about the vCenter backup workaround) but think I've got stranded snapshot files in the data store :(

Code: Select all

Removing snapshot
RemoveSnapshot failed, snapshotRef "snapshot-69", timeout "3600000" 
Failed to login to "SERVER" by SOAP, port 443, user "USER", proxy srv: port:0 
Unable to connect to the remote server 
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond IP:443 
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. 
Possible causes for snapshot removal failure: 
- Network connectivity issue, or vCenter Server is too busy to serve the request 
- ESX host was unable to process snapshot removal request in a timely manner 
- Snapshot was already removed by another application
Thing is I can now see files in the data store as SERVER_1-ctk.vmdk that I think are the orphaned files?

Does this mean I could end up with snapshot problems? I've seen a script in another thread that's meant to sort it but want to know what's the Veeam supported way to tidy this up? Would redefining the job using the ESX host as the way to get to vCenter clean up the orphaned files or will it be a manual process?
Vitaliy S.
VP, Product Management
Posts: 27347
Liked: 2785 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: problem removing veeam snapshots

Post by Vitaliy S. »

On the subsequent job run, we will trigger an API call to remove that snapshot, if this call fails for some reason, you should delete existing snapshot manually using the recommendations mentioned above. Redefining the backup job through ESX host wouldn't make any difference.

If you continue experiencing snapshot removal problems I would suggest opening the support case with VMware, so they could shed some light on this. Thanks.
gshaw
Enthusiast
Posts: 51
Liked: never
Joined: Apr 07, 2011 2:25 pm
Full Name: Gerrard Shaw
Contact:

Re: problem removing veeam snapshots

Post by gshaw »

I think it failed because Veeam was using vCenter to find the vCenter VM, which I understand times out once it freezes the VM via VSS so hopefully if I redefine the job as per here...

Back up vCenter VM?

It should remove the old snapshot?
Vitaliy S.
VP, Product Management
Posts: 27347
Liked: 2785 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: problem removing veeam snapshots

Post by Vitaliy S. »

If you continue using existing job this snapshot should be removed, otherwise you should delete it manually.
gshaw
Enthusiast
Posts: 51
Liked: never
Joined: Apr 07, 2011 2:25 pm
Full Name: Gerrard Shaw
Contact:

Re: problem removing veeam snapshots

Post by gshaw »

Vitaliy S. wrote:If you continue using existing job this snapshot should be removed, otherwise you should delete it manually.
Ah, that's a problem then as vCenter will never complete on that job if I understand correctly?

Or could I run the backup job with VSS disabled on the vCenter VM just to remove the snapshot then delete the backup files and start again with a new job directed at the ESX host to get a VSS-enabled vCenter backup
Vitaliy S.
VP, Product Management
Posts: 27347
Liked: 2785 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: problem removing veeam snapshots

Post by Vitaliy S. »

gshaw wrote:Or could I run the backup job with VSS disabled on the vCenter VM just to remove the snapshot then delete the backup files and start again with a new job directed at the ESX host to get a VSS-enabled vCenter backup
Exactly.
gshaw
Enthusiast
Posts: 51
Liked: never
Joined: Apr 07, 2011 2:25 pm
Full Name: Gerrard Shaw
Contact:

Re: problem removing veeam snapshots

Post by gshaw »

Just ran the backup from the same job and it didn't remove the -ctk.vmdk files... unless they're meant to stay there (are they change block tracking files or orphaned snapshot files?)

Maybe I'm worrying when I don't need to... what's the best tell-tale sign of orphaned snapshot files being present?
Vitaliy S.
VP, Product Management
Posts: 27347
Liked: 2785 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: problem removing veeam snapshots

Post by Vitaliy S. »

...or use VMware script to commit all the snapshots.
gshaw
Enthusiast
Posts: 51
Liked: never
Joined: Apr 07, 2011 2:25 pm
Full Name: Gerrard Shaw
Contact:

Re: problem removing veeam snapshots

Post by gshaw »

If any were orphaned would they be visible when browsing the datastore? Can't see anything in there apart from the ctk files at the moment so that's a good sign :)
glennda
Novice
Posts: 3
Liked: never
Joined: Jul 24, 2012 11:51 am
Full Name: Toby Glenn
Contact:

Veeam Removing Snapshots but Disks needing Consolidation

Post by glennda »

[merged]

I've got a Server 2008 machine which is backing up fine Via Veeam - although its removing any snapshots its creating the disks still need consolidating as otherwise the Luns just fill up and the server runs off snapshots for a day. Being a Large File Server (about 1tb). These snapshots tend to get rather large and the Luns fill up - server halts.

If you manually consolidate the disks it does this but its annoying Veeam isn't doing it.

Any ideas where to track this down?
joergr
Veteran
Posts: 391
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Veeam Removing Snapshots but Disks needing Consolidation

Post by joergr »

VMware? If yes, which ESXi Version and how much space is left on this LUN?
Best regards, Joerg
kpnz
Novice
Posts: 8
Liked: never
Joined: Feb 12, 2013 9:34 pm
Contact:

[MERGED] CTK files not being removed and incrementing

Post by kpnz »

We use B&R for our 5.0 environment and hoping to be able to use this to offer a DR solution to our clients. However there have been a couple on instances where our 'test' servers have failed a replication for unknown reasons and we've then found that subsequent replications have resulted in additional CTK files being produced with no option to remove the snapshots or consolidate the disks. I thought this was a replication issue however I noticed a couple of weeks ago one of our vCenter servers, which isn't replicated but backed up with Veeam, had the same issue. For that I ended up preforming a restore since the DB is hosted elsewhere.

Has anyone else experienced this before and know what conditions may cause this to occur? The first couple of servers that this happened to were being both rep0licated and backed up, albeit at different times, however the vCenter server issue in this case was only being backed up. I had a search through the forums and only found an thread which applied to B&R 4.1. We're using 6.5

Thanks
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: problem removing veeam snapshots

Post by foggy »

Is there a chance you are experiencing the problem similar to the one discussed above? Otherwise, please contact technical support for assistance.
Post Reply

Who is online

Users browsing this forum: chris.mcdonald and 51 guests