Virtual Appliance mode leaving Snapshots

[schaffeb]

WARNING!!

I had installed the hotfix and it seemed as though my replications were going fine (no orphaned snapshots). However, today I was performing a replica and I stopped it prior to completion. This seems to have left me with an orphaned snapshot. I performed another replication, and it went fine, however I still have the orphaned snapshot from the original canceled.

I'm really in a bad spot here as this is a huge and very important production VM. I urge caution to all those using Virtual Appliance mode, even with the hotfix. It looks like there are still issues to work out.

[floss]

I yes i would agree, i have applied the fix and another one of of our severs hung on a job and thus the snapshot trial become corrupted this was also in SAN mode, leaving me with no option than a full clone and downtime to sort out i still do not have the confidence to use this in production with tier one servers.
im no expert but it looks to me like there are issues around clean snapshot removals
Will this be looked into within the next update as its the only problem i can see with a super product

[Gostev]

I just wanted to clarify (to avoid confusion and issues mix up), that orphaned snapshots on stop job operation is a separate issue we have been tracking in the following thread: Helper snapshot not removed when vStorage API job is stopped.

As I posted there in my latest post a few days ago, we have introduced significant changes in stop job algorithm in version 4.1 which addresses this issue. Because of scale of the changes, we will not be able to provide hot fix addressing this issue for version 4.0, it will be available in version 4.1 we are planning to make generally available first half of December (real soon).

For now, please avoid stopping jobs running in vStorage API mode (other modes are not affected) unless absolutely necessary. With changed block tracking, incremental job cycle is extremely fast and it is best to wait couple of minutes until it finishes - instead of stopping it.

[schaffeb]

I'm glad to hear that, and I eagerly look forward to the 4.1 update. I guess it was not clear to me when I received the hotfix that stopping jobs would still be an issue.

Is there any way to get out of this mess without a clone?

Thanks for the quick responses as always!

[schaffeb]

As a side note, I don't think we may not be talking about the same issue Gosteve. I don't see a "consolidate helper-x" snapshot.

I don't see any snapshots in the snapshot manager at all.

[Gostev]

Ben, it is the same issue but you are using the different vStorage API job modes. As for removing snapshot, I believe that creating additional snapshot on the same VM will reveal another snapshot and you should be able to remove it.

[zeki]

hello
i had this issue too. the only way to remove the snapshots was to write the actual snapshot f.e ...-00003.vmdk into a new vmdk file on the service console with vmkfstools

greets
zeki

[schaffeb]

Hi Zeki,

I would be so grateful if you could describe vmkfstools method in more detail! Sounds like a lifesaver.

Gostev, unfortunately, creating a new snapshot does not reveal the old snapshot.

[floss]

Hi I have looked into this and spoke to Vmware support and basically the adding another snapshot and deleteing will not sort out the chain, if you also run a cat on .vmsd file you will see odd info relating to snapshots

The the safest way is a full offline clone but you will need downtime, also > Gostev if you run a Vstorage API job and this hangs 99% of the time you have no option but stop the job, so this a risk i personally would not take until the 4.1 update

[Gostev]

Floss, we did not have any with hang job @ 99% reported past month since 4.0 release... if you are experiencing this, please contact our support so that we could add the fix for this in 4.1 as well.

[zeki]

hello schaffeb

to do this metho you have to shutdown the vm.

1. shutdown the vm
2. make a backup of the whole vm folder (if anything gets wrong)
3. detect what snapshot is actualy used -> properties of the vm, edit the disk (maybe you see server-000002.vmdk)
4. ssh to esx and go to /vmfs/volumes/yourshared storage/vm folder
5. vmkfstools -i server-000002.vmdk server-C.vmdk (takes some time....)
6. delete the disk you get the properties on 3.)
7. add a existing disk to the vm (server-C.vmdk)

thanks it. now you can start the VM and delete the unused server.vmdk, server-00001.vmdk, server-00002.vmdk, ....

greets
zeki

[floss]

Floss, we did not have any with hang job @ 99% reported past month since 4.0 release... if you are experiencing this, please contact our support so that we could add the fix for this in 4.1 as well.

Gostev , i think i may have explianend myself wrong i meant that WHEN the jobs hangs which is very rare, 99% of time you will need to forcefully stop the job,
so waiting is not usually option

By the way i know this thread is slightly negative slant on Veeam 4.0 but this is the only issue i have seen which causes concern and also the support from you and your guys is first class and i cant fault it..

[Gostev]

Ah, OK - thanks for clarification.

Thank you, we try our best we do apologize for any inconveniences - v4 has been a huge shift in technologies (vSphere and ESX4, changed block tracking, vStorage API, Virtual Appliance mode, transactional backup storage) and a few issues did slip through lab and beta testing.

But all known issues are addressed in 4.1 and I am sure will be rock solid... so that we could concentrate on v5 and all the new goodness.

[zeki]

nice nice
gostev, any news about the slow network mode from vmware?

http://www.vnotion.com/?p=38
greets
zeki

[Gostev]

I just heard the fix did not make it into ESX4 Update 1. I guess I need to talk to VMware support engineer handling my support case... want to get official confirmation and to ask what happened and what are the plans.

[schaffeb]

Just an update on my snapshot saga...

I've tried everything and there really appears to be no way to get rid of these orphaned snapshots without a clone

Oh well, fortunately a clone will only take 4 hours. I thought it was going to take 10, so I'm not in as bad of shape as I thought. Veeam is great, but I cannot wait until 4.1 comes out!

[Personne]

I have not read eveything on this thread but this

http://kb.vmware.com/selfservice/micros ... Id=1012383

fix all my issues with the ctk files

In short:
1) run the python script on your volumes (see link above)
2) create a snapshot for each of your VMs (easier with powershell if you have tens of VM)
3) delete all snapshot (easier with powershell if you have tens of VM)
4) no more ctk !

Worked for me like a charm

[TrevorBell]

If you are GUI like myself an alternative way is this -:

1. Power down VM
2. Make a clone of that VM with auto power on enabled ( just to auto run after its created )
3. Delete original Vm after checking all is working with the clone

I did this all weekend on my 30 production Vm`s all went fine with no problems

Trev

[schaffeb]

Trevor,

Wish I could do this, unfortunately this VM has rapidly changing user data. So I'll lose all the data from when the original snapshot for the clone was taken.

Hopefully this python script will work, I'm going to give it a try.

[lohelle]

Check the vmdk files for the snapshots (the small 00000x.vmdk, not the delta! ) Look for a line with the text ddb.deletable
Is this line there? Is this set to "false" ? I had this issue with Vmware Data Recovery.

Fix: Power off VM. Change all the snapshot vmdk's from having the line ddb.deletable = "false" to ddb.deletable = "true"

You should now create a new snapshot. When doing a "delete all" in snapshot manager they should all go away (if there was ddb.deletable = "false" statements in the vmdk's)

by the way:
If you login by ssh (or console) you can issuse this command to look for open/orphaned snapshots: "find /vmfs/volumes/ -name \*delta\*"
More in this nice blog: http://blog.scottlowe.org/2009/06/29/sn ... -recovery/

[Personne]

To lohelle,

this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383

see 4 posts above

[schaffeb]

Personne,

THANK YOU!

That script worked beautifully!!!!!!!!!!

[lohelle]

To lohelle,

this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383
see 4 posts above

Sorry, did not catch that this was the same solution. (did not click the link....)

[eric]

Agh! I am coming into this thread with the same problem. I tested running Veeam 4.0 in Appliance Mode on several test boxes and everything seemed fine. Now I have orphaned snapshots on several production boxes. I found out about the problem when I went to Storage vMotion one of the machines and it complained about link-mode snapshots....

I am going to go with the cold Clone mode to fix this issue. I don't want to screw with merging/deleting snapshots. I have been there and that is a major pain plus risky.

Looking forward to 4.1 as well. Veeam should send an email to everyone not to use Appliance Mode for now....

Eric

[zeki]

i wait for virtual mode till the BUG in vstorage API is solved...

@gostev: any news about this?

[Gostev]

VMware has just released vStorage API 1.1.1 with many bugs fixed, and we have incorporated these bits into Veeam Backup 4.1 release. Feedback from beta is great so far - we are planning to release 4.1 next week if the final testing goes well.

[floss]

Hi

Is the Vstorage API 1.1.1 a standalone fix that can be downloaded or is it incorperated in Vsphere 4u1

Regards

[Gostev]

Hello, vStorage API 1.1.1 is a set of VMware components that is included and redistributed with our product. You can download those components separately, but you cannot just drop them into Veeam Backup 4.0 installation, because of a number of dependencies between these components and Veeam Backup code.

Veeam Backup 4.1 will be shipped with vStorage API 1.1.1, and it will fully support this release.

[TheJourney]

Is 4.1 going to clean up this mess for us??

[zeki]

i hope... virtual appliance mode is not usable at the moment...