Comprehensive data protection for all workloads
schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Nov 24, 2009 4:16 am

WARNING!!

I had installed the hotfix and it seemed as though my replications were going fine (no orphaned snapshots). However, today I was performing a replica and I stopped it prior to completion. This seems to have left me with an orphaned snapshot. I performed another replication, and it went fine, however I still have the orphaned snapshot from the original canceled.

I'm really in a bad spot here as this is a huge and very important production VM. I urge caution to all those using Virtual Appliance mode, even with the hotfix. It looks like there are still issues to work out.

floss
Enthusiast
Posts: 31
Liked: never
Joined: Feb 15, 2009 8:31 pm
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by floss » Nov 24, 2009 9:42 am

I yes i would agree, i have applied the fix and another one of of our severs hung on a job and thus the snapshot trial become corrupted this was also in SAN mode, leaving me with no option than a full clone and downtime to sort out i still do not have the confidence to use this in production with tier one servers.
im no expert but it looks to me like there are issues around clean snapshot removals
Will this be looked into within the next update as its the only problem i can see with a super product

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Nov 24, 2009 2:07 pm

I just wanted to clarify (to avoid confusion and issues mix up), that orphaned snapshots on stop job operation is a separate issue we have been tracking in the following thread: Helper snapshot not removed when vStorage API job is stopped.

As I posted there in my latest post a few days ago, we have introduced significant changes in stop job algorithm in version 4.1 which addresses this issue. Because of scale of the changes, we will not be able to provide hot fix addressing this issue for version 4.0, it will be available in version 4.1 we are planning to make generally available first half of December (real soon).

For now, please avoid stopping jobs running in vStorage API mode (other modes are not affected) unless absolutely necessary. With changed block tracking, incremental job cycle is extremely fast and it is best to wait couple of minutes until it finishes - instead of stopping it.

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Nov 24, 2009 4:01 pm

I'm glad to hear that, and I eagerly look forward to the 4.1 update. I guess it was not clear to me when I received the hotfix that stopping jobs would still be an issue.

Is there any way to get out of this mess without a clone?

Thanks for the quick responses as always!

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Nov 24, 2009 4:06 pm

As a side note, I don't think we may not be talking about the same issue Gosteve. I don't see a "consolidate helper-x" snapshot.

I don't see any snapshots in the snapshot manager at all.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Nov 24, 2009 8:00 pm

Ben, it is the same issue but you are using the different vStorage API job modes. As for removing snapshot, I believe that creating additional snapshot on the same VM will reveal another snapshot and you should be able to remove it.

zeki
Enthusiast
Posts: 26
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by zeki » Nov 24, 2009 8:12 pm

hello
i had this issue too. the only way to remove the snapshots was to write the actual snapshot f.e ...-00003.vmdk into a new vmdk file on the service console with vmkfstools

greets
zeki

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Nov 24, 2009 8:57 pm

Hi Zeki,

I would be so grateful if you could describe vmkfstools method in more detail! Sounds like a lifesaver.

Gostev, unfortunately, creating a new snapshot does not reveal the old snapshot.

floss
Enthusiast
Posts: 31
Liked: never
Joined: Feb 15, 2009 8:31 pm
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by floss » Nov 24, 2009 8:59 pm

Hi I have looked into this and spoke to Vmware support and basically the adding another snapshot and deleteing will not sort out the chain, if you also run a cat on .vmsd file you will see odd info relating to snapshots

The the safest way is a full offline clone but you will need downtime, also > Gostev if you run a Vstorage API job and this hangs 99% of the time you have no option but stop the job, so this a risk i personally would not take until the 4.1 update

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Nov 24, 2009 9:29 pm

Floss, we did not have any with hang job @ 99% reported past month since 4.0 release... if you are experiencing this, please contact our support so that we could add the fix for this in 4.1 as well.

zeki
Enthusiast
Posts: 26
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by zeki » Nov 24, 2009 9:44 pm

hello schaffeb

to do this metho you have to shutdown the vm.

1. shutdown the vm
2. make a backup of the whole vm folder (if anything gets wrong)
3. detect what snapshot is actualy used -> properties of the vm, edit the disk (maybe you see server-000002.vmdk)
4. ssh to esx and go to /vmfs/volumes/yourshared storage/vm folder
5. vmkfstools -i server-000002.vmdk server-C.vmdk (takes some time....)
6. delete the disk you get the properties on 3.)
7. add a existing disk to the vm (server-C.vmdk)

thanks it. now you can start the VM and delete the unused server.vmdk, server-00001.vmdk, server-00002.vmdk, ....

greets
zeki

floss
Enthusiast
Posts: 31
Liked: never
Joined: Feb 15, 2009 8:31 pm
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by floss » Nov 24, 2009 10:21 pm

Gostev wrote:Floss, we did not have any with hang job @ 99% reported past month since 4.0 release... if you are experiencing this, please contact our support so that we could add the fix for this in 4.1 as well.
Gostev , i think i may have explianend myself wrong i meant that WHEN the jobs hangs which is very rare, 99% of time you will need to forcefully stop the job,
so waiting is not usually option

By the way i know this thread is slightly negative slant on Veeam 4.0 but this is the only issue i have seen which causes concern and also the support from you and your guys is first class and i cant fault it..

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Nov 24, 2009 11:02 pm

Ah, OK - thanks for clarification.

Thank you, we try our best :) we do apologize for any inconveniences - v4 has been a huge shift in technologies (vSphere and ESX4, changed block tracking, vStorage API, Virtual Appliance mode, transactional backup storage) and a few issues did slip through lab and beta testing.

But all known issues are addressed in 4.1 and I am sure will be rock solid... so that we could concentrate on v5 and all the new goodness. :D

zeki
Enthusiast
Posts: 26
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by zeki » Nov 24, 2009 11:16 pm

nice nice ;)
gostev, any news about the slow network mode from vmware?

http://www.vnotion.com/?p=38
greets
zeki

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Nov 24, 2009 11:24 pm

I just heard the fix did not make it into ESX4 Update 1. I guess I need to talk to VMware support engineer handling my support case... want to get official confirmation and to ask what happened and what are the plans.

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Nov 25, 2009 4:16 pm

Just an update on my snapshot saga...

I've tried everything and there really appears to be no way to get rid of these orphaned snapshots without a clone :(

Oh well, fortunately a clone will only take 4 hours. I thought it was going to take 10, so I'm not in as bad of shape as I thought. Veeam is great, but I cannot wait until 4.1 comes out!

Personne
Influencer
Posts: 11
Liked: never
Joined: Nov 11, 2009 3:07 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Personne » Nov 29, 2009 10:08 pm

I have not read eveything on this thread but this

http://kb.vmware.com/selfservice/micros ... Id=1012383

fix all my issues with the ctk files

In short:
1) run the python script on your volumes (see link above)
2) create a snapshot for each of your VMs (easier with powershell if you have tens of VM)
3) delete all snapshot (easier with powershell if you have tens of VM)
4) no more ctk !

Worked for me like a charm :D

TrevorBell
Expert
Posts: 355
Liked: 10 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by TrevorBell » Nov 30, 2009 7:28 am

If you are GUI like myself an alternative way is this -:

1. Power down VM
2. Make a clone of that VM with auto power on enabled ( just to auto run after its created )
3. Delete original Vm after checking all is working with the clone :)

I did this all weekend on my 30 production Vm`s all went fine with no problems

Trev

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Nov 30, 2009 4:42 pm

Trevor,

Wish I could do this, unfortunately this VM has rapidly changing user data. So I'll lose all the data from when the original snapshot for the clone was taken.

Hopefully this python script will work, I'm going to give it a try.

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by lohelle » Dec 01, 2009 8:40 am

Check the vmdk files for the snapshots (the small 00000x.vmdk, not the delta! ) Look for a line with the text ddb.deletable
Is this line there? Is this set to "false" ? I had this issue with Vmware Data Recovery.

Fix: Power off VM. Change all the snapshot vmdk's from having the line ddb.deletable = "false" to ddb.deletable = "true"

You should now create a new snapshot. When doing a "delete all" in snapshot manager they should all go away (if there was ddb.deletable = "false" statements in the vmdk's)

by the way:
If you login by ssh (or console) you can issuse this command to look for open/orphaned snapshots: "find /vmfs/volumes/ -name \*delta\*"
More in this nice blog: http://blog.scottlowe.org/2009/06/29/sn ... -recovery/

Personne
Influencer
Posts: 11
Liked: never
Joined: Nov 11, 2009 3:07 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Personne » Dec 01, 2009 3:22 pm

To lohelle,

this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383

see 4 posts above

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by schaffeb » Dec 01, 2009 5:35 pm

Personne,

THANK YOU!

That script worked beautifully!!!!!!!!!!

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by lohelle » Dec 01, 2009 6:29 pm

Personne wrote:To lohelle,

this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383
see 4 posts above
Sorry, did not catch that this was the same solution. :) (did not click the link....)

eric
Enthusiast
Posts: 39
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by eric » Dec 07, 2009 12:47 am

Agh! I am coming into this thread with the same problem. I tested running Veeam 4.0 in Appliance Mode on several test boxes and everything seemed fine. Now I have orphaned snapshots on several production boxes. I found out about the problem when I went to Storage vMotion one of the machines and it complained about link-mode snapshots....

I am going to go with the cold Clone mode to fix this issue. I don't want to screw with merging/deleting snapshots. I have been there and that is a major pain plus risky.

Looking forward to 4.1 as well. Veeam should send an email to everyone not to use Appliance Mode for now....

Eric

zeki
Enthusiast
Posts: 26
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by zeki » Dec 07, 2009 9:42 am

i wait for virtual mode till the BUG in vstorage API is solved...

@gostev: any news about this?

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Dec 07, 2009 10:43 am

VMware has just released vStorage API 1.1.1 with many bugs fixed, and we have incorporated these bits into Veeam Backup 4.1 release. Feedback from beta is great so far - we are planning to release 4.1 next week if the final testing goes well.

floss
Enthusiast
Posts: 31
Liked: never
Joined: Feb 15, 2009 8:31 pm
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by floss » Dec 07, 2009 12:26 pm

Hi

Is the Vstorage API 1.1.1 a standalone fix that can be downloaded or is it incorperated in Vsphere 4u1

Regards

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by Gostev » Dec 07, 2009 12:50 pm

Hello, vStorage API 1.1.1 is a set of VMware components that is included and redistributed with our product. You can download those components separately, but you cannot just drop them into Veeam Backup 4.0 installation, because of a number of dependencies between these components and Veeam Backup code.

Veeam Backup 4.1 will be shipped with vStorage API 1.1.1, and it will fully support this release.

TheJourney
Influencer
Posts: 14
Liked: 4 times
Joined: Dec 10, 2009 8:44 pm
Full Name: Sam Journagan
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by TheJourney » Dec 10, 2009 9:02 pm

Is 4.1 going to clean up this mess for us??

zeki
Enthusiast
Posts: 26
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Virtual Appliance mode leaving Snapshots

Post by zeki » Dec 10, 2009 9:04 pm

i hope... virtual appliance mode is not usable at the moment...

Locked

Who is online

Users browsing this forum: No registered users and 22 guests