-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
WARNING!!
I had installed the hotfix and it seemed as though my replications were going fine (no orphaned snapshots). However, today I was performing a replica and I stopped it prior to completion. This seems to have left me with an orphaned snapshot. I performed another replication, and it went fine, however I still have the orphaned snapshot from the original canceled.
I'm really in a bad spot here as this is a huge and very important production VM. I urge caution to all those using Virtual Appliance mode, even with the hotfix. It looks like there are still issues to work out.
I had installed the hotfix and it seemed as though my replications were going fine (no orphaned snapshots). However, today I was performing a replica and I stopped it prior to completion. This seems to have left me with an orphaned snapshot. I performed another replication, and it went fine, however I still have the orphaned snapshot from the original canceled.
I'm really in a bad spot here as this is a huge and very important production VM. I urge caution to all those using Virtual Appliance mode, even with the hotfix. It looks like there are still issues to work out.
-
- Enthusiast
- Posts: 31
- Liked: never
- Joined: Feb 15, 2009 8:31 pm
- Contact:
Re: Virtual Appliance mode leaving Snapshots
I yes i would agree, i have applied the fix and another one of of our severs hung on a job and thus the snapshot trial become corrupted this was also in SAN mode, leaving me with no option than a full clone and downtime to sort out i still do not have the confidence to use this in production with tier one servers.
im no expert but it looks to me like there are issues around clean snapshot removals
Will this be looked into within the next update as its the only problem i can see with a super product
im no expert but it looks to me like there are issues around clean snapshot removals
Will this be looked into within the next update as its the only problem i can see with a super product
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
I just wanted to clarify (to avoid confusion and issues mix up), that orphaned snapshots on stop job operation is a separate issue we have been tracking in the following thread: Helper snapshot not removed when vStorage API job is stopped.
As I posted there in my latest post a few days ago, we have introduced significant changes in stop job algorithm in version 4.1 which addresses this issue. Because of scale of the changes, we will not be able to provide hot fix addressing this issue for version 4.0, it will be available in version 4.1 we are planning to make generally available first half of December (real soon).
For now, please avoid stopping jobs running in vStorage API mode (other modes are not affected) unless absolutely necessary. With changed block tracking, incremental job cycle is extremely fast and it is best to wait couple of minutes until it finishes - instead of stopping it.
As I posted there in my latest post a few days ago, we have introduced significant changes in stop job algorithm in version 4.1 which addresses this issue. Because of scale of the changes, we will not be able to provide hot fix addressing this issue for version 4.0, it will be available in version 4.1 we are planning to make generally available first half of December (real soon).
For now, please avoid stopping jobs running in vStorage API mode (other modes are not affected) unless absolutely necessary. With changed block tracking, incremental job cycle is extremely fast and it is best to wait couple of minutes until it finishes - instead of stopping it.
-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
I'm glad to hear that, and I eagerly look forward to the 4.1 update. I guess it was not clear to me when I received the hotfix that stopping jobs would still be an issue.
Is there any way to get out of this mess without a clone?
Thanks for the quick responses as always!
Is there any way to get out of this mess without a clone?
Thanks for the quick responses as always!
-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
As a side note, I don't think we may not be talking about the same issue Gosteve. I don't see a "consolidate helper-x" snapshot.
I don't see any snapshots in the snapshot manager at all.
I don't see any snapshots in the snapshot manager at all.
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Ben, it is the same issue but you are using the different vStorage API job modes. As for removing snapshot, I believe that creating additional snapshot on the same VM will reveal another snapshot and you should be able to remove it.
-
- Influencer
- Posts: 19
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
hello
i had this issue too. the only way to remove the snapshots was to write the actual snapshot f.e ...-00003.vmdk into a new vmdk file on the service console with vmkfstools
greets
zeki
i had this issue too. the only way to remove the snapshots was to write the actual snapshot f.e ...-00003.vmdk into a new vmdk file on the service console with vmkfstools
greets
zeki
-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Hi Zeki,
I would be so grateful if you could describe vmkfstools method in more detail! Sounds like a lifesaver.
Gostev, unfortunately, creating a new snapshot does not reveal the old snapshot.
I would be so grateful if you could describe vmkfstools method in more detail! Sounds like a lifesaver.
Gostev, unfortunately, creating a new snapshot does not reveal the old snapshot.
-
- Enthusiast
- Posts: 31
- Liked: never
- Joined: Feb 15, 2009 8:31 pm
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Hi I have looked into this and spoke to Vmware support and basically the adding another snapshot and deleteing will not sort out the chain, if you also run a cat on .vmsd file you will see odd info relating to snapshots
The the safest way is a full offline clone but you will need downtime, also > Gostev if you run a Vstorage API job and this hangs 99% of the time you have no option but stop the job, so this a risk i personally would not take until the 4.1 update
The the safest way is a full offline clone but you will need downtime, also > Gostev if you run a Vstorage API job and this hangs 99% of the time you have no option but stop the job, so this a risk i personally would not take until the 4.1 update
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Floss, we did not have any with hang job @ 99% reported past month since 4.0 release... if you are experiencing this, please contact our support so that we could add the fix for this in 4.1 as well.
-
- Influencer
- Posts: 19
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
hello schaffeb
to do this metho you have to shutdown the vm.
1. shutdown the vm
2. make a backup of the whole vm folder (if anything gets wrong)
3. detect what snapshot is actualy used -> properties of the vm, edit the disk (maybe you see server-000002.vmdk)
4. ssh to esx and go to /vmfs/volumes/yourshared storage/vm folder
5. vmkfstools -i server-000002.vmdk server-C.vmdk (takes some time....)
6. delete the disk you get the properties on 3.)
7. add a existing disk to the vm (server-C.vmdk)
thanks it. now you can start the VM and delete the unused server.vmdk, server-00001.vmdk, server-00002.vmdk, ....
greets
zeki
to do this metho you have to shutdown the vm.
1. shutdown the vm
2. make a backup of the whole vm folder (if anything gets wrong)
3. detect what snapshot is actualy used -> properties of the vm, edit the disk (maybe you see server-000002.vmdk)
4. ssh to esx and go to /vmfs/volumes/yourshared storage/vm folder
5. vmkfstools -i server-000002.vmdk server-C.vmdk (takes some time....)
6. delete the disk you get the properties on 3.)
7. add a existing disk to the vm (server-C.vmdk)
thanks it. now you can start the VM and delete the unused server.vmdk, server-00001.vmdk, server-00002.vmdk, ....
greets
zeki
-
- Enthusiast
- Posts: 31
- Liked: never
- Joined: Feb 15, 2009 8:31 pm
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Gostev , i think i may have explianend myself wrong i meant that WHEN the jobs hangs which is very rare, 99% of time you will need to forcefully stop the job,Gostev wrote:Floss, we did not have any with hang job @ 99% reported past month since 4.0 release... if you are experiencing this, please contact our support so that we could add the fix for this in 4.1 as well.
so waiting is not usually option
By the way i know this thread is slightly negative slant on Veeam 4.0 but this is the only issue i have seen which causes concern and also the support from you and your guys is first class and i cant fault it..
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Ah, OK - thanks for clarification.
Thank you, we try our best we do apologize for any inconveniences - v4 has been a huge shift in technologies (vSphere and ESX4, changed block tracking, vStorage API, Virtual Appliance mode, transactional backup storage) and a few issues did slip through lab and beta testing.
But all known issues are addressed in 4.1 and I am sure will be rock solid... so that we could concentrate on v5 and all the new goodness.
Thank you, we try our best we do apologize for any inconveniences - v4 has been a huge shift in technologies (vSphere and ESX4, changed block tracking, vStorage API, Virtual Appliance mode, transactional backup storage) and a few issues did slip through lab and beta testing.
But all known issues are addressed in 4.1 and I am sure will be rock solid... so that we could concentrate on v5 and all the new goodness.
-
- Influencer
- Posts: 19
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
nice nice
gostev, any news about the slow network mode from vmware?
http://www.vnotion.com/?p=38
greets
zeki
gostev, any news about the slow network mode from vmware?
http://www.vnotion.com/?p=38
greets
zeki
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
I just heard the fix did not make it into ESX4 Update 1. I guess I need to talk to VMware support engineer handling my support case... want to get official confirmation and to ask what happened and what are the plans.
-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Just an update on my snapshot saga...
I've tried everything and there really appears to be no way to get rid of these orphaned snapshots without a clone
Oh well, fortunately a clone will only take 4 hours. I thought it was going to take 10, so I'm not in as bad of shape as I thought. Veeam is great, but I cannot wait until 4.1 comes out!
I've tried everything and there really appears to be no way to get rid of these orphaned snapshots without a clone
Oh well, fortunately a clone will only take 4 hours. I thought it was going to take 10, so I'm not in as bad of shape as I thought. Veeam is great, but I cannot wait until 4.1 comes out!
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Nov 11, 2009 3:07 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
I have not read eveything on this thread but this
http://kb.vmware.com/selfservice/micros ... Id=1012383
fix all my issues with the ctk files
In short:
1) run the python script on your volumes (see link above)
2) create a snapshot for each of your VMs (easier with powershell if you have tens of VM)
3) delete all snapshot (easier with powershell if you have tens of VM)
4) no more ctk !
Worked for me like a charm
http://kb.vmware.com/selfservice/micros ... Id=1012383
fix all my issues with the ctk files
In short:
1) run the python script on your volumes (see link above)
2) create a snapshot for each of your VMs (easier with powershell if you have tens of VM)
3) delete all snapshot (easier with powershell if you have tens of VM)
4) no more ctk !
Worked for me like a charm
-
- Veteran
- Posts: 357
- Liked: 17 times
- Joined: Feb 13, 2009 10:13 am
- Full Name: Trevor Bell
- Location: Worcester UK
- Contact:
Re: Virtual Appliance mode leaving Snapshots
If you are GUI like myself an alternative way is this -:
1. Power down VM
2. Make a clone of that VM with auto power on enabled ( just to auto run after its created )
3. Delete original Vm after checking all is working with the clone
I did this all weekend on my 30 production Vm`s all went fine with no problems
Trev
1. Power down VM
2. Make a clone of that VM with auto power on enabled ( just to auto run after its created )
3. Delete original Vm after checking all is working with the clone
I did this all weekend on my 30 production Vm`s all went fine with no problems
Trev
-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Trevor,
Wish I could do this, unfortunately this VM has rapidly changing user data. So I'll lose all the data from when the original snapshot for the clone was taken.
Hopefully this python script will work, I'm going to give it a try.
Wish I could do this, unfortunately this VM has rapidly changing user data. So I'll lose all the data from when the original snapshot for the clone was taken.
Hopefully this python script will work, I'm going to give it a try.
-
- Service Provider
- Posts: 77
- Liked: 15 times
- Joined: Jun 03, 2009 7:45 am
- Full Name: Lars O Helle
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Check the vmdk files for the snapshots (the small 00000x.vmdk, not the delta! ) Look for a line with the text ddb.deletable
Is this line there? Is this set to "false" ? I had this issue with Vmware Data Recovery.
Fix: Power off VM. Change all the snapshot vmdk's from having the line ddb.deletable = "false" to ddb.deletable = "true"
You should now create a new snapshot. When doing a "delete all" in snapshot manager they should all go away (if there was ddb.deletable = "false" statements in the vmdk's)
by the way:
If you login by ssh (or console) you can issuse this command to look for open/orphaned snapshots: "find /vmfs/volumes/ -name \*delta\*"
More in this nice blog: http://blog.scottlowe.org/2009/06/29/sn ... -recovery/
Is this line there? Is this set to "false" ? I had this issue with Vmware Data Recovery.
Fix: Power off VM. Change all the snapshot vmdk's from having the line ddb.deletable = "false" to ddb.deletable = "true"
You should now create a new snapshot. When doing a "delete all" in snapshot manager they should all go away (if there was ddb.deletable = "false" statements in the vmdk's)
by the way:
If you login by ssh (or console) you can issuse this command to look for open/orphaned snapshots: "find /vmfs/volumes/ -name \*delta\*"
More in this nice blog: http://blog.scottlowe.org/2009/06/29/sn ... -recovery/
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Nov 11, 2009 3:07 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
To lohelle,
this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383
see 4 posts above
this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383
see 4 posts above
-
- Enthusiast
- Posts: 50
- Liked: never
- Joined: Apr 14, 2009 10:44 pm
- Full Name: Ben S
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Personne,
THANK YOU!
That script worked beautifully!!!!!!!!!!
THANK YOU!
That script worked beautifully!!!!!!!!!!
-
- Service Provider
- Posts: 77
- Liked: 15 times
- Joined: Jun 03, 2009 7:45 am
- Full Name: Lars O Helle
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Sorry, did not catch that this was the same solution. (did not click the link....)Personne wrote:To lohelle,
this is what this script does + more
http://kb.vmware.com/selfservice/micros ... Id=1012383
see 4 posts above
-
- Enthusiast
- Posts: 28
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Agh! I am coming into this thread with the same problem. I tested running Veeam 4.0 in Appliance Mode on several test boxes and everything seemed fine. Now I have orphaned snapshots on several production boxes. I found out about the problem when I went to Storage vMotion one of the machines and it complained about link-mode snapshots....
I am going to go with the cold Clone mode to fix this issue. I don't want to screw with merging/deleting snapshots. I have been there and that is a major pain plus risky.
Looking forward to 4.1 as well. Veeam should send an email to everyone not to use Appliance Mode for now....
Eric
I am going to go with the cold Clone mode to fix this issue. I don't want to screw with merging/deleting snapshots. I have been there and that is a major pain plus risky.
Looking forward to 4.1 as well. Veeam should send an email to everyone not to use Appliance Mode for now....
Eric
-
- Influencer
- Posts: 19
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
i wait for virtual mode till the BUG in vstorage API is solved...
@gostev: any news about this?
@gostev: any news about this?
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
VMware has just released vStorage API 1.1.1 with many bugs fixed, and we have incorporated these bits into Veeam Backup 4.1 release. Feedback from beta is great so far - we are planning to release 4.1 next week if the final testing goes well.
-
- Enthusiast
- Posts: 31
- Liked: never
- Joined: Feb 15, 2009 8:31 pm
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Hi
Is the Vstorage API 1.1.1 a standalone fix that can be downloaded or is it incorperated in Vsphere 4u1
Regards
Is the Vstorage API 1.1.1 a standalone fix that can be downloaded or is it incorperated in Vsphere 4u1
Regards
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Hello, vStorage API 1.1.1 is a set of VMware components that is included and redistributed with our product. You can download those components separately, but you cannot just drop them into Veeam Backup 4.0 installation, because of a number of dependencies between these components and Veeam Backup code.
Veeam Backup 4.1 will be shipped with vStorage API 1.1.1, and it will fully support this release.
Veeam Backup 4.1 will be shipped with vStorage API 1.1.1, and it will fully support this release.
-
- Influencer
- Posts: 22
- Liked: 4 times
- Joined: Dec 10, 2009 8:44 pm
- Full Name: Sam Journagan
- Contact:
Re: Virtual Appliance mode leaving Snapshots
Is 4.1 going to clean up this mess for us??
-
- Influencer
- Posts: 19
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Virtual Appliance mode leaving Snapshots
i hope... virtual appliance mode is not usable at the moment...
Who is online
Users browsing this forum: Google [Bot], yasufumi.matsumoto and 66 guests