Virtual Appliance mode leaving Snapshots

[floss]

Hi, can any one help on this ?

I have set up Veeam 4.0 in virtual appliance mode , i have seen multple backup run fine and i understand the VA will place the VM in snapshot mode , enable Changed log tracking , this snapshot will then be deleted and a VMDK-CTK file will be created. this VMDK-CTK file will be changed every time a backup runs and there will only ever be one per Virtual machine and the machine will nver be left in Snapshot mode between passes.

The problem i am seeing is backups are succesfully completing but if go into the VM directory i am seeing various snapshot files and CTK files that are linked, But which is strange if i go into Snapshot manager it displays no snapshots, it seems to me that the Snapshot Chain is geeting muddled

This means the VM has to be cloned to merge the snapshots as there is no way of deleting them. and this is a worry .. I raised a call with VMware and they could explain this

Is this something anyone else is aware of

[tsightler]

The CTK files are the change tracking files, they never go away. They "track the changes" (wow, descriptive huh?) that occur to the VMDK files between backups.

So, are you seeing "delta" files that are associated with your VMDKs? These are the "snapshots". Perhaps a directory listing of the files would clear things up a little so that we could see what you're talking about.

[floss]

I thanks for your reply , yes i am aware one CTK will always exist

This is the structure of the VM folder

CDDC-NEW-000001-ctk.vmdk CDDC-NEW-000003-ctk.vmdk CDDC-NEW-ctk.vmdk CDDC-NEW.vmxf vmware-36.log
CDDC-NEW-000001-delta.vmdk CDDC-NEW-000003-delta.vmdk CDDC-NEW-flat.vmdk vmware-31.log vmware.log
CDDC-NEW-000001.vmdk CDDC-NEW-000003.vmdk CDDC-NEW.nvram vmware-32.log
CDDC-NEW-000002-ctk.vmdk CDDC-NEW-103383ff.hlog CDDC-NEW.vmdk vmware-33.log
CDDC-NEW-000002-delta.vmdk CDDC-NEW-103383ff.vswp CDDC-NEW.vmsd vmware-34.log
CDDC-NEW-000002.vmdk CDDC-NEW-aux.xml CDDC-NEW.vmx vmware-35.log

This is a Cat of the VMSD file

.encoding = "UTF-8"
snapshot.lastUID = "8"
snapshot.numSnapshots = "0"
snapshot.current = "0"
snapshot0.uid = "8"
snapshot0.filename = "CDDC-NEW-Snapshot8.vmsn"
snapshot0.displayName = "Consolidate Helper- 0"
snapshot0.description = "Helper snapshot for online consolidate."
snapshot0.createTimeHigh = "292709"
snapshot0.createTimeLow = "1582675055"
snapshot0.numDisks = "1"
snapshot0.disk0.fileName = "CDDC-NEW-000003.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot.needConsolidate = "FALSE"

[tsightler]

There will be one CTK file for every virtual disk, not one per virtual machine. So you're right, you system appears to have a single disk, with multiple snapshots. Is it possible that the old VMDK files have simply been abandoned? An "ls -ltr" listing would be nice, along with the contents of the vmx file and the CDDC-NEW.vmdk file. It certainly appears that this system is in a very strange state, almost as if there was a consolidate that never finished, but the system thought that it did.

[Gostev]

I have heard from someone about very similar issue some time ago, as far as I remember it was only appearing on ESX 4.0 hosts upgraded from 3.5 - but did not exist for ESX 4.0 clean installs. Do you know if the host in question was clean install, or upgraded?

[floss]

HI Gostev

Yes this was a upgrade from 3.5 to 4.0 is this a issue that only appears in applaince mode? as i suspect its something a lot of people will see and could be a showstopper

Is there a fix or a workaround

Thanks

[Gostev]

Right now if this is only my assumption, I just heard a while back that ESX 4.0 hosts upgraded from ESX 3.5 have the same issue with another product/vendor. Sounds like the ESX upgrade procedure does not set or initialize something correctly. We will try to reproduce this internally tomorrow, and see if we can do something from our side.

[floss]

Thanks for your help - i can set up a web-ex if its any help...

[Gostev]

Please create support case for tracking purposes. We may need webex if we are not able to reproduce this in lab. Also, just in case - please let me know what ESX 3.5 update level you had before upgrading to ESX 4.0.

[floss]

Hi Yes i will , it was 3 node cluster 3.5 update 4 upgrade to 4.0 with the latest build via update manager

[floss]

Hi Gostev

Im not sure how your getting on with your testing but i think this may be relevent there is a issue with Vsphere whereby if you delete a LUN ,Vsphere sometimes does not clean up properly it can cause VMs 'to hang every 30 mins and cause issues with snapshots, basically the Hostd goes mad trying to failover ect as it still sees the LUNS ,the only way round this is to reboot every host in the cluster

http://communities.vmware.com/thread/21 ... 0&tstart=0

http://communities.vmware.com/thread/21 ... 5&tstart=0

I logged a call with Vmware and apparantly a fix is in QA. i rebooted our last in the cluster last night and ran some test backup and they all seem fine today
I may be wrong but it seems to be more than a coincedence so fingers crossed


Regards

[schaffeb]

I am unfortunately seeing this same issue. The VM that I was testing on has several of these abandoned snapshots (snapshot manager shows no snapshots). My hosts are all esx 4.0 (clean installs). This seems to be somewhat of a showstopper.

Should I create a support ticket for this issue?

Edit: We run NFS datastores, here's the output from "ls -la" on the datastore:

-rw------- 1 root root 1311232 2009-10-28 16:46 Flex-000001-ctk.vmdk
-rw------- 1 root root 67151872 2009-10-28 16:46 Flex-000001-delta.vmdk
-rw------- 1 root root 382 2009-10-29 13:42 Flex-000001.vmdk
-rw------- 1 root root 1311232 2009-10-28 17:12 Flex-000002-ctk.vmdk
-rw------- 1 root root 67151872 2009-10-28 17:12 Flex-000002-delta.vmdk
-rw------- 1 root root 389 2009-10-29 13:42 Flex-000002.vmdk
-rw------- 1 root root 1311232 2009-10-29 13:41 Flex-000003-ctk.vmdk
-rw------- 1 root root 167815168 2009-10-29 13:41 Flex-000003-delta.vmdk
-rw------- 1 root root 389 2009-10-29 13:42 Flex-000003.vmdk
-rw------- 1 root root 1311232 2009-11-01 07:31 Flex-000004-ctk.vmdk
-rw------- 1 root root 134260736 2009-11-03 09:08 Flex-000004-delta.vmdk
-rw------- 1 root root 388 2009-11-01 07:31 Flex-000004.vmdk
-rw-r--r-- 1 root root 37 2009-10-22 15:06 Flex-48c1910d.hlog
-rw------- 1 root root 1073741824 2009-08-05 11:46 Flex-48c1910d.vswp
-rw------- 1 root root 13 2009-06-15 10:18 Flex-aux.xml
-rw------- 1 root root 1311232 2009-10-28 16:35 Flex-ctk.vmdk
-rw------- 1 root root 21474836480 2009-10-28 16:35 Flex-flat.vmdk
-rw------- 1 root root 8684 2009-11-01 01:01 Flex.nvram
-rw------- 1 root root 654 2009-10-29 13:42 Flex.vmdk
-rw------- 1 root root 893 2009-11-01 07:31 Flex.vmsd
-rwxr-xr-x 1 root root 3420 2009-11-01 07:31 Flex.vmx
-rw------- 1 root root 259 2009-10-29 13:41 Flex.vmxf
-rwxrwxrwx 1 root root 84 2009-11-03 09:09 .lck-3601000000000000
-rwxrwxrwx 1 root root 84 2009-11-03 09:09 .lck-3a01000000000000
-rwxrwxrwx 1 root root 84 2009-11-03 09:09 .lck-3e01000000000000
-rwxrwxrwx 1 root root 84 2009-11-03 09:09 .lck-9900000000000000
-rwxrwxrwx 1 root root 84 2009-11-03 09:09 .lck-9f00000000000000
-rwxrwxrwx 1 root root 84 2009-11-03 09:09 .lck-b001000000000000
-rw-r--r-- 1 root root 103649 2009-08-17 09:37 vmware-4.log
-rw-r--r-- 1 root root 295524 2009-09-30 09:51 vmware-5.log
-rw-r--r-- 1 root root 79576 2009-10-01 12:11 vmware-6.log
-rw-r--r-- 1 root root 192967 2009-10-22 14:35 vmware-7.log
-rw-r--r-- 1 root root 80690 2009-10-22 14:57 vmware-8.log
-rw-r--r-- 1 root root 80721 2009-10-22 15:06 vmware-9.log
-rw-r--r-- 1 root root 250234 2009-11-01 07:31 vmware.log

[schaffeb]

Something interesting I thought I'd add. If I manually create a new snapshot through vclient interface I get three new files:

-rw------- 1 root root 1311232 2009-11-03 09:55 Flex-000005-ctk.vmdk
-rw------- 1 root root 16820224 2009-11-03 09:56 Flex-000005-delta.vmdk
-rw------- 1 root root 365 2009-11-03 09:55 Flex-000005.vmdk

Why is it creating at ctk file with a manual snapshot? This doesn't make sense to me.

[tsightler]

Well, it still needs to track changes in the snapshot file so this doesn't really surprise me. There's not really any difference between you taking a "manual" snapshot and the vStorage API taking a snapshot. That being said, it is quite concerning that your system obviously has snapshots even though vCenter doesn't show it. Would you be willing to post the contents of our vmx and vmsd file? I probably won't be able to help, but as an ESX admin, I'm mainly just curious.

[Gostev]

Should I create a support ticket for this issue?

Yes please, and it would help if when opening the support case, you also provide the following information:

Veeam Backup VM and Backed Up VM:
1. Are they located on different datastores?
2. VMX and VMDK descriptor files (you can obtain those using FastSCP functionality).
3. VM log files from both VMs.

[schaffeb]

I have started a support ticked. Hopefully we can get this issue resolved ASAP because I'm chomping at the bit to go to production!

[Gostev]

We believe we fully understand the issue at this time.

It will almost always happen if both of the following conditions are met:
• vStorage API "Virtual Appliance" mode is used.
• You are either replicating, or backing up to Linux server or another ESX host.

Seems that regular backups to locally attached storage or CIFS share are not affected.

In short, what happens is the following:
1. On job completion (or failure) Veeam Backup will issue VMware API command to remove snapshot.
2. VMware API will ask ESX to perform snapshot removal.
3. ESX will first clean up internal ESX configuration database from the snapshot record (not sure why this is not done only after snapshot is removed - design bug?)
3. ESX will then proceed to removing actual snapshot file.
4. In most cases, actual VMDK files will still be locked by hot-add process at this time (VMDKs are still "un-hot-adding"). Appears that there are no inter-process synchronization in this scenario, and if ESX cannot remove snapshot files immediately, it fails right away instead of waiting.
5. And so snapshot file removal fails.
6. But snapshot record is already gone from ESX configuration database, so snapshot files remain orphaned.

We have a fix in the works which should resolve this, it is in testing right now. Timeouts in right places, and some other magic should help to work around this.

[schaffeb]

Wow that was quick, Veeam support is fantastic!

[Gostev]

Fix is available through our support and will be included in the next update release.

[Felix]

You should be able to make the snapshot visible, by adding another on top of it. Then you cam commit both from snapshot manager. I have similar issue with stuck consolidate helper snapshots when aborting backup jobs in Veeam Backup 4. Btw. My server also were updare from 3.5 U4 to vSphere.

[floss]

HI adding another snapshot up on top and then commiting will not delete the orphaned snapshots(try it) Consolidate snapshots are a different problem and you are correct you can usually commit by adding another snapshot. I have spoke to VMware support about this and we have tried every possible action to merge and the only way you can merge is to clone the whole machine.

[Felix]

Hmm, now that you mention it, I see a similar issue on one VM, although I'm not using virtual appliance mode, but physical veeam backup host and vStorage API SAN/NBD with fallback:

Code: Select all

-rw------- 1 root root 962K Oct 30 02:53 SAPPRD64-000001-ctk.vmdk
-rw------- 1 root root  17M Oct 30 02:53 SAPPRD64-000001-delta.vmdk
-rw------- 1 root root  370 Oct 30 02:52 SAPPRD64-000001.vmdk
-rw------- 1 root root 962K Oct 30 03:09 SAPPRD64-000002-ctk.vmdk
-rw------- 1 root root  17M Oct 30 03:09 SAPPRD64-000002-delta.vmdk
-rw------- 1 root root  377 Oct 30 02:53 SAPPRD64-000002.vmdk
-rw------- 1 root root    0 Oct 30 23:36 SAPPRD64-0ac9d37a.vswp
-rw------- 1 root root 2.6M Oct 30 02:53 SAPPRD64_1-000001-ctk.vmdk
-rw------- 1 root root  17M Oct 30 02:53 SAPPRD64_1-000001-delta.vmdk
-rw------- 1 root root  376 Oct 30 02:52 SAPPRD64_1-000001.vmdk
-rw------- 1 root root 2.6M Oct 30 03:09 SAPPRD64_1-000002-ctk.vmdk
-rw------- 1 root root  33M Oct 30 03:09 SAPPRD64_1-000002-delta.vmdk
-rw------- 1 root root  383 Oct 30 02:53 SAPPRD64_1-000002.vmdk
-rw------- 1 root root 2.6M Nov  8 17:02 SAPPRD64_1-ctk.vmdk
-rw------- 1 root root  40G Nov  8 17:06 SAPPRD64_1-flat.vmdk
-rw------- 1 root root  566 Nov  8 17:02 SAPPRD64_1.vmdk
-rw------- 1 root root 6.3M Nov  8 17:02 SAPPRD64_2-ctk.vmdk
-rw------- 1 root root 100G Nov  8 11:31 SAPPRD64_2-flat.vmdk
-rw------- 1 root root  568 Nov  8 17:02 SAPPRD64_2.vmdk
-rw------- 1 root root 3.8M Oct 30 02:53 SAPPRD64_5-000001-ctk.vmdk
-rw------- 1 root root 122K Oct 30 02:52 SAPPRD64_5-000001-delta.vmdk
-rw------- 1 root root  377 Oct 30 02:52 SAPPRD64_5-000001.vmdk
-rw------- 1 root root 3.8M Oct 30 03:09 SAPPRD64_5-000002-ctk.vmdk
-rw------- 1 root root  17M Oct 30 03:09 SAPPRD64_5-000002-delta.vmdk
-rw------- 1 root root  384 Oct 30 03:08 SAPPRD64_5-000002.vmdk
-rw------- 1 root root 3.8M Nov  8 17:02 SAPPRD64_5-ctk.vmdk
-rw------- 1 root root  60G Nov  7 22:21 SAPPRD64_5-flat.vmdk
-rw------- 1 root root  567 Nov  8 17:02 SAPPRD64_5.vmdk
-rw------- 1 root root 4.7M Oct 30 02:53 SAPPRD64_6-000001-ctk.vmdk
-rw------- 1 root root  17M Oct 30 02:53 SAPPRD64_6-000001-delta.vmdk
-rw------- 1 root root  377 Oct 30 02:52 SAPPRD64_6-000001.vmdk
-rw------- 1 root root 4.7M Oct 30 03:09 SAPPRD64_6-000002-ctk.vmdk
-rw------- 1 root root  17M Oct 30 03:09 SAPPRD64_6-000002-delta.vmdk
-rw------- 1 root root  384 Oct 30 03:08 SAPPRD64_6-000002.vmdk
-rw------- 1 root root 4.7M Nov  8 17:02 SAPPRD64_6-ctk.vmdk
-rw------- 1 root root 150G Nov  8 01:37 SAPPRD64_6-flat.vmdk
-rw------- 1 root root  568 Nov  8 17:02 SAPPRD64_6.vmdk
-rw-r--r-- 1 root root   13 Nov  8 17:01 SAPPRD64-aux.xml
-rw------- 1 root root 962K Nov  8 17:02 SAPPRD64-ctk.vmdk
-rw------- 1 root root  16G Nov  8 17:05 SAPPRD64-flat.vmdk
-rw------- 1 root root 8.5K Nov  8 16:58 SAPPRD64.nvram
-rw------- 1 root root  562 Nov  8 17:02 SAPPRD64.vmdk
-rw------- 1 root root 2.3K Nov  8 17:02 SAPPRD64.vmsd
-rwxr-xr-x 1 root root 5.1K Nov  8 17:02 SAPPRD64.vmx
-rw------- 1 root root 1.9K Nov  8 01:38 SAPPRD64.vmxf
-rwxr-xr-x 1 root root 5.0K Jun  7 23:45 SAPPRD64.vmx.mybak
-rw-r--r-- 1 root root 7.4M Jul 12 17:08 vmware-26.log
-rw-r--r-- 1 root root 3.8M Jul 24 16:13 vmware-27.log
-rw-r--r-- 1 root root 7.0M Aug 23 18:11 vmware-28.log
-rw-r--r-- 1 root root  11M Oct  4 16:46 vmware-29.log
-rw-r--r-- 1 root root 4.5M Oct 30 03:09 vmware-30.log
-rw-r--r-- 1 root root 633K Oct 30 23:33 vmware-31.log
-rw-r--r-- 1 root root 2.0M Nov  8 17:01 vmware.log

There are some additional VMDKs on local storage:

Code: Select all

-rw------- 1 root root 289K Nov  8 17:02 SAPPRD64_1-ctk.vmdk
-rw------- 1 root root 4.5G Nov  8 17:12 SAPPRD64_1-flat.vmdk
-rw------- 1 root root  564 Nov  8 17:02 SAPPRD64_1.vmdk
-rw------- 1 root root 1.9M Nov  8 17:02 SAPPRD64-ctk.vmdk
-rw------- 1 root root  30G Nov  8 17:12 SAPPRD64-flat.vmdk
-rw------- 1 root root  562 Nov  8 17:03 SAPPRD64.vmdk

The SAPPRD64.vmsd file looks like this:

Code: Select all

.encoding = "UTF-8"
snapshot.lastUID = "1368"
snapshot.numSnapshots = "0"
snapshot.current = "0"
snapshot0.uid = "1368"
snapshot0.filename = "SAPPRD64-Snapshot1368.vmsn"
snapshot0.displayName = "Consolidate Helper- 6"
snapshot0.description = "Helper snapshot for online consolidate."
snapshot0.createTimeHigh = "292830"
snapshot0.createTimeLow = "866587753"
snapshot0.numDisks = "7"
snapshot0.disk0.fileName = "SAPPRD64.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot0.disk1.fileName = "SAPPRD64_1.vmdk"
snapshot0.disk1.node = "scsi1:1"
snapshot0.disk2.fileName = "SAPPRD64_2.vmdk"
snapshot0.disk2.node = "scsi1:2"
snapshot0.disk3.fileName = "/vmfs/volumes/489b6447-57886d6c-1fcb-00215acee85e/SAPPRD64/SAPPRD64.vmdk"
snapshot0.disk3.node = "scsi1:3"
snapshot0.disk4.fileName = "/vmfs/volumes/489b6447-57886d6c-1fcb-00215acee85e/SAPPRD64/SAPPRD64_1.vmdk"
snapshot0.disk4.node = "scsi1:4"
snapshot0.disk5.fileName = "SAPPRD64_5.vmdk"
snapshot0.disk5.node = "scsi1:5"

snapshot.needConsolidate = "FALSE"

snapshot0.disk2.mode = "0"
snapshot0.disk3.mode = "0"
snapshot0.disk4.mode = "0"

snapshot0.disk6.fileName = "SAPPRD64_6.vmdk"
snapshot0.disk6.node = "scsi1:6"

snapshot1.uid = "1302"
snapshot1.filename = "SAPPRD64-Snapshot1302.vmsn"
snapshot1.parent = "1301"
snapshot1.displayName = "VEEAM BACKUP TEMPORARY SNAPSHOT"
snapshot1.description = "Please do not delete this snapshot. It is being used by Veeam Backup."
snapshot1.createTimeHigh = "292637"
snapshot1.createTimeLow = "-1582424371"
snapshot1.numDisks = "7"
snapshot1.disk0.fileName = "SAPPRD64.vmdk"
snapshot1.disk0.node = "scsi0:0"
snapshot1.disk1.fileName = "SAPPRD64_1.vmdk"
snapshot1.disk1.node = "scsi1:1"
snapshot1.disk2.fileName = "SAPPRD64_2.vmdk"
snapshot1.disk2.node = "scsi1:2"
snapshot1.disk2.mode = "1"
snapshot1.disk3.fileName = "/vmfs/volumes/489b6447-57886d6c-1fcb-00215acee85e/SAPPRD64/SAPPRD64.vmdk"
snapshot1.disk3.node = "scsi1:3"
snapshot1.disk3.mode = "1"
snapshot1.disk4.fileName = "/vmfs/volumes/489b6447-57886d6c-1fcb-00215acee85e/SAPPRD64/SAPPRD64_1.vmdk"
snapshot1.disk4.node = "scsi1:4"
snapshot1.disk4.mode = "1"
snapshot1.disk5.fileName = "SAPPRD64_5.vmdk"
snapshot1.disk5.node = "scsi1:5"
snapshot1.disk6.fileName = "SAPPRD64_6.vmdk"
snapshot1.disk6.node = "scsi1:6"

Adding a new snapshot on the VM didn't reveal those two snapshots.

[Gostev]

Felix, actually all snapshots you have are from Oct 30, when I assume you have been experimenting with the product?

As for the 2 fresh snapshots (consolidate helper), you have already posted about this problem here:
Helper snapshot not removed when vStorage API job is stopped

If you are not using Virtual appliance mode right now, but still can confirm this issue with physical SAN/NBD host, then it is definitely some separate issue, so let's move this to a separate thread, otherwise this thread will be really confusing if we start mixing all the issues together.

[floss]

Hi Just to let to let everyone know i have tried to source this fix via support and it is not available as i speak to my advice would be Do not use Virtual Appliance mode for backup or replication until is it available or less you run the risk of deep snapshot issues

[Vitaliy S.]

Hello Stuart,

I've just double-checked with the support about the fix, you'll be sent it right now, it seems like it wasn't available for the moment when you've requested it.

Thank you

[floss]

Thanks have rec and installed all ok

[schaffeb]

Floss,

Does the fix work for you? I have installed the fix and I am still seeing the issue. Still working with support on this.

[Vitaliy S.]

Ben,

Yes, please work with our support team, I believe we will be able to determine the reason why it didn't work for you.

[floss]

Hi , it looks fine as we speak but it seems to me it occurs when you have VM with muliple vmdk's that this problem occurs , backups or replications with single disk VM's are fine

[zeki]

hello, same issue here, on the datastore we have 3-5 snapshots created and no chance to delete it from the gui. the only chance is to move the content with vmkfstool to a new vmdk file.... :/