[ID#5156540]Snapshot left behind by Veeam Backup - vRDM LUNs

[habibalby]

Hello,

Veeam Backup was running fine, by mistake I stopped the job which was running on the Exchange 2007 VM. This exchange has got 1 VMDK for OS and 13 vRDM LUNs for Database.

VM-ctk.vmdk files left behind in the RDM Mapping LUN for every vRDM.
VM-ctk.vmdk file left behind in the LUN where the VMDK OS located.


Backup is no more can run due to UnFreeze Error, cannot create a Shadow copy.

ddb.deletable in all the CTK files are set to ddb.deletable = "true"

What if V2V cloning would be resolve the issue?

If V2V cloning done, what a about the vRDM will they be reconvert it to VMDK?

Thanks,

[Vitaliy S.]

Hello Hussain,

V2V cloning will convert your vRDM disks into VMDKs, see this topic for more info: http://communities.vmware.com/message/1748804

I would recommend using a python script mentioned in this topic to consolidate all your existing snapshots: Virtual Appliance mode leaving Snapshots

Thank you.

[habibalby]

hello,
Thanks but the python script indicates that the vmdk files should have ddb.deletable = false but in my case all the vmdks point to this particular VM is ddb.deletable = false which means are allowed to be deleted.

Beside that, I don't have a snapshot left in the directory of the VM. All I Have is ctk-vmdk which worries me.

[root@SX01 MAIL001]# ls
MAIL001-56e9e21b.vswp MAIL001.nvram MAIL001.vmxf vmware-56.log vmware.log
MAIL001-aux.xml MAIL001.vmdk vmware-0.log vmware-57.log
MAIL001-ctk.vmdk MAIL001.vmsd vmware-54.log vmware-58.log
MAIL001-flat.vmdk MAIL001.vmx vmware-55.log vmware-59.log
[root@SX01 MAIL001]#

RDM Mapping LUN:


[root@ESX01 MAIL001]# ls
MAIL001_10-ctk.vmdk MAIL001_14-rdm.vmdk MAIL001_2.vmdk MAIL001_7-ctk.vmdk
MAIL001_10-rdm.vmdk MAIL001_14.vmdk MAIL001_3-ctk.vmdk MAIL001_7-rdm.vmdk
MAIL001_10.vmdk MAIL001_15-ctk.vmdk MAIL001_3-rdm.vmdk MAIL001_7.vmdk
MAIL001_11-ctk.vmdk MAIL001_15-rdm.vmdk MAIL001_3.vmdk MAIL001_8-ctk.vmdk
MAIL001_11-rdm.vmdk MAIL001_15.vmdk MAIL001_4-ctk.vmdk MAIL001_8-rdm.vmdk
MAIL001_11.vmdk MAIL001_16-ctk.vmdk MAIL001_4-rdm.vmdk MAIL001_8.vmdk
MAIL001_12-ctk.vmdk MAIL001_16-rdm.vmdk MAIL001_4.vmdk MAIL001_9-ctk.vmdk
MAIL001_12-rdm.vmdk MAIL001_16.vmdk MAIL001_5-ctk.vmdk MAIL001_9-rdm.vmdk
MAIL001_12.vmdk MAIL001_1-ctk.vmdk MAIL001_5-rdm.vmdk MAIL001_9.vmdk
MAIL001_13-ctk.vmdk MAIL001_1-rdm.vmdk MAIL001_5.vmdk MAIL001-ctk.vmdk
MAIL001_13-rdm.vmdk MAIL001_1.vmdk MAIL001_6-ctk.vmdk MAIL001-rdm.vmdk
MAIL001_13.vmdk MAIL001_2-ctk.vmdk MAIL001_6-rdm.vmdk MAIL001.vmdk
MAIL001_14-ctk.vmdk MAIL001_2-rdm.vmdk MAIL001_6.vmdk
[root@SX01 MAIL001]#

[Vitaliy S.]

Beside that, I don't have a snapshot left in the directory of the VM. All I Have is ctk-vmdk which worries me.

Having CTK files on your datastore is absolutely normal as these files are used to keep track of changed blocks in your VM.

If CTK files are gone, the VMware API will let us know that change information is not available and we will ask VMware to re-enable change tracking back for this VM.

[habibalby]

Hello,

I have another support case with VMWare and they are saying the VSS it seems its gone

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {fecfee60-3ad5-4550-81a2-c8697a3e4c37}
State: [9] Failed
Last error: Timed out

[Vitaliy S.]

Not sure how VMware support team can assist you with Microsoft Exchange VSS writer , but If I were you I would look through Event logs on the Exchange server and try to match event IDs with Microsoft KB articles.

If it doesn't help, then it's better to reach Microsoft support team directly for assistance.

[habibalby]

Hello,

VMware will not help me with VSS issue, but from the logs they have seen and the way how I have descried the issue for them they pin-point the VSS is the issue, still they have to comeback and will see where this dilemma get me off

I'm trying to match the event IDs with Microsoft and other blogs, I have seen lots of people having this trouble and some of them mentioned Veeam product.

Let me see who it goes, if you have any tips or tricks can assist please let me know.

Highly appreciated.

[habibalby]

Here's the timestamps of the Veeam Job when it runs and Event IDs from Exchange Server. The Events clearly shows the VSS aborting the Database and timing out.

Code: Select all

Veeam:

Start time: 11/28/2011 12:52:15 PM
End time: 11/28/2011 12:56:01 PM
Duration: 0:03:45

Freezing guest operating system
Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{fecfee60-3ad5-4550-81a2-c8697a3e4c37}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].]


Exchange: Administrative Event.

Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:37
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Information Store (336) Shadow copy instance 12 aborted. 

Exchange Application Event:

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:03
Event ID:      9606
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 60193528-c79f-4e6a-a36e-74427522ad2d) has prepared for backup successfully. 

Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:14
Event ID:      2005
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Information Store (336) Shadow copy instance 12 starting. This will be a Full shadow copy. 

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:14
Event ID:      9811
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 12) has successfully prepared the database engine for a full or copy backup of storage group 'Second Storage Group'. The following 1 databases are mounted and will be backed up:
               Public Folder Database


Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:14
Event ID:      9811
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 12) has successfully prepared the database engine for a full or copy backup of storage group 'Staff-SG'. The following 4 databases are mounted and will be backed up:

                Staff-DB1-H
                Staff-DB2-N
                Staff-DB3-L
                Staff-DB4-O

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:14
Event ID:      9811
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 12) has successfully prepared the database engine for a full or copy backup of storage group 'Students-SG2'. The following 5 databases are mounted and will be backed up:

                Students-DB10
                Students-DB6
                Students-DB7
                Students-DB8
                Students-DB9

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:14
Event ID:      9811
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 12) has successfully prepared the database engine for a full or copy backup of storage group 'Students-SG'. The following 5 databases are mounted and will be backed up:

                Students-DB1
                Students-DB2
                Students-DB3
                Students-DB4
                Students-DB5

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:14
Event ID:      9608
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 60193528-c79f-4e6a-a36e-74427522ad2d:12) has prepared for Snapshot successfully. 

Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:14
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Information Store (336) Shadow copy instance 12 freeze started. 


Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:14
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
MSExchangeIS (336) Second Storage Group: Shadow copy instance 12 freeze started. 


Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:14
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
MSExchangeIS (336) Staff-SG: Shadow copy instance 12 freeze started.


Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:14
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
MSExchangeIS (336) Students-SG: Shadow copy instance 12 freeze started. 

Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:14
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
MSExchangeIS (336) Students-SG2: Shadow copy instance 12 freeze started. 

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:15
Event ID:      9610
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 60193528-c79f-4e6a-a36e-74427522ad2d:12) has frozen the storage group(s) successfully. 


Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:37
Event ID:      2003
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Information Store (336) Shadow copy instance 12 freeze ended. 

Log Name:      Application
Source:        ESE
Date:          28/11/2011 12:54:37
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Information Store (336) Shadow copy instance 12 aborted. 


Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:37
Event ID:      9614
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 60193528-c79f-4e6a-a36e-74427522ad2d:12) has aborted the backup successfully. 


Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2011 12:54:57
Event ID:      9648
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
Exchange VSS Writer (instance 60193528-c79f-4e6a-a36e-74427522ad2d:12) has processed the backup shutdown event successfully. 

Log Name:      Application
Source:        VSS
Date:          28/11/2011 12:57:57
Event ID:      8224
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MAIL001.domain.com
Description:
The VSS service is shutting down due to idle timeout.