Exchange backup problem

[jingxi02]

I haven't not able to get the Exchange to backup the database in consistent state. I ran the job with VSS integration enable and the VSS backup was successful. However, when I restore the Exchange database backup to the server by the file level restore mode. The database is in "Dirty Shut Down" state when I run the command "ESEUTIL /mh priv1.edb" to check the consistency of the database. The the database is in "Dirty Shutdown" state, it needs to be repaired by the command "ESEUTIL /p priv1.edb" before the database can be mounted on the server.
Here is the sample output.

C:\Temp>cd mdbdata

C:\Temp\mdbdata>dir
Volume in drive C has no label.
Volume Serial Number is E033-0B44

Directory of C:\Temp\mdbdata

08/19/2009 10:41 PM <DIR> .
08/19/2009 10:41 PM <DIR> ..
08/19/2009 10:40 PM 10,493,952 priv1.edb
08/19/2009 10:40 PM 4,202,496 priv1.stm
08/19/2009 10:40 PM 16,785,408 pub1.edb
08/19/2009 10:40 PM 2,105,344 pub1.stm
4 File(s) 33,587,200 bytes
2 Dir(s) 12,821,794,816 bytes free

C:\Temp\mdbdata>eseutil /mh priv1.edb

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
Database: priv1.edb

File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,11
Engine ulVersion: 0x620,11
Created ulVersion: 0x620,9
DB Signature: Create time:08/11/2009 01:45:42 Rand:2062566 Computer:
cbDbPage: 4096
dbtime: 66324 (0x10314)
State: Dirty Shutdown
Log Required: 6-6 (0x6-0x6)
Streaming File: Yes
Shadowed: Yes
Last Objid: 808
Scrub Dbtime: 0 (0x0)
Scrub Date: 00/00/1900 00:00:00
Repair Count: 0
Repair Date: 00/00/1900 00:00:00
Old Repair Count: 0
Last Consistent: (0x6,7E,196) 08/17/2009 18:12:15
Last Attach: (0x6,80,16) 08/17/2009 18:12:16
Last Detach: (0x0,0,0) 00/00/1900 00:00:00
Dbid: 1
Log Signature: Create time:08/11/2009 01:45:39 Rand:2066147 Computer:
OS Version: (5.2.3790 SP 2)

Previous Full Backup:
Log Gen: 4-4 (0x4-0x4)
Mark: (0x4,14D2,2E)
Mark: 08/13/2009 22:01:17

Previous Incremental Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00

Current Full Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00

cpgUpgrade55Format: 0
cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

ECC Fix Success Count: none
Old ECC Fix Success Count: none
ECC Fix Error Count: none
Old ECC Fix Error Count: none
Bad Checksum Error Count: none
Old bad Checksum Error Count: none

Operation completed successfully in 3.32 seconds.


C:\Temp\mdbdata>

When I look at the backup process in the Veeam console, it appears the backup job script create a VMware shapshot first, and than Quieces the VM by the VMware tools and finally VSS the OS inside the VM. After these three steps, the Veeam starts to backup the VMDK files.

I think when the script creates the VMware snapshot, it will stop disk witing to the VMDK files and create a seperate shatshot file to store all disk writing activity. Once the shapshot is created, nothing can write anything into the original VMDK files. The backup job will attempt to backup the "Frozen" VMDK files while it's not being written. When the backup is done, the script released the snapshot file and commits all changes back to the original VMDK file.

I believe the problem is the following:

1. When the snapshot is being created, the VSS hasn't started yet. If the VSS is not running, the Exchange database is in a Read/Write mode which is in an inconsisten state and stores in the original VMDK file. This VMDK file will be backed up later.
2. When the script starts the VSS process, it only VSS the active staging file which is the snapshot file. However, this snapshot file is not going to be backed up.


I was looking around the software to see if there is a way can make the VSS process happen before the snapshot process.

???

[Gostev]

Hello Leslie, I've split your post into the separate topic to keep the original post clean from technical support issues.

Your understanding of the backup process is incorrect - with Veeam Backup, VM snapshot is created only after Exchange VSS Writer reports us that Exchange VSS freeze was completed successfully, so at the time of snapshot creation Exchange is being fully quiesced including MDB files (unless there is some bug with Exchange VSS implementation - which I doubt - or some 3rd party tool on your server affects it).

Your problem must be unrelated to Veeam Backup or its VSS processing because it would be reported by now by other customers (Exchange VSS integration code is in the product for over a year now). Are you sure you are using Veeam VSS integration and not built-in VMware VSS? Feel free to open the case with our technical support to investigate, we should be able to investigate if Exchange freeze completed correctly before the snapshot was created.

[jingxi02]

The Exchange server that is being backed up is a new install. It's a single server with active directory and Exchange 2003 SP2. The antivirus software is TrendMicro. Very simple configuration. The Exchange MDB file is only 10MB. I have checked many times on the backup job to make sure the job is VSS enabled. The System log in the Gest also shows the Veeam VSS support started successfully. The job finished successfully without any error. It appears to be the job is setup correctly but the Exchange database is not able to backup cleanly. I'm going to setup a test server with only OS and the exchange. No Antivirus software just to make sure that has nothing in the issue.

[Gostev]

I do not have strong Exchange admin skills, but I did some Googling and read up on this "dirty shutdown" thing....

Can you clarify if you are seeing dirty shutdown mentions in the Application event log, like these:

Code: Select all

Log Name:      Application
Source:        Storage Group Consistency Check
Date:          9/25/2008 11:13:11 AM
Event ID:      200
Task Category: Database Header Validation
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      email.company.net
Description:
Instance 1: Database headers have been successfully validated. All
databases are in a dirty shutdown state. To bring these databases to a
clean shutdown state, log generations 8367 (0x000020af) to 8379
(0x000020bb) will be required.

Or you only dirty shutdown in the above-mentioned utility output? Because the latter might be normal for VSS freeze - while Exchange and MDBs are fully quiesced, the actual operating system and Exchange services where still running when snapshot was taken, while may be the dirty shutdown bit is only getting cleared when store.exe is properly shutdown, which is not happening during shadow copy processing.

Just thinking out loud - this may be the same things as when you are backing up regular Windows server with VSS, it will always report the improper shutdown on the first reboot just because at the time Windows was not designed with hot full-image backups in mind, and thus it considers presence of shutdown bit a problem despite the image was created with VSS freeze. VSS should really take care about this bit as well, may be we will see actually happening in future - I did not have a chance yet to test this on Windows 7 or Windows 2008 R2 because VMware does not support/recognize these OS yet.

[goodmantc]

I am experiencing this issue as well. I have a ticket open but thought that someone may have seen this before or knows something to try. Our EDB and STM files are huge nearly 150GB each so just throwing those around to test is quite a long process. I also see that it appears to be saying it requires a log file as well which I don't understand as this backup should be complete without needing any log files as long as I have the EDB and STM file right?

[Gostev]

Thomas, can you please clarify what issue exactly do you have, and when it happens?

[goodmantc]

As the original poster stated. eseutil says my database state is "Dirty Shutdown" I am having concerns that I would not be able to restore the database or would have to do a rebuild before it could be restored which would take hours to do on top of the hours it would take to write out the databases from the backup.

[Gostev]

Could this be your issue:
http://alexwho.com/blog/2009/09/exchang ... tdown.html

[tsightler]

I don't believe that this is an error. Exchange will always report the database in "Dirty Shutdown" state if you back it up without dismounting the store. This is that case even for "Exchange Aware" backup agents. Of more concern would be the "Log Required" entry. This should actually be the "active" log and should basically simply recover automatically when you attempt to mount it, just as if the system had "crashed" at that exact moment. If you are attempting to restore Exchange using a file level restore you should restore not just the MDB and STM files, but also the LOG files because the active log is needed to preform automatic recovery.

I believe that the original posters mistake was that he used a file level restore, but did not restore the logs. Yes the MDB file would still be "Shutdown Dirty" but, if he restored the logs as well, he should be able to bring it to "Clean Shutdown" state by simply running "eseutil /r /l /d" or by simply mounting it.