Exchange "backed up" flag?

Availability for the Always-On Enterprise

Exchange "backed up" flag?

Veeam Logoby kallex » Mon Jan 04, 2010 11:45 pm

Hi!

I lost the thread where was it Gostev asked what are the reasons (if any) for not using Veeam as a sole backup tool for Exchange and alike mission critical servers.

One thing that came to mind that we bumped with few different in-machine image backuppers and with Exchange 2007 was that, the Exchange needs to know it has been backed up. Otherwise it will store some transaction logs all the way up to the level where it fills the disk.

We had a situation where Exchange disks got full and identified it as such that the Exchange clears the logs only after it recognizes the backed up state.


Let me know if you need more assistance, we can hook up with externally connected lab with Exchange set up (no ESX host access though), if you need to testdrive the backup flag.


Br,

Kalle
kallex
Lurker
 
Posts: 2
Liked: never
Joined: Sun Jan 03, 2010 9:46 am
Full Name: Kalle Launiala

Re: Exchange "backed up" flag?

Veeam Logoby Gostev » Tue Jan 05, 2010 12:51 am

Hello Kalle, with Veeam your Exchange transaction logs will be cleaned up after each backup (if you have Veeam VSS enabled). There are a few existing threads discussing this, here's the latest one for instance :D

Just please note that Exchange 2010 is not yet fully supported by Veeam VSS. Microsoft has changed a lot of things in Exchange 2010 VSS, and we are researching these changes right now.
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Exchange "backed up" flag?

Veeam Logoby m.novelli » Tue Jan 05, 2010 8:58 am

Well, I think that Exchange and Active Directory servers require not only a full VM backup but also a backup-aware copy of Information Store and NTDS database.

I'm a long date Windows Sysadmin and I would not recommed to restore a Domain Controller or an Exchange Server just using the latest full VM copy.

Just my 2 cents

Marco
m.novelli
Veeam ProPartner
 
Posts: 313
Liked: 37 times
Joined: Tue Dec 29, 2009 12:48 pm
Location: Asti - Italy
Full Name: Marco Novelli

Re: Exchange "backed up" flag?

Veeam Logoby tsightler » Tue Jan 05, 2010 2:22 pm

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn't just "take a VM copy", the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a "backup-aware copy of the Information Store and NTDS database", and it uses the Windows recommended VSS processes to achieve this.
tsightler
Veeam Software
 
Posts: 5090
Liked: 2012 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Exchange "backed up" flag?

Veeam Logoby m.novelli » Tue Jan 05, 2010 3:15 pm

Ok, let's assume you are right and you have full VM backup of your Domain Controllers (let's assume you have at least 2 DC in your domain)

Now imagine you have some damage to AD database (you deleted accidentally an OU or you imported a schema that created issues with your directory)

What is the procedure that you will follow to bring back AD to a consistent state?

Marco
m.novelli
Veeam ProPartner
 
Posts: 313
Liked: 37 times
Joined: Tue Dec 29, 2009 12:48 pm
Location: Asti - Italy
Full Name: Marco Novelli

Re: Exchange "backed up" flag?

Veeam Logoby tsightler » Tue Jan 05, 2010 4:36 pm

There are severals ways to do this based on whether you want to recover only the OU (non-authoritative restore of AD followed by authoritative restore leaf object), or revert your entire AD to a point before the OU (authoritative restore of entire AD). Also, are you willing to restore the entire Domain Controller or do you just want to restore AD itself. The procedures would vary based on your answers to these questions. The simplest is obviously to revert the entire domain controller/AD to the point before the error and then mark it as an authoritative restore. For that option the procedure is basically:

1. Restore the AD VM
2. Boot the VM in Directory Services Restore mode
3. Run 'ntdsutil' at the command prompt and type "authoritative restore" and then "restore database".
4. Reboot into normal mode. Since this server has been marked as authoritative for the domain it will replicate to the other domain controllers.

If you didn't want to restore the entire VM you'd need to boot the system up into Directory Service Restore mode and uses Veeam's File Level restore functionality to simply restore the NTDS and SYSVOL folders. The "System State" would not normally be needed in the Veeam restore scenario. The "SystemState" backup includes all of the components on which AD is dependent, for example system startup files, system registry, COM+ class registration database, File Replication service (the SYSVOL directory), Certificate Services database (if it is installed), Domain Name System (if it is installed), Cluster service (if it is installed). These are needed for a disaster recovery "Windows reinstall" AD restore, but with Veeam you would almost always restore the entire VM in a DR scenario, not attempt to do a Windows resinstall and then a "pick and choose" file level restore. The restore of the VM would restore the system state as well.

I really don't think restoring AD is that different when using other tools. Our previous backup tool used "SystemState" and and "Active Directory" backup API's but a restore still required booting the system into Directory Services Restore mode, restoring the "AD backup" and running ntdsutil commands to complete the restore. There might have been some check boxes that would run these command for you (it seems like there was a "Perform an Authoritative Restore" checkbox) but overall the procedure was about the same.

Please note that this is note that the above is not an attempt to provide a complete guide to AD recovery with Veeam, only to answer the question about "What is the procedure". The actual steps we would take would vary based on the nature of the issue we experienced, it's ability to be corrected without a restore (we consider the restore of AD to be a last-resort option). The point really is, all you truly need to restore AD is a consistent backup of the AD components, and Veeam, using VSS provides that.
tsightler
Veeam Software
 
Posts: 5090
Liked: 2012 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Exchange "backed up" flag?

Veeam Logoby m.novelli » Tue Jan 05, 2010 8:40 pm

Long story short: this procedure doesn't work for a restored Domain Controller from an image-level backup (Veeam Backup, VCB-integrated backup, SAN snapshot)

The DC will start with the netlogon service paused and in Event Viewer you will find the error "The Active Directory database has been restored using an unsupported restoration procedure"

Then you will not be able to autorithative restore the directory objects.

I've personally tested this procedure with Windows 2000 and Windows 2003 DC. Not yet on Windows 2008 DC.

Let's look at this KB: http://support.microsoft.com/kb/875495

This is the actual Microsoft recommendation about backupping AD related data: http://support.microsoft.com/kb/888794

"To roll back the contents of Active Directory to a previous point in time, restore a valid system state backup. A system state backup can be restored up to the tombstone lifetime number of days after the backup was performed. The backup must have also been made on the same operating system installation as the operating system that you are restoring.

Active Directory does not support other methods to roll back the contents of Active Directory. In particular, Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent."

I'm feeling the same recommendation exist for Exchange CCR and mirrorer SQL Servers

Marco
m.novelli
Veeam ProPartner
 
Posts: 313
Liked: 37 times
Joined: Tue Dec 29, 2009 12:48 pm
Location: Asti - Italy
Full Name: Marco Novelli

Re: Exchange "backed up" flag?

Veeam Logoby Gostev » Tue Jan 05, 2010 8:48 pm

Thank you very much Tom for taking your time to write this guide. I am on vacation until Jan 10th, which explains my very sporadic forum attendance lately. :mrgreen:

m.novelli wrote:Long story short: this procedure doesn't work for a restored Domain Controller from an image-level backup (Veeam Backup, VCB-integrated backup, SAN snapshot)

Marco, you are 100% correct, just restoring latest VM copy of DC for example will cause USN rollback which completely trashes your DC (you have to demote it, and go through pains of manually cleaning up references to old DC in AD configuration).

This is exactly why Veeam is shipping proprietary VSS integration module for "proper" backups and restores. Our agent executes automatically before the actual VM snapshot is created, if you have Veeam VSS enabled, of course. Just investigate the Windows Event log for DC/Exchange/SQL/etc. after backup with Veeam, and you will see the corresponding VSS events there. Also, after restore with Veeam, you will see events of successful shadow copy restore.

I actually have 1.5 years old videos for both scenarios (restoring simple DC VM copy, and restoring Veeam DC backup).
Preparing the test lab (2 DCs) (Windows 2003)
Restoring regular DC VM copy (ouch, USN rollback)
Restoring Veeam DC backup (feel the power of Veeam VSS)

While you are mentioning DC and Exchange, it should be noted that these two require even more complex VSS backup/restore approach than other VSS-aware apps, as Microsoft requires certain custom restore procedures performed for these applications to ensure successful restore. For example, DC should be first booted into the safe mode (Directory Services Restore mode) to ensure Active Directory files are not locked by additional processes like antivirus when VSS restore is being performed). This is something Veeam VSS also implements, and it is fully automated - as you can see from the video above, no manual steps are required (well, in case of non-authoritative restore, and Tom has already covered the authoritative one).

All this functionality is actually unique to Veeam among all image-level backups... yet another reason to choose Veeam - by the way, we have this functionality since 2008.
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Exchange "backed up" flag?

Veeam Logoby Gostev » Tue Jan 05, 2010 9:00 pm

m.novelli wrote:I've personally tested this procedure with Windows 2000

By the way, please keep in mind that Windows 2000 does not have VSS at all.
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Exchange "backed up" flag?

Veeam Logoby tsightler » Tue Jan 05, 2010 11:07 pm

m.novelli wrote:I've personally tested this procedure with Windows 2000 and Windows 2003 DC. Not yet on Windows 2008 DC.


So you personally tested this with the Veeam VSS agent enabled and working? That's not really possible with Windows 2000 since it doesn't have VSS. With Windows 2003 it does work, I've tested it without issue.

m.novelli wrote:Let's look at this KB: http://support.microsoft.com/kb/875495

This is the actual Microsoft recommendation about backupping AD related data: http://support.microsoft.com/kb/888794


I'm quite aware of the requirements for backup of AD and the Microsoft articles are correct, simply taking a snapshot and rolling it back is NOT a valid way to backup AD. Fortunately, that's also NOT what Veeam does. Assuming you enable it, Veeam uses VSS to place AD into a consistent state prior to taking the snapshot, the snapshot is then made, and then VSS is signaled to return to normal operations. When you preform a restore of the VM you boot the system into Directory Services Restore mode and, since VSS had the NTDS database in a consistent state, you CAN perform an authoritative restore.

Since you like to link to MS articles here's a like on VSS Backup and Restore of AD:
http://msdn.microsoft.com/en-us/library ... 85%29.aspx

Yes, Exchange and SQL have the same requirements, but assuming you have the VSS writers for these applications installed and they are working properly, and you use Veeam VSS agent, then yes they will be backed up in a consistent state as well. Notice that Exchange even purges the logs. Veeam doesn't do that itself, it signals the VSS writer that the "backup" was complete (actually just a snapshot) and the VSS writer purges the logs when it "unfreezes".
tsightler
Veeam Software
 
Posts: 5090
Liked: 2012 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Exchange "backed up" flag?

Veeam Logoby donikatz » Wed Jan 06, 2010 12:00 am

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I've performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I've also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven't had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it's just a matter of updating our runbook and training. Honestly, if there's one area you certainly don't need to lose sleep over with Veeam, it's with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier... ;)
donikatz
Expert
 
Posts: 124
Liked: never
Joined: Sun Jan 01, 2006 1:01 am

Re: Exchange "backed up" flag?

Veeam Logoby m.novelli » Wed Jan 06, 2010 10:40 am

Well guys, you are right
My direct restore experience was with a Windows 2000 DC (not supporting VSS) and with a Windows 2003 DC that now I suppose wasn't backupped with VSS integration

Marco
m.novelli
Veeam ProPartner
 
Posts: 313
Liked: 37 times
Joined: Tue Dec 29, 2009 12:48 pm
Location: Asti - Italy
Full Name: Marco Novelli

Re: Exchange "backed up" flag?

Veeam Logoby tsightler » Wed Jan 06, 2010 2:18 pm

BTW, I do want to say that I would never criticize the idea of using a secondary backup method for critical information. It never hurts to have more than one way to restore a system. For example, we use Oracle RMAN to backup many of our Oracle databases, even though most of them are already backed up with Veeam. This way the DBA's can preform their own restores using the technology that their familiar with, but we still have Veeam backups for DR restores of the entire system. We've used both methods for restores of production systems with great success.
tsightler
Veeam Software
 
Posts: 5090
Liked: 2012 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Exchange "backed up" flag?

Veeam Logoby donikatz » Wed Jan 06, 2010 2:53 pm

tsightler wrote:we use Oracle RMAN to backup many of our Oracle databases, even though most of them are already backed up with Veeam. This way the DBA's can preform their own restores using the technology that their familiar with, but we still have Veeam backups for DR restores of the entire system.


Tom-- Not to go too off-topic, but do you use an Oracle freeze/thaw script for the Veeam backups or do you rely on RMAN to recover if Oracle doesn't come back up clean? Also, what Oracle version do you use? We're in the process of optimizing our Oracle infrastructure (including P2Ving the last of our physical clusters), so very interested in your experience. Thanks!
donikatz
Expert
 
Posts: 124
Liked: never
Joined: Sun Jan 01, 2006 1:01 am

Re: Exchange "backed up" flag?

Veeam Logoby Gostev » Wed Jan 06, 2010 5:23 pm

Doni, if you are running Oracle on Windows, Oracle VSS is a good option - see here: How to backup Oracle. Also in this topic there are some commands mentioned which you can use instead of VSS with the pre-freeze/post-thaw scripts (for example, on Linux).
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], Google Feedfetcher and 32 guests