Exchange "backed up" flag?

[kallex]

Hi!

I lost the thread where was it Gostev asked what are the reasons (if any) for not using Veeam as a sole backup tool for Exchange and alike mission critical servers.

One thing that came to mind that we bumped with few different in-machine image backuppers and with Exchange 2007 was that, the Exchange needs to know it has been backed up. Otherwise it will store some transaction logs all the way up to the level where it fills the disk.

We had a situation where Exchange disks got full and identified it as such that the Exchange clears the logs only after it recognizes the backed up state.


Let me know if you need more assistance, we can hook up with externally connected lab with Exchange set up (no ESX host access though), if you need to testdrive the backup flag.


Br,

Kalle

[Gostev]

Hello Kalle, with Veeam your Exchange transaction logs will be cleaned up after each backup (if you have Veeam VSS enabled). There are a few existing threads discussing this, here's the latest one for instance

Just please note that Exchange 2010 is not yet fully supported by Veeam VSS. Microsoft has changed a lot of things in Exchange 2010 VSS, and we are researching these changes right now.

[m.novelli]

Well, I think that Exchange and Active Directory servers require not only a full VM backup but also a backup-aware copy of Information Store and NTDS database.

I'm a long date Windows Sysadmin and I would not recommed to restore a Domain Controller or an Exchange Server just using the latest full VM copy.

Just my 2 cents

Marco

[tsightler]

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn't just "take a VM copy", the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a "backup-aware copy of the Information Store and NTDS database", and it uses the Windows recommended VSS processes to achieve this.

[m.novelli]

Ok, let's assume you are right and you have full VM backup of your Domain Controllers (let's assume you have at least 2 DC in your domain)

Now imagine you have some damage to AD database (you deleted accidentally an OU or you imported a schema that created issues with your directory)

What is the procedure that you will follow to bring back AD to a consistent state?

Marco

[tsightler]

There are severals ways to do this based on whether you want to recover only the OU (non-authoritative restore of AD followed by authoritative restore leaf object), or revert your entire AD to a point before the OU (authoritative restore of entire AD). Also, are you willing to restore the entire Domain Controller or do you just want to restore AD itself. The procedures would vary based on your answers to these questions. The simplest is obviously to revert the entire domain controller/AD to the point before the error and then mark it as an authoritative restore. For that option the procedure is basically:

1. Restore the AD VM
2. Boot the VM in Directory Services Restore mode
3. Run 'ntdsutil' at the command prompt and type "authoritative restore" and then "restore database".
4. Reboot into normal mode. Since this server has been marked as authoritative for the domain it will replicate to the other domain controllers.

If you didn't want to restore the entire VM you'd need to boot the system up into Directory Service Restore mode and uses Veeam's File Level restore functionality to simply restore the NTDS and SYSVOL folders. The "System State" would not normally be needed in the Veeam restore scenario. The "SystemState" backup includes all of the components on which AD is dependent, for example system startup files, system registry, COM+ class registration database, File Replication service (the SYSVOL directory), Certificate Services database (if it is installed), Domain Name System (if it is installed), Cluster service (if it is installed). These are needed for a disaster recovery "Windows reinstall" AD restore, but with Veeam you would almost always restore the entire VM in a DR scenario, not attempt to do a Windows resinstall and then a "pick and choose" file level restore. The restore of the VM would restore the system state as well.

I really don't think restoring AD is that different when using other tools. Our previous backup tool used "SystemState" and and "Active Directory" backup API's but a restore still required booting the system into Directory Services Restore mode, restoring the "AD backup" and running ntdsutil commands to complete the restore. There might have been some check boxes that would run these command for you (it seems like there was a "Perform an Authoritative Restore" checkbox) but overall the procedure was about the same.

Please note that this is note that the above is not an attempt to provide a complete guide to AD recovery with Veeam, only to answer the question about "What is the procedure". The actual steps we would take would vary based on the nature of the issue we experienced, it's ability to be corrected without a restore (we consider the restore of AD to be a last-resort option). The point really is, all you truly need to restore AD is a consistent backup of the AD components, and Veeam, using VSS provides that.

[m.novelli]

Long story short: this procedure doesn't work for a restored Domain Controller from an image-level backup (Veeam Backup, VCB-integrated backup, SAN snapshot)

The DC will start with the netlogon service paused and in Event Viewer you will find the error "The Active Directory database has been restored using an unsupported restoration procedure"

Then you will not be able to autorithative restore the directory objects.

I've personally tested this procedure with Windows 2000 and Windows 2003 DC. Not yet on Windows 2008 DC.

Let's look at this KB: http://support.microsoft.com/kb/875495

This is the actual Microsoft recommendation about backupping AD related data: http://support.microsoft.com/kb/888794

"To roll back the contents of Active Directory to a previous point in time, restore a valid system state backup. A system state backup can be restored up to the tombstone lifetime number of days after the backup was performed. The backup must have also been made on the same operating system installation as the operating system that you are restoring.

Active Directory does not support other methods to roll back the contents of Active Directory. In particular, Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent."

I'm feeling the same recommendation exist for Exchange CCR and mirrorer SQL Servers

Marco

[Gostev]

Thank you very much Tom for taking your time to write this guide. I am on vacation until Jan 10th, which explains my very sporadic forum attendance lately.

Long story short: this procedure doesn't work for a restored Domain Controller from an image-level backup (Veeam Backup, VCB-integrated backup, SAN snapshot)

Marco, you are 100% correct, just restoring latest VM copy of DC for example will cause USN rollback which completely trashes your DC (you have to demote it, and go through pains of manually cleaning up references to old DC in AD configuration).

This is exactly why Veeam is shipping proprietary VSS integration module for "proper" backups and restores. Our agent executes automatically before the actual VM snapshot is created, if you have Veeam VSS enabled, of course. Just investigate the Windows Event log for DC/Exchange/SQL/etc. after backup with Veeam, and you will see the corresponding VSS events there. Also, after restore with Veeam, you will see events of successful shadow copy restore.

I actually have 1.5 years old videos for both scenarios (restoring simple DC VM copy, and restoring Veeam DC backup).
Preparing the test lab (2 DCs) (Windows 2003)
Restoring regular DC VM copy (ouch, USN rollback)
Restoring Veeam DC backup (feel the power of Veeam VSS)

While you are mentioning DC and Exchange, it should be noted that these two require even more complex VSS backup/restore approach than other VSS-aware apps, as Microsoft requires certain custom restore procedures performed for these applications to ensure successful restore. For example, DC should be first booted into the safe mode (Directory Services Restore mode) to ensure Active Directory files are not locked by additional processes like antivirus when VSS restore is being performed). This is something Veeam VSS also implements, and it is fully automated - as you can see from the video above, no manual steps are required (well, in case of non-authoritative restore, and Tom has already covered the authoritative one).

All this functionality is actually unique to Veeam among all image-level backups... yet another reason to choose Veeam - by the way, we have this functionality since 2008.

[Gostev]

I've personally tested this procedure with Windows 2000

By the way, please keep in mind that Windows 2000 does not have VSS at all.

[tsightler]

I've personally tested this procedure with Windows 2000 and Windows 2003 DC. Not yet on Windows 2008 DC.

So you personally tested this with the Veeam VSS agent enabled and working? That's not really possible with Windows 2000 since it doesn't have VSS. With Windows 2003 it does work, I've tested it without issue.

Let's look at this KB: http://support.microsoft.com/kb/875495

This is the actual Microsoft recommendation about backupping AD related data: http://support.microsoft.com/kb/888794

I'm quite aware of the requirements for backup of AD and the Microsoft articles are correct, simply taking a snapshot and rolling it back is NOT a valid way to backup AD. Fortunately, that's also NOT what Veeam does. Assuming you enable it, Veeam uses VSS to place AD into a consistent state prior to taking the snapshot, the snapshot is then made, and then VSS is signaled to return to normal operations. When you preform a restore of the VM you boot the system into Directory Services Restore mode and, since VSS had the NTDS database in a consistent state, you CAN perform an authoritative restore.

Since you like to link to MS articles here's a like on VSS Backup and Restore of AD:
http://msdn.microsoft.com/en-us/library ... 85%29.aspx

Yes, Exchange and SQL have the same requirements, but assuming you have the VSS writers for these applications installed and they are working properly, and you use Veeam VSS agent, then yes they will be backed up in a consistent state as well. Notice that Exchange even purges the logs. Veeam doesn't do that itself, it signals the VSS writer that the "backup" was complete (actually just a snapshot) and the VSS writer purges the logs when it "unfreezes".

[donikatz]

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I've performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I've also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven't had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it's just a matter of updating our runbook and training. Honestly, if there's one area you certainly don't need to lose sleep over with Veeam, it's with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier...

[m.novelli]

Well guys, you are right
My direct restore experience was with a Windows 2000 DC (not supporting VSS) and with a Windows 2003 DC that now I suppose wasn't backupped with VSS integration

Marco

[tsightler]

BTW, I do want to say that I would never criticize the idea of using a secondary backup method for critical information. It never hurts to have more than one way to restore a system. For example, we use Oracle RMAN to backup many of our Oracle databases, even though most of them are already backed up with Veeam. This way the DBA's can preform their own restores using the technology that their familiar with, but we still have Veeam backups for DR restores of the entire system. We've used both methods for restores of production systems with great success.

[donikatz]

we use Oracle RMAN to backup many of our Oracle databases, even though most of them are already backed up with Veeam. This way the DBA's can preform their own restores using the technology that their familiar with, but we still have Veeam backups for DR restores of the entire system.

Tom-- Not to go too off-topic, but do you use an Oracle freeze/thaw script for the Veeam backups or do you rely on RMAN to recover if Oracle doesn't come back up clean? Also, what Oracle version do you use? We're in the process of optimizing our Oracle infrastructure (including P2Ving the last of our physical clusters), so very interested in your experience. Thanks!

[Gostev]

Doni, if you are running Oracle on Windows, Oracle VSS is a good option - see here: How to backup Oracle. Also in this topic there are some commands mentioned which you can use instead of VSS with the pre-freeze/post-thaw scripts (for example, on Linux).

[donikatz]

Understood, thanks Anton, interested in how Tom's decided to deploy in his own production.

Tom-- In particular, what piqued my interest was your referring to RMAN as being used so DBA's can do their own thing. I infer that means you would otherwise find no need for RMAN with Veeam, in which case would you'd be using a freeze/thaw script (since I believe from other threads you're on RHEL5). Of course freeze/thaw scripts can be problematic in systems that require high availability, so another reason for RMAN could be so you don't have to use a freeze/thaw script. So I'm just curious how and why you're productionized.

Thanks!

[tsightler]

Yes, we run about 95% of our Oracle databases on RHEL5. We do not use freeze/thaw scripts as I think they are largely unnecessary. Prior to moving to a virtual environment we used storage based LUN snapshots to preform cold backups of our Oracle database. I found that, as long as the logs files were the last volume snapped, or as long as the entire system was snapped as an atomic operation, that media recovery was successful 100% of the time.

Several of our DBA's were skeptical of this practice. We wrote a script that would take our snapshots and mount them on a remote host and then start Oracle to preform a media recovery, then shutdown, then run a cold backup of the files. This process ran for years without a single media recovery failure. We would preform clones from the snapshots and they would work. After hundreds of successful recoveries from snapshot backups, the DBA's would generally become accepting. Still, we use RMAN because it allows them to monitor their own backups, decide their own backup strategies for different systems, perform restores to specific points in time, restore individual tablespaces, etc, etc. But do I feel like my Veeam backups are 100% restorable, yes. Just last week I restored 6 Oracle databases from almost 3 weeks ago after a failed 11g upgrade. I restored the 10g Oracle homes and all 6 database files using the FLR appliance, and they all fired up just fine after a quick media recovery.

I think the reason this works is because VMware snapshots are atomic. The entire machine is "stunned" at the instant the snapshot is taken, so there's really no risk of a partial write. This is safer than a physical machine that crashes because a physical machine could be in the middle of a write when it looses power, or could have a kernel panic or whatever, but even in most of those cases Oracle will recover with a simple media recovery.

Please note that I'm not saying this method is full proof, and I'm sure it's better to run a hot backup script than not to, but we've got about 25 Oracle servers, including busy ERP systems, and we've never had a problem with a snapshot backup and we just decided, based on a failure rate 0%, that it wasn't worth maintaining the scripts and they seemed to make the backup process more fragile.

The worst case scenario that we could come up with was the possibility of a smeared backup that automatic media recovery wouldn't be able to recover from (perhaps due to an SCN in a datafile newer than the log file). We figured even in this case we could revert to the previous nights backup, then use FLR to restore the next 24 hours of archive logs from the next night's backup, then preform a media recovery with the archive logs. This is trivial assuming you have the skills to begin with but might be too much if your not Oracle savvy.

[donikatz]

Tom, many thanks for the detailed response, very helpful! I've also found in Veeam restore tests that Oracle came up cleanly each time, I've just been reluctant to trust my limited results. Knowing you've had the same non-quiesced restore results with and without Veeam certainly makes me feel better. One concern I've had is with snapshot removal freeze. I've begun testing Veeam on our heaviest Oracle VM (we've been using it on lighter Oracle VMs for quite a while) and I'm finding that when the snapshot is removed, the i/o freeze is significant enough to impact our connecting application servers. Do you have the same issue with any of your Oracle VMs and how have you managed it? On a related note, have you started using CBT for periodic mid-day backups, and if so how have your results been in terms of differential size, speed, performance? Obviously everything is relative to environment, just curious if you're battling any issues and what approach you're taking.

(Note that we're still running Oracle 9i on RHEL4, but with the P2V of our remaining physical cluster, we're finally going to begin the move to 10g or 11g on RHEL5.)

Thanks again!

[tsightler]

In the past we have had some problems with dropped connections during a snapshot removal on some fairly busy VM's, but I haven't really seen this issue since VMware revamped their snapshot removal process after ESX 3.5 U2 (we currently run ESX4 not U1 yet). Our experiences with ESX4 snapshot removal has so far been flawless. That being said, our busiest Oracle database is still a physical server and is likely to stay that way for quite some time, not because we don't think VM's are completely solid, but because the system is 8 CPU's and VMware simply wants too much money for an Enterprise Plus license. We're a 24x7 shop, but we run our backups during the quietest portion of the night so perhaps that helps. In other words, it's not a problem we've experienced at all in recent years, but I can believe it might still be an issue for busy servers, especially if there's other IO demand on your storage. I've found that snapshot removal performance in ESX is heavily dependent on how fast your storage is at processing small block I/O, which is generally down to how busy it is, and how many spindles, and how much cache. We work hard to keep our busy servers on fast disk with plenty of IOP overhead and that seems to help.

We're replicating a few servers with Veeam, but they pretty light from a change perspective. For our Oracle databases we use DataGuard. Once again because it something the DBA's can take care of and monitor. It's uses far less bandwidth than Veeam, and generally keeps our Oracle databases virtually identical even across the WAN.

[donikatz]

Thanks again for the detailed response. Your Oracle deployment is much larger than ours, however we have Enterprise Plus (4.0 U1) and hope to test 8 vSMP for our largest system (the one yet to be P2Ved) when some new hardware arrives this month. Being spindle-challenged has been a problem and I've warned that we'll need more for the upcoming P2V -- I'm working on ROI for buy-in -- but our existing "heaviest" Oracle VM is really not THAT heavy and doesn't contend for storage I/O as much as some other systems do, so I've been a bit surprised by the snapshot removal issues. That said, I'm going to continue testing and perhaps this will help weight the ROI. Either that, or I'll find a mistake and fix it.