Veeam Exchange best practice

[loop]

We are about to start using Veeam and are keen to know if there are any best practices for using this product with Exchange 2003/2007.

Ideally we would like mailbox and message level backup, we have Backup Exec but would be happy to purchase something else if recommended.

The log issue previously mentioned in the forums is of concern to us. Is this still an issue?

Thanks.

Loop.

[Gostev]

Clint, what log issue are you talking about?

[loop]

Previous posts have menioned Exchange logs not been erased or being erased before backups are confirmed?

[Gostev]

Hmm, not sure what posts are you looking at...
Does Veeam commit exchange logs

[loop]

Hi, I was reading this thread:
Backing up Exchange 2003 with VSS

[Gostev]

This thread also confirms there are no issues... you have probably missed this post from the original poster.

Actually, after some additional testing, it seems that Veeam is indeed purging the Exchange 2003 logs via VSS. We're thinking that we were possibly seeing unpredictable behavior because we still had another product backing up Exchange as well. Since stopping the other products backup from running it appears that Veeam is properly purging Exchange log files.

[loop]

Ah, ok excellent. Do you have any recommendations regarding granular mailbox and message restoration? Is there a whitepaper etc?

Do you think we should use Backup Exec too?

[TrevorBell]

Loop,

In my scenario, i backup exchange 2003 with Veeam at 20:00 every night, but also do a BE 12.5 backup of exchange and a few non VM`d servers ( sql and oracle ) i like the granular restore with BE12.5 which is very easy to use.

Im awaiting the New Veeam Offering to hopefully sway my away from using BE for exchange for ever. ( i will also use Veeam for Oracle and SQL when the servers are out of warranty also )

Trev

[tsightler]

If you do not have access to any tools to perform mailbox/message/item level restores, and the functionality is important to you, then you should either use a product that has this feature, or purchase a product that can be used with Veeam. With use Quest Recovery Manager because it can open the EDB files straight from the mounted Veeam backups and restore items directly to Exchange. We simply install QRM directly on the Veeam server.

You could probably also use Veeam in concert with Exchange Recovery Groups, but it would require many more manual steps.

I don't know if Veeam still drops the logs as soon as the VSS snapshot is created though, I think this is a minor design flaw. To delete the logs before the backup is confirmed is non-ideal although admittedly not usually a problem.

[fredbloggs]

Veeam does indeed commit the logs, even if you don't backup the drives with Exchange databases on it. We have reverted to using BE 12.5 to backup Exchange as we need to know that the database is being backed up correctly. (Have had to turn off Veeam VSS processing for this, but the database is consistent in the BE backup so not a concern)

Have had a few customers in the past that experienced corrupt exchange stores, a proper Exchange backlup tool wouldn't purge the logs in this case so you can perform the recommended action of restoring the last good database backup and replaying the logs to ensure consistency. Veeam is unable to offer this test.

Not sure if Surebackup will as it doesn't seem to mention it will check the exchange database consistency before purging the logs, simply says it won't say the backup succeded if the exchange databases don't cvome on-line following backup. This is too late in my opinion.

With regards to message level restore, very rarely have a reason to use this as we have Exchange retention set to allow user recovery 30 days after deletion of a message. Though, it's still nice. Our exchange backups are primarily to allow recovery in case of a disaster.

[tsightler]

We also keep a 30 day Exchange retention and also have an archive server that logs each message which users can access their older mail and restore single items. We rarely need to restore individual Exchange items, but we have had a few cases, including some corruption of a public folder and a problem with some Windows Mobile phones that was deleting everyones "notes" that people didn't notice right away. Wasn't killer stuff, but was nice to be able to get it with minimal effort via Veeam and Quest Recovery Manager.

The commiting logs issue is a problem, but since Veeam is almost 100% successful in making a backup it seems like it's not a major flaw. I'm not sure I followed your reasoning as to why this would be a major problem. Assuming you had a backup that completed successfully, it would include the Exchange logs. The problem I have is that Veeam takes the snapshot, then purges the logs, then starts the backup of the snapshot. Assuming that the backup actually completes sucessfully, all is good, since the snapshot does contain the logs, but if for some reason that backup fails and the snapshot is removed, you can rerun the backup, but those logs would be lost forever. We use a simple script which monitors the log directory and rsyncs to a log server as soon as they drop. Pretty simple, and gives us a second place to get at the logs.

[fredbloggs]

Hi Tom,

Our problem with Veeam not confirming that the Exchange database is consistent before purging the logs is as follows.

Sometimes when the Exchange database is inconsistent it stays mounted, most users don't even notice. Tools like Backup Exec, DPM etc. will notice that the database is inconsistent and not purge the logs.

Tools like Veeam (and other competing VMware type backups) that don't use a real Exchange agent performing an eseutil check before purging the logs don't notice the database is corrupt.

Now, say on the 1st April my database becomes corrupt. Users continue working (except perhaps one user can't access an email) for the whole of April and Veeam keeps on backing up this corrupt database and purging the logs.
Now, I need to restart the Exchange store for some reason, maintenance whatever, and I can't because the database is corrupt.

My previous months worth of Veeam backups also contain the corrupt database and no logs as they have all been purged. The only repair option I have is something potentially destructive like eseutil /p which means my database is offline and may not works anymore (depending on where the corrupted table lies). Or possibly if I have every day I could recover all the logs from the individual backups and replay them, assumes I have all backups in the chain though.

MS recommendations (for something like Backup Exec) are to restore the last good backup database and then Exchange simply replays the logs as they haven't been committed and still remain on the server.

Hopefully that explains that a little further.

Now, with regards to SureBackup, i'm not convinced this will really resolve the issue. Whilst it will confirm my backup has succeeded / failed checksums or whatever, it has already purged the logs on my live system, which has also carried on receiving new email. I'm unlikely to notice until the next morning when I check my reports.

Where I go from there i'm not sure. Do I restore my Exchange database from the day before and the logs from the latest backup so that Exchange can then at least replay that day's worth of logs, but I'll need to ensure I don't restore the patch, chk file or remove any of the transaction logs since the backup started so hopefully I can recover to that point. Would just be tidier and easier not to purge the logs in the first place.

Again, any Exchange (and in all honesty SQL) backups should confirm that the backup has succeeded (either by mounting the backed up VM and testing) before purging the logs.

I have had too many customers where their databases have become corrupt because of failing hardware (block level), but continued working; the only ackowledgement being the BE reports telling them so.

If Surebackup doesn't perform this sort of check before purging the logs I still won't use it to backup my Exchange databases. I do sincerely hope it does as I really like the ideas of it.

[tsightler]

Sometimes when the Exchange database is inconsistent it stays mounted, most users don't even notice. Tools like Backup Exec, DPM etc. will notice that the database is inconsistent and not purge the logs.

I'd mostly agree, but probably change this to "Tools like BE and DPM might notice..." as in my experience they rarely do. I've seen BE happily backup corrupt Exchange datastores.

My previous months worth of Veeam backups also contain the corrupt database and no logs as they have all been purged. The only repair option I have is something potentially destructive like eseutil /p which means my database is offline and may not works anymore (depending on where the corrupted table lies). Or possibly if I have every day I could recover all the logs from the individual backups and replay them, assumes I have all backups in the chain though.

Once again, I think we're in almost 100% agreement. As long as no Veeam job failed, you could actually restore Exchange from 30 days before, and then use the FLR to get the logs from each backup in-between, but what a pain, not only that, but even a single failed Veeam job, and the chain is broken, that's why this is a problem and that's why we replicate our logs off to a secondary server for added protection. It would help at least a little if Veeam actually allowed some sort of retention policy for the logs, it still wouldn't cover your scenario 100% but if you could tell Veeam to only purge logs older than 7 days, it would make it much easier to use the FLR appliance to restore logs if you ever needed to roll weeks worth of logs due to corruption.

[Frosty]

This prompts me to ask: is there anything I could do in order to check whether any corruption has gotten into Exchange (e.g. a scheduled task / batch file process, that would email me if problems are detected)?

[SecondCity]

Just curious if there was ever really a resolution to this, as we're evaluating Veeam as a BackupExec replacement.

[Gostev]

Hi John, please clarify - resolution to what exactly? There are a few points discussed above.

[joergr]

maybe it would be a good idea for you guys to let a command line script run which executes eseutil /mh, eseutil /ml, eseutil /g and then paste the output to a logfile and then send this logfile to yourself via e.g. blat or some similar tool. that way you would notice if your exchange db got corrupted. and if you would use an if based on the %errorlevel% output you could even automate it. i myself have multiple exchange 2007 and 2010 servers running and had a single 1018 in 12 years. not too many

and to take a look in the future (but this is implemented today in exchange 2010): since e2010 ms provides an api to specially check the consistency of the snapshot. it is called CHKSGFILES and Microsoft calls this an alternative for eseutil pre/post backup check.

take a look:
http://msdn.microsoft.com/en-us/library/bb891802.aspx

[Gostev]

How about running such script as a part of SureBackup verification job on the backed up VM, without putting any stress the actual production Exchange VM, or delaying the backup process? If integrity check completes fine on Exchange server running of backed up image, then obviously original VM is fine as well

[joergr]

yeah, anton is right.

[SecondCity]

Hi Gostev,

Sorry, I was referring to issues where the transaction logs may be corrupted during backup.

John

[TrevorBell]

I currently run BE v12.5 for my exchange backups aswell as backing up the VM with Veeam. With v5 of Veeam they will offer the same file level recovery you get with BE. My advice is to stay with BE until V5 is released - thats when ill be moving over to Veeam for my exchange backups

hope that helps

[Gostev]

Sorry, I was referring to issues where the transaction logs may be corrupted during backup.

John, could you please elaborate? I have never heard of transaction logs corruption issues before. Could not find anything in this thread either.

[SecondCity]

Gostev,

I suppose it's more of the fear of corruption or of inaccurate backups than it is actual corruption. The idea that the transaction logs are applied first, then the backup commences...and there's no real checking of whether the database is corrupt after the fact.

The post from joergr (and your followup) seem to address it, really.

John

[Gostev]

The idea that the transaction logs are applied first, then the backup commences

Oh, I see what you are talking about. Yes, this is addressed in v5. We provide some new options around transaction logs handling, and one of the options is to only flush logs upon successful backup. The other two options are not to flush logs at all (in case you backup Exchange with more than one tool), or flush them immediately (which is the current v4 behavior).

[Neurobit]

Thanks Anton,

Can't wait for v5 to get here!

Paul

[SecondCity]

Spectacular, thanks Gostev! Echoing Paul: Can't wait for v5 to arrive!

[m.novelli]

Oh, I see what you are talking about. Yes, this is addressed in v5. We provide some new options around transaction logs handling, and one of the options is to only flush logs upon successful backup. The other two options are not to flush logs at all (in case you backup Exchange with more than one tool), or flush them immediately (which is the current v4 behavior).

Really really nice!

Now I had to disable VSS integration on Exchange VM because log flushing disrupted incremental backups with TSM and/or Backup Exec!

Many many thanks!!!

Marco

[vmmatty]

Is there a way to disable VSS at the VM level if you backup at a top level item like a folder? I've got a folder named Messaging which contains all VMs related to the Exchange environment, including mailbox servers, Hub Transport, etc. There doesn't appear to be a way to disable VSS on a per VM basis if VMs are in a folder. It seems like my only options are to disable it for the entire folder or create another folder just for VMs I don't want to use VSS on.

Does anyone know of an easier way?

[Gostev]

Yes, v5 provides this capability.

[vmmatty]

I can't wait for v5 as this is a production system. Is the only workaround to create separate folders in 4.1.1?