Exchange Backup outside of Veeam

[mdornfeld]

We have very aggressive RPO's in regards to our Exchange 2007/2010 implementation, and while Veeam is the appropriate tool for the backup of lot of our servers, I'm questioning that is the correct tool for our Exchange backups given our situation.

Pretty much, it's a funciton of:
- Exchange generates a LOT of changes on the backend storage (most of our systems take 15 minutes with CBT, Exchange takes up to 8 hours)
- Highly aggressive RPO that's not being achieved
- Backend Storage that only performs at a certain level.
- Thus far, I have had no help in the way of e-mail retention policies or true quotas from the company. It is what it is.

Given this, I could break our Exchange system into a number of smaller servers to make them back up more quickly with Veeam, but I also want to explore other tools. Pretty soon, I'm going to be expected to implement message journaling, deleted message retrieval, etc.

I'm posting on Veeam's forum, because I would expect, that even though Veeam is a superior product, that any number of other admins are in the same boat, and I just wanted to get an idea of what you may be using in conjunction with Veeam to fill this need.

If Veeam doesn't like 3rd party products being posted on the forum, a private message would suffice.

Thank you all,
Matt

[Gostev]

Hi Matt, we will work fine along with most 3rd party products out there. There can be problems only with those which install own VSS providers (which "brake" VSS framework and so our own VSS processing), but not necessarily (must be tested).

While we are at it, what is your minimum acceptable RPO for Exchange that would make you not look for another solution? Now, I know 1 sec or something would be best, but if we talk reasonable numbers that you are willing to trade single solution and no management overhead for? I am asking because we have some ideas in mind on how we could help to reduce RPO quite dramatically down the road (not a part of v6 release though).

Thanks.

[mdornfeld]

Hey Anton,
At this point in time, we are shooting for 4 hours.

I'm all for single solution, so if Veeam could pull a rabbit out of it's hat that would be immaculate.

[wonderboy]

I'm facing the same issue with Exchange backups using veeam - the CBT simply does not work when dealing with Exchange databases.
I have a couple of clients using NETAPP to backup the Exchage database, which works very well, but I then cannot use Veeam to backup the VM itself.

[Gostev]

the CBT simply does not work when dealing with Exchange databases

Huh? Why is that?

[Gostev]

At this point in time, we are shooting for 4 hours.

Good to hear, because we are shooting for much less that that (by an order of magnitude)

[tsightler]

I'm curious as to what problems you are seeing? How big is this Exchange server and how big are the logs that it generates daily? Are you using reverse incremental or copying the data offsite? We have a reasonably busy Exchange server with ~600 users and typically a Veeam backup with CBT takes no more than an hour with forward incrementals, but I would agree that for replication or hourly backup, Veeam probably isn't going to get the job done.

Still, it's pretty easy to simply setup an archive process to copy the Exchange logs on a regular basis. We simply rsync our logs to a remote location every 15 minutes and backup with Veeam once a day. That way, to restore Exchange we simply restore the VM and replay the logs and we've met our 1 hour RPO and then some.

It would be interesting to know what your RTO is, as well as the size and number of users. Also, why not use Microsoft features such as DAG if your RPO is truly that tight? Exchange can be built into an exceptionally robust platform and can meet RPO's in the minutes quite easily so I'm curious as to exactly what's causing the concerns.

[mdornfeld]

Tom,
I use reverse incremental. Have you found forward incremental to work significantly more quickly?

I did some back of the napkin math, and it looks like for my sample period of the last 6 hours, we generate about .5GB/hr in logs during the day per server. More or less at night based on whatever Exchange defragging it's doing. The snapshot sizes while Veeam is doing the backup get up to around 11GB in size. Tom, I'm guessing you probably have significantly more horsepower in your backend storage if your incremental only takes an hour for a whole days worth of exchange data. As I mentioned, ours is not optimal (and I'm certainly not blaming Veeam for that).

That all said, you actually answered the question spot on, but instead of a fancy appliance, you just copy off the logs to another server to achieve your desired RPO. We might look at doing something as simple as that.

Thanks for the input.
Matt

[Gostev]

Have you found forward incremental to work significantly more quickly?

Sure, forward incremental is at least 3 times faster than reversed incremental (because of 3x less I/O on target). You have to pay for this during weekend run though, when new synthetic full is assembled

[mdornfeld]

Sure, forward incremental is at least 3 times faster than reversed incremental (because of 3x less I/O on target). You have to pay for this during weekend run though, when new synthetic full is assembled

That is very good to know. Thank you Anton. My disk already spends all day getting thrashed by 3 file servers assembling new synthetic fulls. Have you done any time studies of synthetic full re-assembly on Optimal vs. Best compression settings? (Assuming 8 high-end cores).

[Gostev]

You will not see any differences in synthetic full re-assembly performance between these 2 compression settings.

[tsightler]

I would say the performance increase might be closer to 5-6 times. Not just because of the reduced I/O, but also the fact that you're only doing writes to the target storage which means you get much less impact from latency. With reversed incrementals, unless you have target storage that is capable of very high IOPS, you'll quickly find yourself IOP bound, which has a very negative impact on read latency. With forward incrementals you have three times less I/O, and thus much lower IOPS. Our daily Exchange backups went from 6-8 hours with reversed incrementals , to 45-90 minutes.

Of course, as Anton stated, you still have to pay the IOP penalty sometime, but we found that building a synthetic full for a full week is still typically faster than a single incremental backup. My guess is you could meet a 4 hour RPO with just this change alone. Even if you had to run a "real" full once a week, a full backup probably takes less than four hours.

[wonderboy]

My Exchange daily backup currently takes over 4 hours using reversed incremental.

EX01 Success 03/04/2011 20:43:56 04/04/2011 00:55:16 500.00 GB 500.00 GB 34 MB/s 4:11:20

A normal Windows File server takes just 18 minutes.

DC01 Success 03/04/2011 18:02:17 03/04/2011 18:20:41 450.00 GB 450.00 GB 417 MB/s 0:18:23

From this (and 3 other instances of Veeam and Exchange 2010) I conclude that CBT is not as effective because of the database files changing.

If anyone has any methods of improving this, I'd love to hear them.

Thanks

[tsightler]

CBT is effective, but because of the significantly higher number of changed blocks (especially considering Veeam's large block size) running a reversed incremental for Exchange puts a lot of IOP stress on the target storage. I'd suggest trying forward incrementals for your Exchange backup.

[wonderboy]

Thanks, I will try that. I will set-up a separate job for the Exchange server using forward incremental.

[Gostev]

Make sure you have Exchange transaction logs truncation enabled in only one job.