Exchange Backups

[withanh]

We ran into some issues with our Exchange server and I am doing a P2V on it tonight. So I'll be starting using Veeam for backup/replication on the Exchange box. It's Exch2003 Sp2 on Win03 Sp2. What kind of issues can I expect to see? I did see a recent thread about Exchange and VSS. Also, our Exchange datastores (4 total) total in teh 75GiB range if that makes any difference.

Thanks!

Darhl

[tsightler]

I can only think of two issues that you might see:

1. Exchange backups seems slow compared to everything else. We get anywhere from 30-50MB/sec on most of our VM's, and a few even higher. Backups of our Exchange 2003 SP2 system are much slower, generally 10MB/sec. Others have reported similar performance with their Exchange serve backups.

2. If an Exchange backup fails, the logs are lost forever. Veeam uses VSS, then takes a snapshot, then stops VSS. The snapshot obviously contains all of the Exchange logs, however, they are purged immediately after VSS is stopped. If the Veeam backup of that snapshots fails for any reason, those logs are lost forever, potentially impacting the granularity of your restore. We work around this by replicating our Exchange logs to another host every 15 mins and keeping ~7 days of logs.

[pizang]

1. I do not see this on my system. Exchange 2003 SP2 has the same rate as other VMs. Why it should be lower?

[withanh]

Good point on the Exchange logs, I was wondering about them.

I am also seeing slowness in the Exchange backup. Of course it's still the first backup on that server so it's backing up everything...the other servers on that same job backed up at 47MB, 47MB, 53MB, and 47MB. Exchange is currently still running and is at 13MB.

[withanh]

Something seems wrong with the Exchange backup. Watching the CPU on the backup server, it went into idle around 8:45 (Pacific) this morning. It's now over 90 minutes later and the remove snapshot job in my vCenter is still stuck at 95%. Exchange is sluggish to respond, RPC Averaged Latency is spiking pretty high as well, seen some in the 1500ms range.

Obviously I know that backing up during the production day is never a great idea, but I started this backup job as soon as we finished the P2V last night. The other VMs in the job appeared to backup nicely, just this one seems like its hanging.

Veeam status shows "Completing current object backup process" and shows time remaining as 00:00:00

Does anyone else have a similar experience with the backups on Exchange?

[pizang]

I have the same experience but with a backup of file server. Backup starts very slowly then it gets data pretty fast and finally it is stuck with remaining 00:00:00 time for 2 hours.
Veeam support answered that this is because of slow target - after I have change configuration a bit it went better.

I have to test it further. I had for example such a situation: this file server had 3 disks - one for system and two for data. When I tried to backup only first disk I noticed that performance is in 100-300KB/s instead of 30-50MB/s! Very strange. When I will find something I will send update to this board.

[tsightler]

1. I do not see this on my system. Exchange 2003 SP2 has the same rate as other VMs. Why it should be lower?

Quite a number of people have reported slower backups of Exchange than their other servers. I don't really know why it's slower, but I have a suspicion that it's because of the amount of changes that are spread all across the disk. Our Exchange server is not huge, about 350GB with a little over 500 users, but the amount of change to the database is significant and random. On a typical night we see about 75-100GB of changes written to the .vbr files, so basically each rollback is around 25-35% of the entire size of the backup. Having this much change each night seems to work against the backup performance. Perhaps if you have a smaller Exchange server, or a backup host with a lot of CPU, this isn't as big of a problem.

I believe that it is caused by the randomness of the changes because we also have a 250GB database server that sees a lot of daily, random changes, and it also backs up with poor performance. We have roughly 30 servers running Veeam backups now, and only these servers with lots of random changes seem to be worse.

[tsightler]

I have to test it further. I had for example such a situation: this file server had 3 disks - one for system and two for data. When I tried to backup only first disk I noticed that performance is in 100-300KB/s instead of 30-50MB/s! Very strange. When I will find something I will send update to this board.

Are you using VCB? I think if you're using VCB that it has to read/transfer all disk, even if you only select one disk and this leads to very slow reported performance since the time spent reading the disks that you're not backing up counts against the backup.

[pizang]

Yes, I am using VCB and for example initial backup of my FileServer had 51 MB/s rate. So rate in KB/s means that something is wrong. But as I said I have to check this as I have moved my VCB proxy and Veeam to 64bit environment.

About your problem with slow performance. We have pretty small Exchange as it has only 20GB database. However, Veeam reads all data and writes only changed one, does it? So it reads that same amount of data no matter how many changes they are (it will chanege in Veeam 4) . Source performance is not a matter here. What performance do you have during initial backup? It is pure change scenario.
Following we have proxy performance and target performance. Did you check your proxy CPU usage during backup? Check it or set a perfmon to trace it. Maybe your proxy is to slow when calculating delta files?
Last one: what target do you use?

[tsightler]

Yes, I am using VCB and for example initial backup of my FileServer had 51 MB/s rate. So rate in KB/s means that something is wrong. But as I said I have to check this as I have moved my VCB proxy and Veeam to 64bit environment.

Ok, that wasn't clear from you first post. You said when you tried to backup only one disk not, when I tried to run incremental of only one disk. So my understanding now is that the full of only 1 disk ran at 51 MB/s but the incremental of only 1 disk is KB/s. I agree, that's wrong.

About your problem with slow performance. We have pretty small Exchange as it has only 20GB database. However, Veeam reads all data and writes only changed one, does it? So it reads that same amount of data no matter how many changes they are (it will chanege in Veeam 4) . Source performance is not a matter here. What performance do you have during initial backup? It is pure change scenario.

Performance during initial backup was somewhere between 30-40MB/s so yes, it seems to be due to change. As I said, we also have a SQL server that sees lots of random changes and it's performance is also not very good for incrementals.

Following we have proxy performance and target performance. Did you check your proxy CPU usage during backup? Check it or set a perfmon to trace it. Maybe your proxy is to slow when calculating delta files?
Last one: what target do you use?

Backup speed doesn't change whether I use VCB proxy, or network agent, it's virtually exaclty the same. Our proxy backs up many servers, most in the 30-50MB/sec range, all from the same storage to the same target. The Exchange server, and SQL server with lots of changes, is always the slowest by far, generally in the 10MB/sec range for Exchange, and 13-15MB/sec for the SQL server. I really think that Veeam has some performance overhead when there are a lot of changed blocks. CPU usage during backup is minimal, generally less than 30%. Storage performance doesn't appear to be a problem. I can run many backups simultaneously and they'll all average 30-40MB/sec except for Exchange.

With only 20GB of Exchange store I don't know if that's enough to even notice the problem anyway. What the size of the vbk and vbr files for this server? My Exchange store alone is about 190GB, the stream file is 30GB, and we generate anywhere from 10GB-30GB of logs every day.

[Gostev]

Something seems wrong with the Exchange backup. Watching the CPU on the backup server, it went into idle around 8:45 (Pacific) this morning. It's now over 90 minutes later and the remove snapshot job in my vCenter is still stuck at 95%. Exchange is sluggish to respond, RPC Averaged Latency is spiking pretty high as well, seen some in the 1500ms range.

We had this issue in production couple of times past years. According to our research, VMware has occasional issues with snapshot removal (we have seen other people reporting the same issue on the internet). Key is to NOT interrupt the process, or you will screw up you Exchange VM like we did. It will eventually complete, but could take a few hours. Even if VIC reports that snapshot removal is complete, but VM is still sluggish, wait more - it is still doing stuff even though the operation is reported as completed.

BTW first time we have experienced the issue, Veeam Backup was not even involved. Our VI admin just issued snapshot removal command with VIC after noticing big snapshot in the latest Veeam Reporter Enterprise report (someone had accidentally created the snasphot on Exchange). He did not wait long enough (as we later found to be suggested workaround) and messed up Exchange VM while doing troubleshooting.

[Gostev]

More information on VCB, disk exclusions, and reported job performance rate:
Backup of System Disk goes extremely slow using VCB SAN Mode

[withanh]

We had this issue in production couple of times past years. According to our research, VMware has occasional issues with snapshot removal (we have seen other people reporting the same issue on the internet). Key is to NOT interrupt the process, or you will screw up you Exchange VM like we did. It will eventually complete, but could take a few hours. Even if VIC reports that snapshot removal is complete, but VM is still sluggish, wait more - it is still doing stuff even though the operation is reported as completed.

BTW first time we have experienced the issue, Veeam Backup was not even involved. Our VI admin just issued snapshot removal command with VIC after noticing big snapshot in the latest Veeam Reporter Enterprise report (someone had accidentally created the snasphot on Exchange). He did not wait long enough (as we later found to be suggested workaround) and messed up Exchange VM while doing troubleshooting.

Yeah, 2hr 45min today to release the snapshot. Your support suggested installing the hotfix from MS KB Article Number(s): 929774. I am doing that today, well, I'm cloning into a lab then running the hotfix on the clone then doing some snapshot grabs and removals to see if that changes anything.

Is there a way to change the order of a backup or replication job? Example, I have 6 VMs in one job and Exchange is the 5th one. I'd like Exchange to go first so the snap releases before the business day. The last two days the snap was still releasing until after 9am.

Another issue with the Exchange backup is the log files are not removing. There's 15GB of logs going back to 8/21. I thought those would go away after the backup was completed?

d

[Gostev]

Darhl, there is no ability to order VMs within the same job, but you could create separate job for your Exchange server.
But I am not sure what could cause Exchange not to remove logs... may be this issue is also specific to Exchange 2007?

[withanh]

Anton,

Add that to the wishlist ...the ability to order VMs in jobs.

If I create a new/separate job for Exchange, will it do its own full backup again, or will it be able to use the full backup from the previous job? My hunch is that it will create its own full backup again...

Does anyone else out there have a similar issue with Exchange not deleting the log files? How are you handling it?

d

[Gostev]

Darhl, we have talked about Exchange transaction logs in this thread very recently, and Tom reported that he is seeing logs properly flushed. May be this discussion will give you some tips.

[withanh]

I saw that thread when I was researching, it really doesn't tell me how to make it purge the Exchange Transaction Logs, all it really says is that it's supposed to. My Exchange has been a VM for 3 days and has been backed up 3 times, yet the Exchange logs from 8/21 (7 days ago) through today still persist. So if it's supposed to purge the logs and it's not, how do I make it purge the logs? As one of the posters in the other thread mentioned I could write a script that does that, but that always makes me nervous, I'd prefer the backup process to purge the logs when they have been committed...maybe I'm missing something...

[tsightler]

Does VSS show that it is working? You should see entries in the Windows Event Log on the server showing the VSS service being started and stopped and on my systems I see events about the Exchange Information Store Service being set to "disabled" and then "auto start" just 15-30 seconds later. The logs appear to be purged just after that window.

[withanh]

Yes, it shows properly as far as I can tell. According to Veeam, the backup started at 10:16:20 PM. The Exchange event logs show:

10:16:36 PM "The Volume Shadow Copy service entered the running state."
10:16:36 PM "The VeeamVssSupport service was successfully sent a start control"
10:16:36 PM "The VeeamVssSupport service entered the running state."
10:16:41 PM "The Microsoft Software Shadow Copy Provider service was successfully sent a start control."
10:16:41 PM "The Microsoft Software Shadow Copy Provider Service entered the running state."
10:16:42 PM "The start type of the Microsoft Exchange Information Store service was changed from auto start to disabled."
10:17:08 PM "The start type of the Microsoft Exchange Information Store service was changed from disabled to auto start."
10:17:11 PM "The VeeamVssSupport service was successfully sent a stop control."
10:17:11 PM "The VeeamVssSupport service entered the stopped state."
10:20:12 PM "The Volume Shadow Copy service entered the stopped state."

Exchange logs create time (obviously a subset), all dated 8/27/2009
9:53pm
10:02pm
10:10pm
10:18pm
10:22pm
10:33pm
10:38pm

Total of logs start 8/21/2009 9:49pm and continue to present time.
5631 log files

Based on what you're saying, I should not see any exchange logs prior to approximately 10:15pm last night, right?

[tsightler]

I can't really explain this, but I usually have a few logs from 5-10 minutes before the backups started. For example, tonight the backup started a 6:50PM, however, I have four logs between 6:45-6:50PM still on the filesystem.

Your event log output looks exactly like mine. Not really sure why yours isn't working. When we first started using Veeam we had exactly the same problem. We were not really sure what was causing but thought it might have been conflicting with the other backup program we were using (that was just a theory because it seemed to magically start working after we disabled the other backup program). I wish I had another suggestion for you but I can say that it does work in our environment so I know it's "supposed" to work. We're Exchange 2003 SP2, what version are you? What OS version? I think you said Exchange 2003 SP2 and Windows 2003 SP2 in your first post but then I thought I saw a mention of Exchange 2007.

[withanh]

We are Exchange 2003 Sp2, Windows 2003 Sp2 x32. The Exch2007 comment was about a Windows hotfix that Veeam support found for me about the VSS release taking "forever". My Exch server is 200GiB total space, 80GiB free space, 75GiB info store (4 database files). Backups on the Exchange server are running around 10 hours, 6MiB/sec. Crazy slow. The snapshot release today took 2hrs 45mins, that's part of the 10 hours. Not sure what the deal is. So I'm trying to clone Exchange into my lab network, but when I try to clone it, it tells me that it can't quiesce things, makes sense because it's so busy working. I'm going to try tonight after people quit using Exchange to see if I can clone it then to try that hotfix on.

[tsightler]

You might want to run "vssadmin list writers" from the command line to verify if the Exchange VSS Writer is properly registered. The list should include the "Microsoft Exchange Writer", it's state should be "[1] Stable", and it should not have any errors.

[tsightler]

We are Exchange 2003 Sp2, Windows 2003 Sp2 x32. The Exch2007 comment was about a Windows hotfix that Veeam support found for me about the VSS release taking "forever". My Exch server is 200GiB total space, 80GiB free space, 75GiB info store (4 database files). Backups on the Exchange server are running around 10 hours, 6MiB/sec. Crazy slow. The snapshot release today took 2hrs 45mins, that's part of the 10 hours. Not sure what the deal is. So I'm trying to clone Exchange into my lab network, but when I try to clone it, it tells me that it can't quiesce things, makes sense because it's so busy working. I'm going to try tonight after people quit using Exchange to see if I can clone it then to try that hotfix on.

So our setup is very similar. We're about 350GB total space, roughly 120GB free (for logs and growth), just over 200GB info store with 3 database files. Backups take about 10 hours and we see about 10MB/sec and takes about 10 hours. Our snapshot release doesn't take that long, typically 45-90 minutes based on how busy the server was during the backup. Removing snapshots feels like it has become much slower with recent versions of ESX server (we're 3.5U4), but it does have much less impact on the performance of the running VM so I assumed that was a tradeoff they decided to implement.

[withanh]

Wow, glad to hear it's not just my server. I'm thinking about running that on its own backup job since Anton says I can't force it to run first inside a job group.

vssadmin list writers shows no errors except for the Exchange one:

Code: Select all

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {7c76d67b-317a-4232-91b5-1109b88fc0bb}
   State: [9] Failed
   Last error: Retryable error

We're running vSphere 4, release build 169004 or whatever the release build is, no updates installed yet.

So other than "not good" what does that error mean?

[withanh]

Thought of a couple more questions, kind of off topic, but I'm trying to judge the load on your server compared to mine.

1. How many users/mailboxes?
I've got about 200, plus 100 Blackberries. They say that one Blackberry = approx 3.5 mailboxes as far as load/performance go, so I've effectively got 550 users worth of load on my mail server.

2. Do you limit your users mail box size?
I hate this, but I inherited it, so I've got no choice, but our mailbox limit is 1GiB. Way too big IMHO.

3. Do you use any mailbox archiving? We use Symantec Enterprise Vault. It works pretty well, but some users get lazy because of the large mailbox limit and the archiver. I have users that have over 28k items in their inbox. You can't tell me they "need" 28,000 items!

[tsightler]

Code: Select all

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {7c76d67b-317a-4232-91b5-1109b88fc0bb}
   State: [9] Failed
   Last error: Retryable error

I wish I could tell you what this error meant, but that's definitely the problem. If the VSS writer ends in error then it will not purge the logs. I don't know how realistic this is in your environment, but have you tried rebooting the server. We thought that our problem was fixed when we removed our old backup software, but maybe it was when we rebooted after removing the software. That error appears to be fairly common, a search for it will find KB articles for Symantic Back Exec and several other backup products as well. There are some registry keys that can be added to create trace files from the VSS services which might provide more insight. You might also want to make sure you have the latest VSS rollup patch for Windows 2003 installed.

1. How many users/mailboxes?
I've got about 200, plus 100 Blackberries. They say that one Blackberry = approx 3.5 mailboxes as far as load/performance go, so I've effectively got 550 users worth of load on my mail server.

We're about 550 users/mailboxes with ~130 Blackberries.

2. Do you limit your users mail box size?
I hate this, but I inherited it, so I've got no choice, but our mailbox limit is 1GiB. Way too big IMHO.

No limits at all.

3. Do you use any mailbox archiving? We use Symantec Enterprise Vault. It works pretty well, but some users get lazy because of the large mailbox limit and the archiver. I have users that have over 28k items in their inbox. You can't tell me they "need" 28,000 items!

We implemented archiving earlier this year using Quest Archive manager. It works well, we probably have 50 or more users that have 25k items or more, heck we got 4-5 users that are getting very close to 100k items. Before archiving several of those users had mailboxes that were nearly 10GiB and we had over 100 users who's mailboxes exceeded 1GiB.

We've debated within the group for years about implementing mailbox limits, but large mailboxes have caused us so little trouble, it's been difficult to justify doing anything. It became even more difficult to justify once Gmail started offering huge mailboxes for next to nothing. We do occasionally send out notes to the "Top 10" asking them to clean up and consider deleting items they don't need.

[withanh]

Code: Select all

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {7c76d67b-317a-4232-91b5-1109b88fc0bb}
   State: [9] Failed
   Last error: Retryable error

I wish I could tell you what this error meant, but that's definitely the problem. If the VSS writer ends in error then it will not purge the logs. I don't know how realistic this is in your environment, but have you tried rebooting the server. We thought that our problem was fixed when we removed our old backup software, but maybe it was when we rebooted after removing the software. That error appears to be fairly common, a search for it will find KB articles for Symantic Back Exec and several other backup products as well. There are some registry keys that can be added to create trace files from the VSS services which might provide more insight. You might also want to make sure you have the latest VSS rollup patch for Windows 2003 installed.

Well, that machine did/does have Backup Exec installed. Maybe I'll try uninstalling BE since we don't use it any more now that it is a VM.

Thanks for the answers about your Exchange Environment. Kind of surprised you don't limit mailboxes, especially with the 100k+ counts and 10GiB mailboxes, but if it works, why fix it right?

[withanh]

Uninstalling BackupExec and rebooting did not resolve the error. I posted on a Microsoft support forum, I will post back here when I get a resolution in case anyone else here has a similar issue.

d

[Gostev]

Thank you Darhl.

[tsightler]

Have you applied the VSS rollup patches from Microsoft? http://support.microsoft.com/kb/940349

You can turn on tracing for VSS and get some additional logging which may give you more clues. Do a search for "Troubleshooting Exchange VSS" to find the registry keys. Any chance that some Exchange logs were deleted manually? This can put the system in an inconsistent state because it's trying to delete logs that aren't there.