Twice now my Exchange 2010 server crashed by a Veeam backup

[shanfont]

I know you Veeam guys will tell me different but I say it's Veeam doing the crashing of my server. I know that Veeam request VMware to create a snapshot and then remove it. During this time of removing it is when I had my VM crash. The first time was during the removal process of a snapshot I got a blue screen on my Exchange server that crashed my mailstore database. I had to get Microsoft involved to help with repairing it. After that I was terrified to use Veeam to backup my Exchange server but I had no choice and my transaction log files was filling up disk space. It backed up for me for about six successful times. But still during those times during the snapshot removal I would lose connection to our mail server for about a minute. This affects all of our 400+ users.

My Exchange server mailbox database is about 543GB's and it's volume is using iSCSI on a RAID 10. I was in the process of moving my current users from our mailstore database and move them to smaller mailstore datastores so I could eventually get rid of the large one. Unfortunately I out of time before I could accomplish this. Veeam was still running it's backup job. My logs was building up and I only had 14% disk space left. I started to expand the disk because I knew the backup job wouldn't finish in time to clear the log files. It wouldn't let me expand the disk using diskpart because I didn't have enough free space.

So I panicked and stopped the backup job which started the removal of the snapshot. In VMware task all I got was "In Progress" for the snapshot removal and creation. I called VMware to see if they could look at it with me and possibly cancel it. After looking at the logs it does show that it's still working and has no errors so far. It's been doing this for 24hrs and they said it could take days. They didn't want to end the task because it being an Exchange Database and risk possibly corrupting it. The technician even told me that they don't ever recommend taking snapshots of a server that is using critical databases while the VM is live.

Because of this we will probably be forced to go to a hosted Exchange and find another backup solution for VMware. I just hope that the task finally complete and I eventually get Exchange back up. So has anyone else had issues like this with large Exchange volumes 1TB or more?

One last thing. I had successful backups, six snapshots from 8/8/2012 at 10:00pm that I ran in SureBackup and got "Restore Point 'server name' is unrecoverable". So much for that idea.

Case # [ID#5209474] [ID#5206812]

[Gostev]

Hi Shane,

So has anyone else had issues like this with large Exchange volumes 1TB or more?

We have such Exchange internally at Veeam, and never had a crash. Also, considering that at least a few thousands of B&R users (out 45'000) are backing up Exchange 2010, I would expect crashes to be reported by more than 1 person for the past few years - if they were actually caused by our product.

[Gostev]

OK, so I read carefully your whole post.

It sounds like the issue you are having right now does not even have to deal with Veeam - just bad planning of Exchange maintenance project on your part.

Of course, you absolutely under no circumstances should create snapshots on VM when doing maintenance operations that cause massive changes to its virtual disks - such as moving multiple mailboxes from 1TB datastore. You have to remember that VMware snapshot is copy on write - every write to virtual disk goes into a separate snapshot file. This will cause the snapshot file to explode in size, and its commit will then take forever.

Besides, you should really have declared Exchange maintenance window before starting database maintenance project, and disabled user access to Exchange. Never a good idea to perform this kind of stuff on live Exchange, virtualized or not! And this would also have prevented transaction logs from growing.

Right now, your best bet would be to wait until snapshot removal finishes. To prevent transaction logs from growing, I recommend declaring Exchange maintenance immediately, and disabling user access to Exchange (it's Saturday night after all).

After that, I recommend the following:
1. Run full backup of the Exchange server to ensure you have a valid restore point. This is needed so you could rollback if your database migration project does not go well for whatever reason. Remember, you should not use regular snapshot for this purpose, or it will grow immense.
2. Perform required database maintenance operations (finish your migrations to smaller datastores, delete large database file).
3. Enable user access, and verify that Exchange is functioning normally and all new databases are online.
4. Perform another full backup.

Be sure to perform FULL backup in steps 1 and 4, because your Exchange maintenance also means that most of the virtual disk blocks are changed, so running incremental backup would result in immense incremental backup file - so, there is just no point to use incremental here.

As soon as your Exchange server is back on track with plenty of free space, see if you keep having the crash issue. Remember that NTFS performance quickly degrades when volume is close to running out of space (in your case, due to transaction logs overfilling it). This could have been one of the triggers for your Exchange instability during high I/O load periods (when VMware snapshot is removed after backup).