Exchange server freezes when recommiting snapshot

[alctel]

Ok, this has been an issue for 6 months, and I raised with with Veeam and VMWare support and noone has a clue, so I'm hoping someone has seen something similar.

We have an exchange 2010 mailbox server that backs up fine. About once every 3-4 days though, it appears to recommit the snapshot fine, but then the machine itself is 'frozen' - you can't RDP into it, and accessing it through the console takes about 20 mins to log on, and once on there, you can't really do anything. It still sends and recieves mail for a short while, sometimes a couple of minites, sometimes a couple of hours, but then eventually stops. I can't do a soft reset, but a full power cycle fixes the VM.

We also have a file server that does the same thing, only even more rarely (once every 2 weeks).

At the moment I'm having to check the server at night after the backup each and every night and it's exhausting. Does anyone have any ideas? I've updated the firmware on everything, set it to a dedicated host, checked the host bios settings, tried on a different host and nothing seems to work.

[Vitaliy S.]

How large is the snapshot file when this issue occurs? is it similar to what you have during normal backup job runs? Also do you have any monitoring tool in place to check datastore performance at the time when your Exchange server hangs?

[nielsengelen]

Also on which type of storage are you running the Exchange server and which Exchange version is being used?

[TobiasF]

I had the same problem a few month ago.
The Problem was that I made a fullbackup of my Exchange 2010 via network mode (NBD) to a new Backup-Repository ( so the backup job was running for a few hours) after the backup was finished, the snapshot was very large and it took some hours to commit all the changes to the vmdk.
That process made a lot of I/O on my storage and on the ESXi Host too ... so the exchange wasn't reachable while the commit process was running.

You should bring more Informations of your Infrastructure:

How long is you Backupjob running?
Do you have Error logs on your Exchange? For example VSS errors?
Do you have a Exchange DAG?
Is Application Aware Processing enabled for that job?
which transport mode ? NBD, SAN or HotAdd?


Because your Exchange is freezing every day, you should check your Exchange Databases (EDB files) for Bad Checksum Errors! => eseutil /mh “NameOfYourDatabase.EDB”

[TobiasF]

AN addition Information:

If VSS is failing or the VEEAM Server can't access your Exchange to truncate the transaction logs, these logs are growing very fast and that need much disk space additionally to your Exchange Database.

An Interesting Information would be how much diskspace do you have on your System Partition where exchange 2010 is installed.

[alctel]

Also on which type of storage are you running the Exchange server and which Exchange version is being used?

Hi, it's Exchange 2010 running on SAN storage

How large is the snapshot file when this issue occurs? is it similar to what you have during normal backup job runs? Also do you have any monitoring tool in place to check datastore performance at the time when your Exchange server hangs?

We don't have any datastore monitoring, what would you suggest?

I had the same problem a few month ago.
The Problem was that I made a fullbackup of my Exchange 2010 via network mode (NBD) to a new Backup-Repository ( so the backup job was running for a few hours) after the backup was finished, the snapshot was very large and it took some hours to commit all the changes to the vmdk.
That process made a lot of I/O on my storage and on the ESXi Host too ... so the exchange wasn't reachable while the commit process was running.

You should bring more Informations of your Infrastructure:

How long is you Backupjob running?
Do you have Error logs on your Exchange? For example VSS errors?
Do you have a Exchange DAG?
Is Application Aware Processing enabled for that job?
which transport mode ? NBD, SAN or HotAdd?


Because your Exchange is freezing every day, you should check your Exchange Databases (EDB files) for Bad Checksum Errors! => eseutil /mh “NameOfYourDatabase.EDB”

The problem isn't restricted to exchange, but that's the server it happens most often to. The snapshot takes around 4-5 minutes to delete, and it all seems to work fine (snapshot removed and backup is fine) and I get a 'backup complete' message - it's just that every few days teh exchange server/file server will have hung.

Incrimental backup job takes around 20mins, full is around 12 hours.
I can't find any errors in the logs - the only thing is VSS timing out, which I'm not sure is a sympton (since the machine froze) or a cause.
No DAG.
AAP is enabled.
Transport mode is hotadd, but we tried NBD and the same thing.

I've attached a pic with the storage setup

[veremin]

Are the conditions at the time when Exchange servers freezes trying to commit snapshot are different from the time when it proceeds as expected? The major contributor might be a snapshot size. The snapshot is bigger than usually and Exchange takes longer to commit it. So, can you provide any information regarding difference in snapshots' sizes?

Also, it might be worth checking whether there are any activity scheduled inside given VM for that day, such as antivirus, defragmentation or other maintenance tasks, as those might affect dramatically the number of changed blocks.

Thanks.

[Vitaliy S.]

We don't have any datastore monitoring, what would you suggest?

You can either use vSphere Client performance tabs or free version of Veeam ONE, that can monitor snapshot size and age of the VMs and alert you on the latency issues that may affect snapshot consolidation process.