Veeam replication job hangs Exchange server at end

[sixpack]

I've just done a replication job on an Exchange 2003 Vmware Guest and the Exchange server stopped responding for a couple of minutes at the end.
At the end of the replication, when it is removing the snapshot, the guest server grinds to a halt and it can't even be pinged. Just multiple ping timeouts.
It comes back online when the snapshot removal is finished, but user's Outlook is hanging in the meantime.

What is going on? I expect a slight performance decrease, but I'm sure its not supposed to drop off the LAN at the end of the backup.

Source: ESX 4.0 Update 1
Target: ESX 4.0
Guest: Windows 2003 32bit and Exchange 2003
Veeam: Vmware Backup and Replication 4.0

Regards

Andrew

[Vitaliy S.]

Hello Andrew,

I believe you'll see the same behaviour with doing snapshot and removing it after a while manually, this happens due to a lot of changes commiting to the source VMDK file, as Exchange Server is very high transactional. By the way, are you using safe snapshot removal feature available in Veeam Backup and Replication 4? This feature should reduce the downtime for your Exchange Server. Here is the useful topic, with Anton describing how this feature works: Safe snapshot removal

Thank you.

[sixpack]

I forgot to mention this, but I only replicated the system drive, the C: drive.

There were no databases on this drive and the snapshot wouldn't/shouldn't have been particularly big.

Also from that link it implies that version ESX 4 has the improved snapshot removal built in.

[sixpack]

Just realised even if you just backup the one drive the snapshot might snapshot all drives.

I've done a manual snapshot of the server and left it running for a couple of hours and removed it with no issues. I'll try using the Safe Snapshot Removal option as you say and see how it goes.

[Gostev]

Safe Snapshot Removal is not needed for ESX 4.0 U1 hosts (as described in the thread referenced above).

Also, there is no differences when the snapshot removal command is initiated manually through VIC, or by Veeam Backup. In both cases, the same VMware API command which initiates snapshot removal is called. Then ESX host performs actual snapshot removal and consolidation. Neither Veeam Backup nor VIC perform anything around actual snapshot removal processing - they only issue a request to remove the snapshot. So this should be reproducible w/out Veeam installed. Actually, VM log file will have all the relevant information including actual VM freeze times - I found it very helpful when I researched snapshot management stuff. Personally I never seen freezes longer than 3-4 sec with ESX4 even under heavy stress testing (snapshot removal while copying large files to VM).

One thing worth checking though: there is a bug in ESX4 which causes super long VM freezes with application timeouts on snapshot removal - if VM has large previously created snapshots (freeze happens while removing the most recent snapshot and is proportional to first snapshot size). I would check this first.

The amount of changes (snapshot size) could be very different in these two tests and this may have produced different results. Although even large snapshots removal should not result timeouts due to how large snapshot removal is implemented in ESX4. So I don't think this is an issue. Again, VM log file captures all this, and with that you can easily confirm what was different between the 2 experiments.

[pufferdude]

Gostev-

Where are the VM logs you speak of? I only see vCenter logs and they don't tell me much. I came here looking for an answer to this same problem I'm having, and saw this thread. I have been using veeam 4 to back up my Exchange 2007 VM for a while now, but it was only once per day at 1am. Yesterday I reconfigured the job to back up Exchange box every four hours, and since then every time the backup occurs it causes Outlook to disconnect briefly (seemingly at the end of the backup, during snapshot removal.)

One of these backups just occurred, and NOW I see there is still a "Consolidate Helper 0" snapshot on this server. Does this have anything to do with the ESX4 bug you refer to above? This host is ESX4 (but not U1), and I have no idea what/why the Consolidate Helper snapshot is there, and if I should now manually delete it or what...

Jim

[Gostev]

Yes, you should remove "Consolidate Helper 0" snapshot as it likely what is causing longer VM stun on snapshot removal for you.
As for VM log files, they are located next to all other VM files like *.VMX