Replication job "freezes" VM

[fconil]

System: 3 ESX server running Vsphere 4.0.0, build 164009
Veeam Backup and Replication 4.0


Hi,

We're trialing Veeam to set up a DR system, and it mostly does a great job, except for one server.

This particular server does a lot of disk writes and reads and does not play particularly nice with VSS (custom built application).
I don't know if it's a direct consequence or not, but at the end of the replication, when VSphere reconciles the snapshot, the VM freeze for 1min or more, with "flat line" disk and memory performance in the ESX performance tab, which in turn makes the clients connected to said server freeze and crash.

Anyone having similar problems? I didn't have the problem on a once a day replication schedule or even 4h schedule, and it started surfacing with a once every 2h schedule. It could just be the problem has always been present but probability helping, it surfaced as more people were susceptible of using their client at the end of the replication process. Is the one minute freeze normal for such a "small" snapshot? (total process takes less than 10 minutes on a 2h schedule, 15 on a 4h schedule)

[antivir]

This is normal for deleting large snapshot. You should check "Safe removal for snapshots larger than 100 Mb". After that you will get freeze time = 2 pings for VM that does a lot of disk writes.

[fconil]

It's already checked, and I'm running Vsphere which, AFAIK, does not need the safe removal option (as per another post in these forums: http://www.veeam.com/forums/viewtopic.p ... 662#p15662 )

[Vitaliy S.]

Hello Francois,

Could you tell us if you have any existing snapshots on this server? If yes, please remove it, because it may cause some downtimes on VMs during the snapshot removal with ESX4. I would also recommend to open a support case with VMware to investigate why snapshot removal causes issues with VM going offline.

Additionally, you may look through an existing thread with similar issue:
http://www.veeam.com/forums/viewtopic.p ... hot+freeze

Hope it help!

[rgs3]

This happens to me too, regardless of snap size or how long the snap was open for. Except for me it happens to every VM which times out for about 30 seconds once the snap is being committed. This prevents me from doing replica's during the day.

Francois, Can I ask what kind of backend storage are you on? Is it NFS?

[antivir]

I have 500 Gb VM and 40 Gb delta-file after backup. Deleting takes about an hour. But freezing time is no more 2 pings.
I have seen long freezing on ESX 4 (w/o U1 and w/o safe removal). I still use safe removal.

[antivir]

This prevents me from doing replica's during the day.

Replication of my VM takes whole day
And it is not only such VM.

[fconil]

VM is replicated using the default settings (SAN with network failover)
Replication is to another ESX server available on the local network, with 5 SATA drives fitted in raid5 and dedicated raid card.

There is no other snapshot on the VM. I'll move the schedule around tonight so I can check on the snapshot size.
Snapshots are fairly small AFAIK, and the whole VM is 160G (but 80G out of it is just an archive disk whose contents change only once a month). The data that actually gets modified is less than 10G at the very most. Incrementals size is around 2-2.5G (once a day schedule)

I'm about to move the replica target off site for DR purposes and I'm getting worried about the freeze, and since we're still trialing the software, this adverse performance could very well cost us the purchase of the licence.

Complete replication job takes a couple hours, and as stated earlier, incrementals are completed well under 20min.

[tsightler]

Do you get the same freeze when you take the snapshot with vCenter, leave it for a while, and then remove it? Veeam doesn't really do anything that vCenter doesn't do so if you're having performance issues during snapshot removal it should be the same with vCenter. If so, then you really should take the issue up with VMware.

[fconil]

Seems that way.

What would be the "standard" options for a snapshot taken by veeam compared to esx? Does it snapshot memory?
I think I might have disabled vmware tools quiescence as a workaround.

[Vitaliy S.]

Francois,

No, we do not snapshot memory for the VM. We snapshot VM in the same manner as ESX host/vCenter does (using the same VMware API call, CreateSnapshot) so the behavior cannot be different. I agree with Tom, this issue should be taken to VMware's support, they will be able to assist you with this issue better.