Comprehensive data protection for all workloads
Post Reply
fconil
Influencer
Posts: 13
Liked: never
Joined: Nov 24, 2009 12:43 am
Full Name: Francois Conil
Contact:

Replication job "freezes" VM

Post by fconil » Jun 01, 2010 5:27 am

System: 3 ESX server running Vsphere 4.0.0, build 164009
Veeam Backup and Replication 4.0


Hi,

We're trialing Veeam to set up a DR system, and it mostly does a great job, except for one server.

This particular server does a lot of disk writes and reads and does not play particularly nice with VSS (custom built application).
I don't know if it's a direct consequence or not, but at the end of the replication, when VSphere reconciles the snapshot, the VM freeze for 1min or more, with "flat line" disk and memory performance in the ESX performance tab, which in turn makes the clients connected to said server freeze and crash.

Anyone having similar problems? I didn't have the problem on a once a day replication schedule or even 4h schedule, and it started surfacing with a once every 2h schedule. It could just be the problem has always been present but probability helping, it surfaced as more people were susceptible of using their client at the end of the replication process. Is the one minute freeze normal for such a "small" snapshot? (total process takes less than 10 minutes on a 2h schedule, 15 on a 4h schedule)

antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Replication job "freezes" VM

Post by antivir » Jun 01, 2010 6:04 am

This is normal for deleting large snapshot. You should check "Safe removal for snapshots larger than 100 Mb". After that you will get freeze time = 2 pings for VM that does a lot of disk writes.

fconil
Influencer
Posts: 13
Liked: never
Joined: Nov 24, 2009 12:43 am
Full Name: Francois Conil
Contact:

Re: Replication job "freezes" VM

Post by fconil » Jun 01, 2010 6:19 am

It's already checked, and I'm running Vsphere which, AFAIK, does not need the safe removal option (as per another post in these forums: http://www.veeam.com/forums/viewtopic.p ... 662#p15662 )

Vitaliy S.
Product Manager
Posts: 22771
Liked: 1523 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication job "freezes" VM

Post by Vitaliy S. » Jun 01, 2010 9:40 am

Hello Francois,

Could you tell us if you have any existing snapshots on this server? If yes, please remove it, because it may cause some downtimes on VMs during the snapshot removal with ESX4. I would also recommend to open a support case with VMware to investigate why snapshot removal causes issues with VM going offline.

Additionally, you may look through an existing thread with similar issue:
http://www.veeam.com/forums/viewtopic.p ... hot+freeze

Hope it help!

rgs3
Enthusiast
Posts: 31
Liked: never
Joined: May 03, 2010 1:48 pm
Full Name: Richard Shilling
Contact:

Re: Replication job "freezes" VM

Post by rgs3 » Jun 01, 2010 4:16 pm

This happens to me too, regardless of snap size or how long the snap was open for. Except for me it happens to every VM which times out for about 30 seconds once the snap is being committed. This prevents me from doing replica's during the day.

Francois, Can I ask what kind of backend storage are you on? Is it NFS?

antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Replication job "freezes" VM

Post by antivir » Jun 01, 2010 7:54 pm

I have 500 Gb VM and 40 Gb delta-file after backup. Deleting takes about an hour. But freezing time is no more 2 pings.
I have seen long freezing on ESX 4 (w/o U1 and w/o safe removal). I still use safe removal.

antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Replication job "freezes" VM

Post by antivir » Jun 01, 2010 8:00 pm

rgs3 wrote:This prevents me from doing replica's during the day.
Replication of my VM takes whole day :)
And it is not only such VM.

fconil
Influencer
Posts: 13
Liked: never
Joined: Nov 24, 2009 12:43 am
Full Name: Francois Conil
Contact:

Re: Replication job "freezes" VM

Post by fconil » Jun 10, 2010 4:17 am

VM is replicated using the default settings (SAN with network failover)
Replication is to another ESX server available on the local network, with 5 SATA drives fitted in raid5 and dedicated raid card.

There is no other snapshot on the VM. I'll move the schedule around tonight so I can check on the snapshot size.
Snapshots are fairly small AFAIK, and the whole VM is 160G (but 80G out of it is just an archive disk whose contents change only once a month). The data that actually gets modified is less than 10G at the very most. Incrementals size is around 2-2.5G (once a day schedule)

I'm about to move the replica target off site for DR purposes and I'm getting worried about the freeze, and since we're still trialing the software, this adverse performance could very well cost us the purchase of the licence.

Complete replication job takes a couple hours, and as stated earlier, incrementals are completed well under 20min.

tsightler
VP, Product Management
Posts: 5381
Liked: 2214 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Replication job "freezes" VM

Post by tsightler » Jun 10, 2010 4:22 am

Do you get the same freeze when you take the snapshot with vCenter, leave it for a while, and then remove it? Veeam doesn't really do anything that vCenter doesn't do so if you're having performance issues during snapshot removal it should be the same with vCenter. If so, then you really should take the issue up with VMware.

fconil
Influencer
Posts: 13
Liked: never
Joined: Nov 24, 2009 12:43 am
Full Name: Francois Conil
Contact:

Re: Replication job "freezes" VM

Post by fconil » Jun 10, 2010 4:35 am

Seems that way.

What would be the "standard" options for a snapshot taken by veeam compared to esx? Does it snapshot memory?
I think I might have disabled vmware tools quiescence as a workaround.

Vitaliy S.
Product Manager
Posts: 22771
Liked: 1523 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication job "freezes" VM

Post by Vitaliy S. » Jun 10, 2010 7:10 am

Francois,

No, we do not snapshot memory for the VM. We snapshot VM in the same manner as ESX host/vCenter does (using the same VMware API call, CreateSnapshot) so the behavior cannot be different. I agree with Tom, this issue should be taken to VMware's support, they will be able to assist you with this issue better.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 54 guests