Discussions specific to the VMware vSphere hypervisor
Post Reply
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: VM hangs while committing the snapshot

Post by myrdin » Mar 26, 2014 4:02 am

Actually it was VEEAM that reenables CBT on the Virtual Machine VMX everytime you run the job! Ideas??

Edit: there is a nice switch on the job configuration "enable CBT automatically" that i had disabled...now to find a way to redisable the CBT without restarting the vm every single time!

myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: VM hangs while committing the snapshot

Post by myrdin » Mar 26, 2014 5:22 am

Actually no it doesnt work. Even with CBT confirmed disabled on both side:

- if i take snapshot in Vmware it works without losing packets when it commit the snapshot
- with Veeam i have at least 10 ping lost

ideas?

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] VM hangs while committing the snapshot

Post by foggy » Mar 26, 2014 5:36 am

myrdin wrote:V7 Patch 3 backing up a machine. As soon as Veeam sends command to remove the snapshot to Vcenter at the end of the backup run, the vm hangs all the way until the remove snapshot process is finished (about 10-20 seconds).
Short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.
myrdin wrote:Removing snapshot IN VMWARE takes less than a second. Removing snapshot in VEEAM, 20 seconds delay while the VM is unresponsive.
Have you kept the manually created snapshot as long as the backup of that VM takes prior to removing it, to let it grow in size accordingly?

myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Snapshot removal issues of a large VM

Post by myrdin » Mar 27, 2014 3:47 am

Hi Foggy and thanks

i am planning an upgrade to v4.1. i will let you know how it goes.

myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Snapshot removal issues of a large VM

Post by myrdin » Mar 27, 2014 11:34 pm 3 people like this post

Upgrade to ESXi 4.1 Update 3 fixed the issue, only 1 lost ping (which is normal). Anyone having the same issue, this happens of a bug with ESX 4.0 and datastore on NFS shares. The machine freezes while removing the snapshot.

I will do 100 push-ups with a bag full of backup tapes on my back as punishment for blaming Veeam!

johntnavi
Enthusiast
Posts: 27
Liked: 1 time
Joined: Apr 27, 2012 6:55 pm
Full Name: John T
Contact:

Re: Snapshot removal issues of a large VM

Post by johntnavi » Jun 02, 2014 6:11 pm

Hi,

We are experiencing the same symptoms with exchange 2010, Veeam 7.x installed so I was wondering if anyone has implemented the suggestions from vmware below:

http://kb.vmware.com/selfservice/micros ... Id=5962168

John

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jun 02, 2014 8:04 pm

Hi John,

I don't think that disabling application-aware image processing for Exchange VM and shutting down its services a good move. In this case you won't be able to use log truncation and application VSS writers, since VMware Tools provide VSS snapshot on file system layer, not the application. Furthermore, Exchange will still not be available during its services restart.

Thanks!

tom11011
Expert
Posts: 144
Liked: 2 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

[MERGED] Question about the safe snapshot removal feature

Post by tom11011 » Jun 04, 2014 2:40 pm

Hi Group,

If your vmware environment is greater than 3.5 Update2, does the veeam 7.0 replication setting "safe removal of snapshots larger than" option have any meaning or bearing?

Have a virtual machine that is momentarily freezing during the snapshot recombine process. ESXI 5.5 update 1 with veeam 7.0 latest patches.

Thanks

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » Jun 04, 2014 2:50 pm

Tom, this option indeed should not be used with newer vSphere versions. That said, short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

[MERGED] I/O is Frozen on the Database

Post by bentabbron » Jul 02, 2014 12:43 pm

Hi,
I've been informed of a problem by our DBA that when Veeam is doing its nighly backup I/O is frozen on our databases for over 30 seconds! :shock:
Has anyone come across similar times?
We are running version Veeam backup & rep v 7.0
This does seem like an excessive peroid of time for the DB to be frozen.

Thanks

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jul 02, 2014 12:55 pm

Hi Ben,

Does it happen during VM snapshot commit operations? If yes, then please review this thread for further details.

Thanks!

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 03, 2014 12:31 pm

Hi Vitaliy,
How would I check that?

Thanks

veremin
Product Manager
Posts: 17077
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » Jul 03, 2014 12:49 pm

Just correlate database freeze with the specific step shown in backup console or vSphere Client. Thanks.

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 04, 2014 9:53 am

Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.

veremin
Product Manager
Posts: 17077
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » Jul 04, 2014 11:00 am

What type of database you're talking about? Are you using Application Aware Image Processing or VMware quiescence option with pre-freeze/post-thaw scripts? Thanks.

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jul 04, 2014 11:03 am

Try to offload the datastore where this SQL Server resides, might help. BTW, what kind of storage do you have at the back-end? If you could use our integration with HP and upcoming for NetApp, then this could potentially address this VMware snapshot commit behavior.

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 04, 2014 12:25 pm

Application Image Aware processing for SQL server databases. Thanks

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 04, 2014 12:28 pm

Hi,
We are using HP 3 Par storage which is doing storage based Snapshots via Veeam.

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jul 04, 2014 1:06 pm

So you're backing up SQL Server VM from these storage snapshots and still experience this issue?
bentabbron wrote:Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.
Can you please check job logs and see what happens in these 5 minutes?

veremin
Product Manager
Posts: 17077
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » Jul 04, 2014 1:07 pm

Is this issue reproducible without Veeam presence? That's being said, what happens if you take a snapshot manually, using vSphere Client? Thanks.

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 04, 2014 2:41 pm

Hi Vitaliy,
Is it the Veeam job logs you would like me to check?

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jul 04, 2014 3:02 pm

Yep, job session logs.

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 08, 2014 12:27 pm

Is it the log file from C:\ProgramData\Veeam\Backup that I need to look at?

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jul 08, 2014 12:33 pm

Not really, I'm referring to VM backup session log that can be found in the Veeam backup console.

bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron » Jul 08, 2014 12:59 pm

Session tab, Statistics or Report?

veremin
Product Manager
Posts: 17077
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » Jul 08, 2014 1:08 pm

I believe Vitaliy was talking about "Session" tab where you can click on a given VM (on the left) and see the steps conducted. Thanks.

VasilyT
Novice
Posts: 4
Liked: never
Joined: Jul 05, 2014 6:55 am
Contact:

Re: Snapshot removal issues of a large VM

Post by VasilyT » Jul 09, 2014 5:40 pm

Hello,

A very intersting thread. We're currently experiencing minor stuns during both snapshot creation and removal, on all VMs. No heavy loads, so no long stuns - total stun periods are about 2-4 seconds per VM during backup, and the stun is not continious.
On a side note, I've found that additional vmdk drives increase stun times considerably.

We're using local storage with 12x10K RPM drives in RAID10 with 1Gb cache controller. VMs are snapshotted one at a time to decrease IOPS load.
We're running backups every 2 hours, and are getting very minor complaints currently. However, we'd like to plan for increased transaction load and/or low-RPO replication.

We've recently went from 6 to 12 drives and the difference in stun times is negligible. ESXi only creates one additional helper snapshot during commit, so I think we're hitting some kind of a design wall here.

My question is: In production, has anyone been able to minimize stun times dramatically, to like <500msec per snapshot create/remove cycle, at least with a light load VM? How did you acheive that? Is it just raw IOPS? What sort of storage have you used?

I've read the topic and understand the tips, but I'd really like to talk to someone who has succeeded in this.

bold_defender
Lurker
Posts: 1
Liked: never
Joined: Sep 27, 2011 3:21 pm
Full Name: Mel Nugent

Re: Snapshot removal issues of a large VM

Post by bold_defender » Jul 14, 2014 3:15 pm

We have same situation here running 7.0.0.771
Its across 2 datacentres on one campus as follows.
2 hosts in each datacentre.
One datacentre has iSCSI hosts and other has SAS.
About 20 Windows VMs in each datacentre.
There is a SQL server running database mirroring and when I back it up the snapshot consolidation is causing a timeout and database failover.
Database timeout is 10sec but we're seeing 20sec plus drops.
SQL server has 3 disks
100GB OS and Programs
580GB DB
1,000GB DBArchive
Veeam is doing hot add backup

Is this a limitation of the infrastructure? The storage is IBM 3700 on both side and the 580Gb is on SSD.

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Jul 14, 2014 3:59 pm

Yes, you can reproduce this situation by creating snapshot manually (keeping it for the same time your backup job usually runs) and then initiating snapshot commit procedure. I'm not sure whether it will be possible or not, but you could try to adjust SQL mirroring timeout values to make it less sensitive, should help.

royp
Enthusiast
Posts: 38
Liked: 2 times
Joined: Mar 27, 2013 12:01 pm
Full Name: Roy P
Location: Wiltshire, UK
Contact:

[MERGED] Lost Ping Question Veeam 7.0.0.871

Post by royp » Jul 15, 2014 1:20 pm

I have read that while you will lose a few ping packets while the veeam BnR snapshots are removed after the backup job.
We lose approx. 3 packets which is enough to effect our Centos 5.7 and Centos 6 machines that run or testing machinery, this locks them up and causes quite an issue.

Is there a way to prevent these drops in pings/network connectivity as these Centos machines are testing 24/7 and writing to a central repository.

Post Reply

Who is online

Users browsing this forum: evilaedmin, Google [Bot] and 12 guests