Host-based backup of VMware vSphere VMs.
Post Reply
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: VM hangs while committing the snapshot

Post by myrdin »

Actually it was VEEAM that reenables CBT on the Virtual Machine VMX everytime you run the job! Ideas??

Edit: there is a nice switch on the job configuration "enable CBT automatically" that i had disabled...now to find a way to redisable the CBT without restarting the vm every single time!
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: VM hangs while committing the snapshot

Post by myrdin »

Actually no it doesnt work. Even with CBT confirmed disabled on both side:

- if i take snapshot in Vmware it works without losing packets when it commit the snapshot
- with Veeam i have at least 10 ping lost

ideas?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] VM hangs while committing the snapshot

Post by foggy »

myrdin wrote:V7 Patch 3 backing up a machine. As soon as Veeam sends command to remove the snapshot to Vcenter at the end of the backup run, the vm hangs all the way until the remove snapshot process is finished (about 10-20 seconds).
Short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.
myrdin wrote:Removing snapshot IN VMWARE takes less than a second. Removing snapshot in VEEAM, 20 seconds delay while the VM is unresponsive.
Have you kept the manually created snapshot as long as the backup of that VM takes prior to removing it, to let it grow in size accordingly?
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Snapshot removal issues of a large VM

Post by myrdin »

Hi Foggy and thanks

i am planning an upgrade to v4.1. i will let you know how it goes.
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Snapshot removal issues of a large VM

Post by myrdin » 3 people like this post

Upgrade to ESXi 4.1 Update 3 fixed the issue, only 1 lost ping (which is normal). Anyone having the same issue, this happens of a bug with ESX 4.0 and datastore on NFS shares. The machine freezes while removing the snapshot.

I will do 100 push-ups with a bag full of backup tapes on my back as punishment for blaming Veeam!
johntnavi
Enthusiast
Posts: 27
Liked: 1 time
Joined: Apr 27, 2012 6:55 pm
Full Name: John T
Contact:

Re: Snapshot removal issues of a large VM

Post by johntnavi »

Hi,

We are experiencing the same symptoms with exchange 2010, Veeam 7.x installed so I was wondering if anyone has implemented the suggestions from vmware below:

http://kb.vmware.com/selfservice/micros ... Id=5962168

John
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Hi John,

I don't think that disabling application-aware image processing for Exchange VM and shutting down its services a good move. In this case you won't be able to use log truncation and application VSS writers, since VMware Tools provide VSS snapshot on file system layer, not the application. Furthermore, Exchange will still not be available during its services restart.

Thanks!
tom11011
Expert
Posts: 192
Liked: 9 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

[MERGED] Question about the safe snapshot removal feature

Post by tom11011 »

Hi Group,

If your vmware environment is greater than 3.5 Update2, does the veeam 7.0 replication setting "safe removal of snapshots larger than" option have any meaning or bearing?

Have a virtual machine that is momentarily freezing during the snapshot recombine process. ESXI 5.5 update 1 with veeam 7.0 latest patches.

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Tom, this option indeed should not be used with newer vSphere versions. That said, short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

[MERGED] I/O is Frozen on the Database

Post by bentabbron »

Hi,
I've been informed of a problem by our DBA that when Veeam is doing its nighly backup I/O is frozen on our databases for over 30 seconds! :shock:
Has anyone come across similar times?
We are running version Veeam backup & rep v 7.0
This does seem like an excessive peroid of time for the DB to be frozen.

Thanks
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Hi Ben,

Does it happen during VM snapshot commit operations? If yes, then please review this thread for further details.

Thanks!
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Hi Vitaliy,
How would I check that?

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

Just correlate database freeze with the specific step shown in backup console or vSphere Client. Thanks.
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

What type of database you're talking about? Are you using Application Aware Image Processing or VMware quiescence option with pre-freeze/post-thaw scripts? Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Try to offload the datastore where this SQL Server resides, might help. BTW, what kind of storage do you have at the back-end? If you could use our integration with HP and upcoming for NetApp, then this could potentially address this VMware snapshot commit behavior.
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Application Image Aware processing for SQL server databases. Thanks
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Hi,
We are using HP 3 Par storage which is doing storage based Snapshots via Veeam.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

So you're backing up SQL Server VM from these storage snapshots and still experience this issue?
bentabbron wrote:Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.
Can you please check job logs and see what happens in these 5 minutes?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

Is this issue reproducible without Veeam presence? That's being said, what happens if you take a snapshot manually, using vSphere Client? Thanks.
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Hi Vitaliy,
Is it the Veeam job logs you would like me to check?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Yep, job session logs.
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Is it the log file from C:\ProgramData\Veeam\Backup that I need to look at?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Not really, I'm referring to VM backup session log that can be found in the Veeam backup console.
bentabbron
Novice
Posts: 8
Liked: never
Joined: Jul 02, 2014 12:36 pm
Full Name: Ben Tabbron
Contact:

Re: Snapshot removal issues of a large VM

Post by bentabbron »

Session tab, Statistics or Report?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

I believe Vitaliy was talking about "Session" tab where you can click on a given VM (on the left) and see the steps conducted. Thanks.
VasilyT
Novice
Posts: 4
Liked: never
Joined: Jul 05, 2014 6:55 am
Contact:

Re: Snapshot removal issues of a large VM

Post by VasilyT »

Hello,

A very intersting thread. We're currently experiencing minor stuns during both snapshot creation and removal, on all VMs. No heavy loads, so no long stuns - total stun periods are about 2-4 seconds per VM during backup, and the stun is not continious.
On a side note, I've found that additional vmdk drives increase stun times considerably.

We're using local storage with 12x10K RPM drives in RAID10 with 1Gb cache controller. VMs are snapshotted one at a time to decrease IOPS load.
We're running backups every 2 hours, and are getting very minor complaints currently. However, we'd like to plan for increased transaction load and/or low-RPO replication.

We've recently went from 6 to 12 drives and the difference in stun times is negligible. ESXi only creates one additional helper snapshot during commit, so I think we're hitting some kind of a design wall here.

My question is: In production, has anyone been able to minimize stun times dramatically, to like <500msec per snapshot create/remove cycle, at least with a light load VM? How did you acheive that? Is it just raw IOPS? What sort of storage have you used?

I've read the topic and understand the tips, but I'd really like to talk to someone who has succeeded in this.
bold_defender
Lurker
Posts: 1
Liked: never
Joined: Sep 27, 2011 3:21 pm
Full Name: Mel Nugent

Re: Snapshot removal issues of a large VM

Post by bold_defender »

We have same situation here running 7.0.0.771
Its across 2 datacentres on one campus as follows.
2 hosts in each datacentre.
One datacentre has iSCSI hosts and other has SAS.
About 20 Windows VMs in each datacentre.
There is a SQL server running database mirroring and when I back it up the snapshot consolidation is causing a timeout and database failover.
Database timeout is 10sec but we're seeing 20sec plus drops.
SQL server has 3 disks
100GB OS and Programs
580GB DB
1,000GB DBArchive
Veeam is doing hot add backup

Is this a limitation of the infrastructure? The storage is IBM 3700 on both side and the 580Gb is on SSD.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Yes, you can reproduce this situation by creating snapshot manually (keeping it for the same time your backup job usually runs) and then initiating snapshot commit procedure. I'm not sure whether it will be possible or not, but you could try to adjust SQL mirroring timeout values to make it less sensitive, should help.
royp
Enthusiast
Posts: 38
Liked: 2 times
Joined: Mar 27, 2013 12:01 pm
Full Name: Roy P
Location: Wiltshire, UK
Contact:

[MERGED] Lost Ping Question Veeam 7.0.0.871

Post by royp »

I have read that while you will lose a few ping packets while the veeam BnR snapshots are removed after the backup job.
We lose approx. 3 packets which is enough to effect our Centos 5.7 and Centos 6 machines that run or testing machinery, this locks them up and causes quite an issue.

Is there a way to prevent these drops in pings/network connectivity as these Centos machines are testing 24/7 and writing to a central repository.
Post Reply

Who is online

Users browsing this forum: Google Feedfetcher and 78 guests