-
- Enthusiast
- Posts: 57
- Liked: 3 times
- Joined: Jul 02, 2013 4:17 am
- Full Name: NIck
- Contact:
Re: VM hangs while committing the snapshot
Actually it was VEEAM that reenables CBT on the Virtual Machine VMX everytime you run the job! Ideas??
Edit: there is a nice switch on the job configuration "enable CBT automatically" that i had disabled...now to find a way to redisable the CBT without restarting the vm every single time!
Edit: there is a nice switch on the job configuration "enable CBT automatically" that i had disabled...now to find a way to redisable the CBT without restarting the vm every single time!
-
- Enthusiast
- Posts: 57
- Liked: 3 times
- Joined: Jul 02, 2013 4:17 am
- Full Name: NIck
- Contact:
Re: VM hangs while committing the snapshot
Actually no it doesnt work. Even with CBT confirmed disabled on both side:
- if i take snapshot in Vmware it works without losing packets when it commit the snapshot
- with Veeam i have at least 10 ping lost
ideas?
- if i take snapshot in Vmware it works without losing packets when it commit the snapshot
- with Veeam i have at least 10 ping lost
ideas?
-
- Veeam Software
- Posts: 21128
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: [MERGED] VM hangs while committing the snapshot
Short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.myrdin wrote:V7 Patch 3 backing up a machine. As soon as Veeam sends command to remove the snapshot to Vcenter at the end of the backup run, the vm hangs all the way until the remove snapshot process is finished (about 10-20 seconds).
Have you kept the manually created snapshot as long as the backup of that VM takes prior to removing it, to let it grow in size accordingly?myrdin wrote:Removing snapshot IN VMWARE takes less than a second. Removing snapshot in VEEAM, 20 seconds delay while the VM is unresponsive.
-
- Enthusiast
- Posts: 57
- Liked: 3 times
- Joined: Jul 02, 2013 4:17 am
- Full Name: NIck
- Contact:
Re: Snapshot removal issues of a large VM
Hi Foggy and thanks
i am planning an upgrade to v4.1. i will let you know how it goes.
i am planning an upgrade to v4.1. i will let you know how it goes.
-
- Enthusiast
- Posts: 57
- Liked: 3 times
- Joined: Jul 02, 2013 4:17 am
- Full Name: NIck
- Contact:
Re: Snapshot removal issues of a large VM
Upgrade to ESXi 4.1 Update 3 fixed the issue, only 1 lost ping (which is normal). Anyone having the same issue, this happens of a bug with ESX 4.0 and datastore on NFS shares. The machine freezes while removing the snapshot.
I will do 100 push-ups with a bag full of backup tapes on my back as punishment for blaming Veeam!
I will do 100 push-ups with a bag full of backup tapes on my back as punishment for blaming Veeam!
-
- Enthusiast
- Posts: 27
- Liked: 1 time
- Joined: Apr 27, 2012 6:55 pm
- Full Name: John T
- Contact:
Re: Snapshot removal issues of a large VM
Hi,
We are experiencing the same symptoms with exchange 2010, Veeam 7.x installed so I was wondering if anyone has implemented the suggestions from vmware below:
http://kb.vmware.com/selfservice/micros ... Id=5962168
John
We are experiencing the same symptoms with exchange 2010, Veeam 7.x installed so I was wondering if anyone has implemented the suggestions from vmware below:
http://kb.vmware.com/selfservice/micros ... Id=5962168
John
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
Hi John,
I don't think that disabling application-aware image processing for Exchange VM and shutting down its services a good move. In this case you won't be able to use log truncation and application VSS writers, since VMware Tools provide VSS snapshot on file system layer, not the application. Furthermore, Exchange will still not be available during its services restart.
Thanks!
I don't think that disabling application-aware image processing for Exchange VM and shutting down its services a good move. In this case you won't be able to use log truncation and application VSS writers, since VMware Tools provide VSS snapshot on file system layer, not the application. Furthermore, Exchange will still not be available during its services restart.
Thanks!
-
- Expert
- Posts: 192
- Liked: 9 times
- Joined: Dec 01, 2010 8:40 pm
- Full Name: Tom
- Contact:
[MERGED] Question about the safe snapshot removal feature
Hi Group,
If your vmware environment is greater than 3.5 Update2, does the veeam 7.0 replication setting "safe removal of snapshots larger than" option have any meaning or bearing?
Have a virtual machine that is momentarily freezing during the snapshot recombine process. ESXI 5.5 update 1 with veeam 7.0 latest patches.
Thanks
If your vmware environment is greater than 3.5 Update2, does the veeam 7.0 replication setting "safe removal of snapshots larger than" option have any meaning or bearing?
Have a virtual machine that is momentarily freezing during the snapshot recombine process. ESXI 5.5 update 1 with veeam 7.0 latest patches.
Thanks
-
- Veeam Software
- Posts: 21128
- Liked: 2137 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Snapshot removal issues of a large VM
Tom, this option indeed should not be used with newer vSphere versions. That said, short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
[MERGED] I/O is Frozen on the Database
Hi,
I've been informed of a problem by our DBA that when Veeam is doing its nighly backup I/O is frozen on our databases for over 30 seconds!
Has anyone come across similar times?
We are running version Veeam backup & rep v 7.0
This does seem like an excessive peroid of time for the DB to be frozen.
Thanks
I've been informed of a problem by our DBA that when Veeam is doing its nighly backup I/O is frozen on our databases for over 30 seconds!
Has anyone come across similar times?
We are running version Veeam backup & rep v 7.0
This does seem like an excessive peroid of time for the DB to be frozen.
Thanks
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
Hi Ben,
Does it happen during VM snapshot commit operations? If yes, then please review this thread for further details.
Thanks!
Does it happen during VM snapshot commit operations? If yes, then please review this thread for further details.
Thanks!
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Hi Vitaliy,
How would I check that?
Thanks
How would I check that?
Thanks
-
- Product Manager
- Posts: 20353
- Liked: 2285 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
Just correlate database freeze with the specific step shown in backup console or vSphere Client. Thanks.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.
Snapshot is then removed at 22.08.
-
- Product Manager
- Posts: 20353
- Liked: 2285 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
What type of database you're talking about? Are you using Application Aware Image Processing or VMware quiescence option with pre-freeze/post-thaw scripts? Thanks.
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
Try to offload the datastore where this SQL Server resides, might help. BTW, what kind of storage do you have at the back-end? If you could use our integration with HP and upcoming for NetApp, then this could potentially address this VMware snapshot commit behavior.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Application Image Aware processing for SQL server databases. Thanks
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Hi,
We are using HP 3 Par storage which is doing storage based Snapshots via Veeam.
We are using HP 3 Par storage which is doing storage based Snapshots via Veeam.
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
So you're backing up SQL Server VM from these storage snapshots and still experience this issue?
Can you please check job logs and see what happens in these 5 minutes?bentabbron wrote:Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.
-
- Product Manager
- Posts: 20353
- Liked: 2285 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
Is this issue reproducible without Veeam presence? That's being said, what happens if you take a snapshot manually, using vSphere Client? Thanks.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Hi Vitaliy,
Is it the Veeam job logs you would like me to check?
Is it the Veeam job logs you would like me to check?
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
Yep, job session logs.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Is it the log file from C:\ProgramData\Veeam\Backup that I need to look at?
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
Not really, I'm referring to VM backup session log that can be found in the Veeam backup console.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jul 02, 2014 12:36 pm
- Full Name: Ben Tabbron
- Contact:
Re: Snapshot removal issues of a large VM
Session tab, Statistics or Report?
-
- Product Manager
- Posts: 20353
- Liked: 2285 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Snapshot removal issues of a large VM
I believe Vitaliy was talking about "Session" tab where you can click on a given VM (on the left) and see the steps conducted. Thanks.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Jul 05, 2014 6:55 am
- Contact:
Re: Snapshot removal issues of a large VM
Hello,
A very intersting thread. We're currently experiencing minor stuns during both snapshot creation and removal, on all VMs. No heavy loads, so no long stuns - total stun periods are about 2-4 seconds per VM during backup, and the stun is not continious.
On a side note, I've found that additional vmdk drives increase stun times considerably.
We're using local storage with 12x10K RPM drives in RAID10 with 1Gb cache controller. VMs are snapshotted one at a time to decrease IOPS load.
We're running backups every 2 hours, and are getting very minor complaints currently. However, we'd like to plan for increased transaction load and/or low-RPO replication.
We've recently went from 6 to 12 drives and the difference in stun times is negligible. ESXi only creates one additional helper snapshot during commit, so I think we're hitting some kind of a design wall here.
My question is: In production, has anyone been able to minimize stun times dramatically, to like <500msec per snapshot create/remove cycle, at least with a light load VM? How did you acheive that? Is it just raw IOPS? What sort of storage have you used?
I've read the topic and understand the tips, but I'd really like to talk to someone who has succeeded in this.
A very intersting thread. We're currently experiencing minor stuns during both snapshot creation and removal, on all VMs. No heavy loads, so no long stuns - total stun periods are about 2-4 seconds per VM during backup, and the stun is not continious.
On a side note, I've found that additional vmdk drives increase stun times considerably.
We're using local storage with 12x10K RPM drives in RAID10 with 1Gb cache controller. VMs are snapshotted one at a time to decrease IOPS load.
We're running backups every 2 hours, and are getting very minor complaints currently. However, we'd like to plan for increased transaction load and/or low-RPO replication.
We've recently went from 6 to 12 drives and the difference in stun times is negligible. ESXi only creates one additional helper snapshot during commit, so I think we're hitting some kind of a design wall here.
My question is: In production, has anyone been able to minimize stun times dramatically, to like <500msec per snapshot create/remove cycle, at least with a light load VM? How did you acheive that? Is it just raw IOPS? What sort of storage have you used?
I've read the topic and understand the tips, but I'd really like to talk to someone who has succeeded in this.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Sep 27, 2011 3:21 pm
- Full Name: Mel Nugent
Re: Snapshot removal issues of a large VM
We have same situation here running 7.0.0.771
Its across 2 datacentres on one campus as follows.
2 hosts in each datacentre.
One datacentre has iSCSI hosts and other has SAS.
About 20 Windows VMs in each datacentre.
There is a SQL server running database mirroring and when I back it up the snapshot consolidation is causing a timeout and database failover.
Database timeout is 10sec but we're seeing 20sec plus drops.
SQL server has 3 disks
100GB OS and Programs
580GB DB
1,000GB DBArchive
Veeam is doing hot add backup
Is this a limitation of the infrastructure? The storage is IBM 3700 on both side and the 580Gb is on SSD.
Its across 2 datacentres on one campus as follows.
2 hosts in each datacentre.
One datacentre has iSCSI hosts and other has SAS.
About 20 Windows VMs in each datacentre.
There is a SQL server running database mirroring and when I back it up the snapshot consolidation is causing a timeout and database failover.
Database timeout is 10sec but we're seeing 20sec plus drops.
SQL server has 3 disks
100GB OS and Programs
580GB DB
1,000GB DBArchive
Veeam is doing hot add backup
Is this a limitation of the infrastructure? The storage is IBM 3700 on both side and the 580Gb is on SSD.
-
- VP, Product Management
- Posts: 27325
- Liked: 2778 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Snapshot removal issues of a large VM
Yes, you can reproduce this situation by creating snapshot manually (keeping it for the same time your backup job usually runs) and then initiating snapshot commit procedure. I'm not sure whether it will be possible or not, but you could try to adjust SQL mirroring timeout values to make it less sensitive, should help.
-
- Enthusiast
- Posts: 38
- Liked: 2 times
- Joined: Mar 27, 2013 12:01 pm
- Full Name: Roy P
- Location: Wiltshire, UK
- Contact:
[MERGED] Lost Ping Question Veeam 7.0.0.871
I have read that while you will lose a few ping packets while the veeam BnR snapshots are removed after the backup job.
We lose approx. 3 packets which is enough to effect our Centos 5.7 and Centos 6 machines that run or testing machinery, this locks them up and causes quite an issue.
Is there a way to prevent these drops in pings/network connectivity as these Centos machines are testing 24/7 and writing to a central repository.
We lose approx. 3 packets which is enough to effect our Centos 5.7 and Centos 6 machines that run or testing machinery, this locks them up and causes quite an issue.
Is there a way to prevent these drops in pings/network connectivity as these Centos machines are testing 24/7 and writing to a central repository.
Who is online
Users browsing this forum: No registered users and 33 guests