Snapshot removal issues of a large VM

[myrdin]

Actually it was VEEAM that reenables CBT on the Virtual Machine VMX everytime you run the job! Ideas??

Edit: there is a nice switch on the job configuration "enable CBT automatically" that i had disabled...now to find a way to redisable the CBT without restarting the vm every single time!

[myrdin]

Actually no it doesnt work. Even with CBT confirmed disabled on both side:

- if i take snapshot in Vmware it works without losing packets when it commit the snapshot
- with Veeam i have at least 10 ping lost

ideas?

[foggy]

V7 Patch 3 backing up a machine. As soon as Veeam sends command to remove the snapshot to Vcenter at the end of the backup run, the vm hangs all the way until the remove snapshot process is finished (about 10-20 seconds).

Short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.

Removing snapshot IN VMWARE takes less than a second. Removing snapshot in VEEAM, 20 seconds delay while the VM is unresponsive.

Have you kept the manually created snapshot as long as the backup of that VM takes prior to removing it, to let it grow in size accordingly?

[myrdin]

Hi Foggy and thanks

i am planning an upgrade to v4.1. i will let you know how it goes.

[myrdin]

Upgrade to ESXi 4.1 Update 3 fixed the issue, only 1 lost ping (which is normal). Anyone having the same issue, this happens of a bug with ESX 4.0 and datastore on NFS shares. The machine freezes while removing the snapshot.

I will do 100 push-ups with a bag full of backup tapes on my back as punishment for blaming Veeam!

[johntnavi]

Hi,

We are experiencing the same symptoms with exchange 2010, Veeam 7.x installed so I was wondering if anyone has implemented the suggestions from vmware below:

http://kb.vmware.com/selfservice/micros ... Id=5962168

John

[Vitaliy S.]

Hi John,

I don't think that disabling application-aware image processing for Exchange VM and shutting down its services a good move. In this case you won't be able to use log truncation and application VSS writers, since VMware Tools provide VSS snapshot on file system layer, not the application. Furthermore, Exchange will still not be available during its services restart.

Thanks!

[tom11011]

Hi Group,

If your vmware environment is greater than 3.5 Update2, does the veeam 7.0 replication setting "safe removal of snapshots larger than" option have any meaning or bearing?

Have a virtual machine that is momentarily freezing during the snapshot recombine process. ESXI 5.5 update 1 with veeam 7.0 latest patches.

Thanks

[foggy]

Tom, this option indeed should not be used with newer vSphere versions. That said, short period of inactivity due to the required stun to commit snapshot data is expected during snapshot removal operations, please see this thread for details.

[bentabbron]

Hi,
I've been informed of a problem by our DBA that when Veeam is doing its nighly backup I/O is frozen on our databases for over 30 seconds!
Has anyone come across similar times?
We are running version Veeam backup & rep v 7.0
This does seem like an excessive peroid of time for the DB to be frozen.

Thanks

[Vitaliy S.]

Hi Ben,

Does it happen during VM snapshot commit operations? If yes, then please review this thread for further details.

Thanks!

[bentabbron]

Hi Vitaliy,
How would I check that?

Thanks

[veremin]

Just correlate database freeze with the specific step shown in backup console or vSphere Client. Thanks.

[bentabbron]

Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.

[veremin]

What type of database you're talking about? Are you using Application Aware Image Processing or VMware quiescence option with pre-freeze/post-thaw scripts? Thanks.

[Vitaliy S.]

Try to offload the datastore where this SQL Server resides, might help. BTW, what kind of storage do you have at the back-end? If you could use our integration with HP and upcoming for NetApp, then this could potentially address this VMware snapshot commit behavior.

[bentabbron]

Application Image Aware processing for SQL server databases. Thanks

[bentabbron]

Hi,
We are using HP 3 Par storage which is doing storage based Snapshots via Veeam.

[Vitaliy S.]

So you're backing up SQL Server VM from these storage snapshots and still experience this issue?

Yes it takes the snapshot at 22.03 and the I/O is then frozen 40 seconds later the I/O is resumed.
Snapshot is then removed at 22.08.

Can you please check job logs and see what happens in these 5 minutes?

[veremin]

Is this issue reproducible without Veeam presence? That's being said, what happens if you take a snapshot manually, using vSphere Client? Thanks.

[bentabbron]

Hi Vitaliy,
Is it the Veeam job logs you would like me to check?

[Vitaliy S.]

Yep, job session logs.

[bentabbron]

Is it the log file from C:\ProgramData\Veeam\Backup that I need to look at?

[Vitaliy S.]

Not really, I'm referring to VM backup session log that can be found in the Veeam backup console.

[bentabbron]

Session tab, Statistics or Report?

[veremin]

I believe Vitaliy was talking about "Session" tab where you can click on a given VM (on the left) and see the steps conducted. Thanks.

[VasilyT]

Hello,

A very intersting thread. We're currently experiencing minor stuns during both snapshot creation and removal, on all VMs. No heavy loads, so no long stuns - total stun periods are about 2-4 seconds per VM during backup, and the stun is not continious.
On a side note, I've found that additional vmdk drives increase stun times considerably.

We're using local storage with 12x10K RPM drives in RAID10 with 1Gb cache controller. VMs are snapshotted one at a time to decrease IOPS load.
We're running backups every 2 hours, and are getting very minor complaints currently. However, we'd like to plan for increased transaction load and/or low-RPO replication.

We've recently went from 6 to 12 drives and the difference in stun times is negligible. ESXi only creates one additional helper snapshot during commit, so I think we're hitting some kind of a design wall here.

My question is: In production, has anyone been able to minimize stun times dramatically, to like <500msec per snapshot create/remove cycle, at least with a light load VM? How did you acheive that? Is it just raw IOPS? What sort of storage have you used?

I've read the topic and understand the tips, but I'd really like to talk to someone who has succeeded in this.

[bold_defender]

We have same situation here running 7.0.0.771
Its across 2 datacentres on one campus as follows.
2 hosts in each datacentre.
One datacentre has iSCSI hosts and other has SAS.
About 20 Windows VMs in each datacentre.
There is a SQL server running database mirroring and when I back it up the snapshot consolidation is causing a timeout and database failover.
Database timeout is 10sec but we're seeing 20sec plus drops.
SQL server has 3 disks
100GB OS and Programs
580GB DB
1,000GB DBArchive
Veeam is doing hot add backup

Is this a limitation of the infrastructure? The storage is IBM 3700 on both side and the 580Gb is on SSD.

[Vitaliy S.]

Yes, you can reproduce this situation by creating snapshot manually (keeping it for the same time your backup job usually runs) and then initiating snapshot commit procedure. I'm not sure whether it will be possible or not, but you could try to adjust SQL mirroring timeout values to make it less sensitive, should help.

[royp]

I have read that while you will lose a few ping packets while the veeam BnR snapshots are removed after the backup job.
We lose approx. 3 packets which is enough to effect our Centos 5.7 and Centos 6 machines that run or testing machinery, this locks them up and causes quite an issue.

Is there a way to prevent these drops in pings/network connectivity as these Centos machines are testing 24/7 and writing to a central repository.