Snapshot removal issues of a large VM

VMware specific discussions

Snapshot removal hangs with 95%

Veeam Logoby mschulte » Thu Sep 01, 2011 11:04 pm

[merged]

Hello,

hope someone can help us.
Sometimes we have trouble with backups in Veeam 5.1.
Backupjob run on VM and hangs with remaining time 0:00:00
The job could not be stopped.

In Vsphere Client we can see a process on the vm : Remove Snapshop 95%
The Snapshot will not be removed for 24h - also the snapshot isn't great. (17MB)

The VM is frozen, no network traffic, no Console

In Vsphere Client we could not stop or restart the VM (because another Task is running - Task: remove snapshot)

Sometimes we can kill the VM in SSH Console, sometimes not.
After killing the vm in sshConsole sometimes you can start the vm , sometimes not. (because of locked files etc.)

What can be the problem ?

Thanks a lot
mschulte
Lurker
 
Posts: 2
Liked: never
Joined: Mon Jun 21, 2010 8:18 pm

Veeam Backup on SQL Server Causing Application Disconnects

Veeam Logoby offwire » Tue Oct 11, 2011 12:37 pm

[merged]

I would like to be able to use Veeam to backup our SQL Server (2008R2) every 4 hours, and then fill in the time in between with SQL backups. However, during working hours, when the Veeam backup is removing the snapshot, random users are disconnected from their applications that use databases on the SQL Server, and I am not sure what, if anything, I can do about it. One is our CRM application, and the other is our order entry/ecommerce application. They typically receive an error message stating that the network connection has timed out or that the connection was lost. They can close down and reopen and everything is fine again.

Has anyone experienced anything similar or have any ideas how I might work towards resolving this? I don't think this is a Veeam issue as much as an application issue, but it's a thorn in my side and I am out of ideas.

Server: Server 2008R2, x64
SQL: Standard Edition, 2008R2
VMWARE: ESXi 4.1 U1
offwire
Lurker
 
Posts: 2
Liked: never
Joined: Thu Jul 28, 2011 1:42 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby Gostev » Tue Oct 11, 2011 3:01 pm

Short summary of things which may help (for more information, please read this topic):

1. Make sure VM does not have any other snapshots (including hidden).
2. Increase CPU reservations in the VM settings.
3. Move snapshot location to a different datastore (via workingDir parameter), preferably backed by faster storage (for example, SSD disk).
Gostev
Veeam Software
 
Posts: 21510
Liked: 2381 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Snapshot removal issues of a large VM

Veeam Logoby KiwiJJ » Tue Oct 11, 2011 8:53 pm

Hi,
We were having problems where our SQL server (and others) were freeezing during snapshot removal. What I have found that works is:

1. In the Veeam backup job go to the vSphere tab under Avanced Settings and untick "Use changed block tracking data"

2. Power off the Virtual Machine and edit settings, go to the Options tab and select Configuration parameters.
Change "ctkEnabled" to false
Also change any “scsi#:#.ctkEnabled” to false
Power on the Virtual Machine

cheers,

JJ
KiwiJJ
Enthusiast
 
Posts: 84
Liked: 1 time
Joined: Tue Feb 16, 2010 8:05 pm
Location: New Zealand
Full Name: John Jones

Re: Snapshot removal issues of a large VM

Veeam Logoby Gostev » Tue Oct 11, 2011 9:38 pm

John, I am guessing that you probably have NFS storage? Indeed, VMware had issues around CBT and NFS storage, but there is hotfix available now which resolves it. For other types of storage, CBT should not present any issues. Plus, disabling it kills incremental backup time...
Gostev
Veeam Software
 
Posts: 21510
Liked: 2381 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Snapshot removal issues of a large VM

Veeam Logoby KiwiJJ » Tue Oct 11, 2011 10:38 pm

Hi Anton,
No, we have iSCSI storage (Dell MD3000i). We do not use incremental backups so that is of no concern to us. But this change definately made a difference. For our Exchange and SQL servers when CBT was on users would lose access while the last part of the snapshot was being written back and our application servers would lose connection to the SQL server. When I made the change the users never lost access and neither did the application servers.

cheers,

JJ
KiwiJJ
Enthusiast
 
Posts: 84
Liked: 1 time
Joined: Tue Feb 16, 2010 8:05 pm
Location: New Zealand
Full Name: John Jones

Re: Snapshot removal issues of a large VM

Veeam Logoby Bunce » Wed Oct 12, 2011 5:52 am

Thats very interesting. Are you positive it was changed block tracking added that extra pause JJ? We might give this a test..
Bunce
Expert
 
Posts: 254
Liked: 8 times
Joined: Fri Sep 18, 2009 9:56 am
Location: Adelaide, Australia
Full Name: Andrew

Re: Snapshot removal issues of a large VM

Veeam Logoby offwire » Wed Oct 12, 2011 3:31 pm

KiwiJJ wrote:Hi,
We were having problems where our SQL server (and others) were freeezing during snapshot removal. What I have found that works is:

1. In the Veeam backup job go to the vSphere tab under Avanced Settings and untick "Use changed block tracking data"

2. Power off the Virtual Machine and edit settings, go to the Options tab and select Configuration parameters.
Change "ctkEnabled" to false
Also change any “scsi#:#.ctkEnabled” to false
Power on the Virtual Machine

cheers,

JJ


This is definitely something I can try if it would be useful to have someone else try it. I am also using iSCSI storage on IBM DS3300's
offwire
Lurker
 
Posts: 2
Liked: never
Joined: Thu Jul 28, 2011 1:42 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby KiwiJJ » Wed Oct 12, 2011 7:38 pm

Hi Bunce,
It is to do with the way snapshots get written back when using CBT. (you can google this to see how this is done) This is a known issue and apparently VMware are working on it.
It was definately CBT that was causing the issue. As soon as I made the changes mentioned the problem went away.

cheers,

JJ
KiwiJJ
Enthusiast
 
Posts: 84
Liked: 1 time
Joined: Tue Feb 16, 2010 8:05 pm
Location: New Zealand
Full Name: John Jones

Snapshot removal issue of random vms

Veeam Logoby Nobody » Thu Oct 13, 2011 1:35 pm

[merged]

Hi everybody

I'm looking out for someone who might see the same issue as we have.
We're backing up around 250 VMs every night with Veeam B&R and Veeam VSS quiesces.

All the systems are monitored with nagios. Nearly Nagios is reporting one or two vms for beeing offline during the backup window.
After reading through a whole bunch of logfiles and support calls on vmware and veeam we know that the vms are freezing for ~60 seconds during the remove snapshot process.

Our vm's are stored on 4x IBM V7000 who are accessed over FC - so the known issue with CBT and NFS does not match in this case.

In our tracing, we found no performace issue on the VMware side - and the snapshots are running only a few minutes without growing too much.

Did anyone out there experience the same issue?
With SAP Servers freezing for around a minute - we've started getting unwanted management attention. :(

Best regards
Nobody
Nobody
Influencer
 
Posts: 10
Liked: never
Joined: Mon Apr 18, 2011 11:32 am
Full Name: Andreas Girrbach

Re: Snapshot removal issues of a large VM

Veeam Logoby daniflexx » Mon Oct 17, 2011 6:42 am

Did you check the storage for bottlenecks? I suggest you to specially check latencies and compare the iops that are hitting the SAN against your SAN limits.

I've been experiencing this network disconnection and log snapshoot removal when I faced iop contention in my old SANs.

I use Veeam Monitor to monitor the SAN performance during backup hours.

dani
daniflexx
Influencer
 
Posts: 19
Liked: 1 time
Joined: Sat Feb 19, 2011 3:19 pm
Full Name: Dani Mora

Veeam and file sharing/locking issues

Veeam Logoby Fox54 » Wed Nov 09, 2011 4:08 pm

[merged]

Hi,

I have an application that uses files in a directory as a shared database.
Since I installed Veeam 5.02.230 (32 bits) following a problem with my original Veeam installation, every time a Backup or replication of the VM where the files for the the application are located, the application hangs at the end of the backup/replication. The backup/replication uses the VSS options as it was before.

It was working perfectly before (I had version 5 but can't remember the build number)

Any idea ?
Fox54
Enthusiast
 
Posts: 29
Liked: never
Joined: Tue May 12, 2009 1:47 am

Re: Snapshot removal issues of a large VM

Veeam Logoby Gostev » Wed Nov 09, 2011 4:16 pm

Hi - for version 5, there were only maintenance (bugfix) releases - they do not change the way product operates. If you did not have this issue before, you should look for other possible recent configuration changes in your hardware or VMware environment (for example, additional snapshots presence - as described on the first page of this topic). Thanks.
Gostev
Veeam Software
 
Posts: 21510
Liked: 2381 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Snapshot removal issues of a large VM

Veeam Logoby Fox54 » Wed Nov 09, 2011 7:19 pm

I have a VM that serves as a file server for an application. There is a single directory with the files being the database and the executable application itself.
Whenever I replicate or backup, at the end of the backup/replication, the application hangs. It started with the installation of version 5.02.230 (32 bits). It was working perfectly in older version like 4.x. Even the first version of 5 were working fine
Fox54
Enthusiast
 
Posts: 29
Liked: never
Joined: Tue May 12, 2009 1:47 am

Best practice to backup a SQL server

Veeam Logoby jjlp » Mon Nov 21, 2011 10:43 am

[merged]

Hi all

Im currently testing veeam 5 on vsphere 5 and am having som trouble backing up my SQL server.
The backup goes ok but when veeam commits the snapshot, the server looses network connectivity.
The snapshot is approx 10 - 15 GB

I have been told that i can set the snapshot for safe removal under advanced settings, but does anyone know at what setting/size is best, and also what happens with the backup if I set this size?

Im really hoping that some of you can help

Thanks
Jesper
jjlp
Lurker
 
Posts: 2
Liked: never
Joined: Tue Oct 11, 2011 7:05 am

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 38 guests