Discussions specific to the VMware vSphere hypervisor
stevenrodenburg1
Expert
Posts: 125
Liked: 19 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: SQL won't freeze after v6 upgrade

Post by stevenrodenburg1 » Mar 05, 2012 6:23 am

We see it too. Since Fix 3 the database-server (SQL2008R2) the Veeam Database ran on could finally be backed-up with VSS. But that SQL server only had the Veeam database on it at that time.
(I migrated the Veeam database away from the main SQL server to a newly build, destined to take over the role of main DB server a couple of weeks later).

So the situation now was:
- Veeam on it's own DB server for a while -> VSS code now froze the system since Fix3
- The rest of the databases still on the old server -> VSS code now froze the system because Veeam DB was not on it anymore.

After two weeks or so, that SQL Server took over the role of main database server and had the other databases migrated to it. It's now home to (the) 7 databases. Since then, VSS has always failed. Not a single job could do a VSS freeze since then.

Although Veeam Jobs display the same error-messages as pre-Fix3, we suspect by now that V6 is quite pickey when it comes to allowing the database server to settle down so that the VSS code can freeze it. It cannot be a coincidence.

A V5 server has fewer problems but is not entirely error-free either (it never was, freeze-problems did occur a couple of times a week).

Conclusion: it was better with V5 but i feel that the "VSS freeze SQL" technology has it's limits on busy SQL servers. This is an industry-wide problem. Try to freeze a fat Oracle server for example. It too will need some time to get it's act straight. Heavy I/O environments are not easy for any backup-server. I mean, on a SAN SnapShot based solution like EMC or NetApp or HP, the database can take minutes to quiece. Just SAN sometimes needs to wait for a couple of minutes before the Winows server side software-component says "yes mr. SAN, now you can take your SnapShot". It's logical.
As far as i can see, the Veeam code is not as patient. I don't know what the time-out value is nor if it can be increased. Can it, guys?

As a workaround, We introduced the following workaround:
- Forget enabling VSS in Veeam as on these kind of servers, it has issues.
- Make hourly dumps of all the dababases, directly to a separate VMDK which is dedicated to holding SQL Backups (from maintainance jobs).
- Those maintainance jobs take place, and are finished before the hourly Veeam replication job starts
- In the Veeam Job, only enable VMtools quiescing. It's enough to flush the pending writebuffers on all disks exluding the actual database VMDK's (those are too busy)
- Don't forget to build "cleanup maintainance jobs" as well to prevent the "Backup files VMDK" from getting too full.

This way, just before the backup / replication starts, the backupfiles are at rest and ensured to be completely on disk (consistent).

Yes the databases are now backed-up in "mid air" but that happens on array-based replication just the same.

Maybe Microsoft and/or Veeam needs to invest more time in VSS on busy database servers (Oracle V11, SQL, Exchange, ADAM/ESE). One can say all they want but V5 has *fewer* issues than V6. That is what daily reality tells me. Support saying that customers have resource-problems all of sudden is something i'm not buying.

My opinion: Since Fix 3, V6 has become stable enough for production. VSS needs work.

We do not want to allocate a Windows Server license to a dedicated SQL server just to house the Veeam DB (basically having the old situation back again). It's silly to have to isolate the Veeam DB on it's own server, just to be able to back it up with VSS (or the other databases for that matter).

StefanSpecht
Influencer
Posts: 16
Liked: never
Joined: Aug 17, 2010 12:21 pm
Full Name: Stefan Specht
Contact:

Backup SQL server holding Veeam DB

Post by StefanSpecht » Apr 20, 2012 8:32 am

[merged]

Hi,

recently I moved the local SQL Express Veeam database to our central, virtual SQL database server.
When trying to backing up this SQL server, I gets an error now: "Unable to release guest VSSControl: Failed to freeze guest, wait timeout
It seems like Veeam can't get the required information about VSS freezing as the I/O of its own database is frozen.....
Is there any workaround to get this server backed up correctly?

Stefan

StefanSpecht
Influencer
Posts: 16
Liked: never
Joined: Aug 17, 2010 12:21 pm
Full Name: Stefan Specht
Contact:

Re: SQL won't freeze after v6 upgrade

Post by StefanSpecht » Apr 20, 2012 9:33 am

Thanks for merging my post. Installing patch3 seems to fixed this issue for me.

JustinCredible
Influencer
Posts: 10
Liked: 3 times
Joined: Mar 01, 2012 11:54 pm
Full Name: Justin D
Contact:

Re: SQL won't freeze after v6 upgrade

Post by JustinCredible » Apr 27, 2012 9:13 pm

I installed Patch 3 and I still have this problem. I am not backing up by selecting the vCenter server, I added each ESXi host individually. Is there some way to move the VEEAM SQL server back to the VEEAM server instead of hosting it on our SQL database? I guess that would fix this problem.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SQL won't freeze after v6 upgrade

Post by Gostev » Apr 27, 2012 9:20 pm

Hi Justin - yes, of course you can easily do move any SQL database between SQL servers if you'd like. Then, you just point Veeam to the new database. There are existing discussions regarding this, so please search the forum and continue posting in the existing topic should you need more information. Thanks!

Here, I actually ran into this topic while looking for something else > SQL DB to another server?

JustinCredible
Influencer
Posts: 10
Liked: 3 times
Joined: Mar 01, 2012 11:54 pm
Full Name: Justin D
Contact:

Re: SQL won't freeze after v6 upgrade

Post by JustinCredible » Apr 27, 2012 10:31 pm

What SQL Server edition should I install (what does VEEAM install when you choose to host the DB locally?) to move the DB to?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SQL won't freeze after v6 upgrade

Post by Gostev » Apr 27, 2012 10:48 pm

You can install any edition of SQL 2005 or 2008. Veeam installs SQL 2005 SP4 Express. Kindly please continue posting regarding the database move in the corresponding topic, not to derail this thread.

JustinCredible
Influencer
Posts: 10
Liked: 3 times
Joined: Mar 01, 2012 11:54 pm
Full Name: Justin D
Contact:

Re: SQL won't freeze after v6 upgrade

Post by JustinCredible » Apr 27, 2012 11:08 pm

OK I did manage to move the DB to local, and changed the registry keys accordingly, after stopping all services of course. Services started back up fine, went to do the backup again and it still says the same errors.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SQL won't freeze after v6 upgrade

Post by Gostev » Apr 28, 2012 12:03 am

May be it's time to call support then ;)

cyberfed
Influencer
Posts: 11
Liked: 2 times
Joined: Nov 03, 2011 11:59 am
Contact:

Re: SQL won't freeze after v6 upgrade

Post by cyberfed » May 14, 2012 7:34 pm

Ok Veeam, what's the fix already?

I just did a fresh install of V6 with patch 3 installed.
Our SQL server which has multiple db's including Veeam is suffering from this issue.

We just dropped over $30,000 in Veeam software and a fix must be provided.

How can we fix it?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SQL won't freeze after v6 upgrade

Post by Gostev » May 14, 2012 9:06 pm

Hi, there are currently no known issues around this functionality with patch 3 installed, so please open a support case for investigation. If there are any real issues with our code, we still have a chance to include the fix in version 6.1. Thanks!

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: SQL won't freeze after v6 upgrade

Post by cffit » May 15, 2012 2:43 pm

I'm having this issue with the SQL server that houses our vCenter database. I've tried multiple times of the day when the load on the server is little of nothing. Was Patch 3 supposed to fix this issue for when a SQL server that is being backed up houses the vCenter database as well? Or just the VEEAM database?

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL won't freeze after v6 upgrade

Post by foggy » May 15, 2012 3:52 pm

Patch #3 addresses the case just with the Veeam DB. Moving vCenter DB onto another server did help in this case, so there might be another issue that should be investigated separately.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: SQL won't freeze after v6 upgrade

Post by cffit » May 15, 2012 4:01 pm

Yes, I don't have that option to move the database to a physical server. I'm going to try and add the esxi host to VEEAM instead of selecting the DB server though the vCenter tree in VEEAM. However, when the time comes to use vmotion and migrate our vcenter server to another host, this will break the backups. Kind of frustrating. I'd be interested to know if setting the timeout on how long it waits for the snapshot to go through helped anyone in this scenario.

Vitaliy S.
Product Manager
Posts: 22289
Liked: 1413 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SQL won't freeze after v6 upgrade

Post by Vitaliy S. » May 16, 2012 8:08 am

cffit wrote:I'm having this issue with the SQL server that houses our vCenter database. I've tried multiple times of the day when the load on the server is little of nothing. Was Patch 3 supposed to fix this issue for when a SQL server that is being backed up houses the vCenter database as well?
Backing up vCenter Server VM with application aware image processing enabled is only possible when you add vCenter Server through a standalone ESX(i) host, patch #3 was not supposed to fix that.
cffit wrote:However, when the time comes to use vmotion and migrate our vcenter server to another host, this will break the backups.
Please look through an existing thread to workaround that: HOWTO: Backing up vCenter host issues

Post Reply

Who is online

Users browsing this forum: Majestic-12 [Bot] and 8 guests