SQL won't freeze after v6 upgrade

[stevenrodenburg1]

We see it too. Since Fix 3 the database-server (SQL2008R2) the Veeam Database ran on could finally be backed-up with VSS. But that SQL server only had the Veeam database on it at that time.
(I migrated the Veeam database away from the main SQL server to a newly build, destined to take over the role of main DB server a couple of weeks later).

So the situation now was:
- Veeam on it's own DB server for a while -> VSS code now froze the system since Fix3
- The rest of the databases still on the old server -> VSS code now froze the system because Veeam DB was not on it anymore.

After two weeks or so, that SQL Server took over the role of main database server and had the other databases migrated to it. It's now home to (the) 7 databases. Since then, VSS has always failed. Not a single job could do a VSS freeze since then.

Although Veeam Jobs display the same error-messages as pre-Fix3, we suspect by now that V6 is quite pickey when it comes to allowing the database server to settle down so that the VSS code can freeze it. It cannot be a coincidence.

A V5 server has fewer problems but is not entirely error-free either (it never was, freeze-problems did occur a couple of times a week).

Conclusion: it was better with V5 but i feel that the "VSS freeze SQL" technology has it's limits on busy SQL servers. This is an industry-wide problem. Try to freeze a fat Oracle server for example. It too will need some time to get it's act straight. Heavy I/O environments are not easy for any backup-server. I mean, on a SAN SnapShot based solution like EMC or NetApp or HP, the database can take minutes to quiece. Just SAN sometimes needs to wait for a couple of minutes before the Winows server side software-component says "yes mr. SAN, now you can take your SnapShot". It's logical.
As far as i can see, the Veeam code is not as patient. I don't know what the time-out value is nor if it can be increased. Can it, guys?

As a workaround, We introduced the following workaround:
- Forget enabling VSS in Veeam as on these kind of servers, it has issues.
- Make hourly dumps of all the dababases, directly to a separate VMDK which is dedicated to holding SQL Backups (from maintainance jobs).
- Those maintainance jobs take place, and are finished before the hourly Veeam replication job starts
- In the Veeam Job, only enable VMtools quiescing. It's enough to flush the pending writebuffers on all disks exluding the actual database VMDK's (those are too busy)
- Don't forget to build "cleanup maintainance jobs" as well to prevent the "Backup files VMDK" from getting too full.

This way, just before the backup / replication starts, the backupfiles are at rest and ensured to be completely on disk (consistent).

Yes the databases are now backed-up in "mid air" but that happens on array-based replication just the same.

Maybe Microsoft and/or Veeam needs to invest more time in VSS on busy database servers (Oracle V11, SQL, Exchange, ADAM/ESE). One can say all they want but V5 has *fewer* issues than V6. That is what daily reality tells me. Support saying that customers have resource-problems all of sudden is something i'm not buying.

My opinion: Since Fix 3, V6 has become stable enough for production. VSS needs work.

We do not want to allocate a Windows Server license to a dedicated SQL server just to house the Veeam DB (basically having the old situation back again). It's silly to have to isolate the Veeam DB on it's own server, just to be able to back it up with VSS (or the other databases for that matter).

[StefanSpecht]

[merged]

Hi,

recently I moved the local SQL Express Veeam database to our central, virtual SQL database server.
When trying to backing up this SQL server, I gets an error now: "Unable to release guest VSSControl: Failed to freeze guest, wait timeout
It seems like Veeam can't get the required information about VSS freezing as the I/O of its own database is frozen.....
Is there any workaround to get this server backed up correctly?

Stefan

[StefanSpecht]

Thanks for merging my post. Installing patch3 seems to fixed this issue for me.

[JustinCredible]

I installed Patch 3 and I still have this problem. I am not backing up by selecting the vCenter server, I added each ESXi host individually. Is there some way to move the VEEAM SQL server back to the VEEAM server instead of hosting it on our SQL database? I guess that would fix this problem.

[Gostev]

Hi Justin - yes, of course you can easily do move any SQL database between SQL servers if you'd like. Then, you just point Veeam to the new database. There are existing discussions regarding this, so please search the forum and continue posting in the existing topic should you need more information. Thanks!

Here, I actually ran into this topic while looking for something else > SQL DB to another server?

[JustinCredible]

What SQL Server edition should I install (what does VEEAM install when you choose to host the DB locally?) to move the DB to?

[Gostev]

You can install any edition of SQL 2005 or 2008. Veeam installs SQL 2005 SP4 Express. Kindly please continue posting regarding the database move in the corresponding topic, not to derail this thread.

[JustinCredible]

OK I did manage to move the DB to local, and changed the registry keys accordingly, after stopping all services of course. Services started back up fine, went to do the backup again and it still says the same errors.

[Gostev]

May be it's time to call support then

[cyberfed]

Ok Veeam, what's the fix already?

I just did a fresh install of V6 with patch 3 installed.
Our SQL server which has multiple db's including Veeam is suffering from this issue.

We just dropped over $30,000 in Veeam software and a fix must be provided.

How can we fix it?

[Gostev]

Hi, there are currently no known issues around this functionality with patch 3 installed, so please open a support case for investigation. If there are any real issues with our code, we still have a chance to include the fix in version 6.1. Thanks!

[cffit]

I'm having this issue with the SQL server that houses our vCenter database. I've tried multiple times of the day when the load on the server is little of nothing. Was Patch 3 supposed to fix this issue for when a SQL server that is being backed up houses the vCenter database as well? Or just the VEEAM database?

[foggy]

Patch #3 addresses the case just with the Veeam DB. Moving vCenter DB onto another server did help in this case, so there might be another issue that should be investigated separately.

[cffit]

Yes, I don't have that option to move the database to a physical server. I'm going to try and add the esxi host to VEEAM instead of selecting the DB server though the vCenter tree in VEEAM. However, when the time comes to use vmotion and migrate our vcenter server to another host, this will break the backups. Kind of frustrating. I'd be interested to know if setting the timeout on how long it waits for the snapshot to go through helped anyone in this scenario.

[Vitaliy S.]

I'm having this issue with the SQL server that houses our vCenter database. I've tried multiple times of the day when the load on the server is little of nothing. Was Patch 3 supposed to fix this issue for when a SQL server that is being backed up houses the vCenter database as well?

Backing up vCenter Server VM with application aware image processing enabled is only possible when you add vCenter Server through a standalone ESX(i) host, patch #3 was not supposed to fix that.

However, when the time comes to use vmotion and migrate our vcenter server to another host, this will break the backups.

Please look through an existing thread to workaround that: HOWTO: Backing up vCenter host issues

[cffit]

The problem isn't backing up the vCenter server exactly, it's backing up the server that houses the vCenter database. In our case, that is a SQL server which is different than the vCenter server. I am able to get it to work by adding the ESXi host and selelcting the database server from that rather than selecting it from vCenter.

I will check out the other thread you linked. Thanks

[kevinnaughton]

I am just setting up Veeam for the first time and was following the evaluator's guide. I backed up my vCenter server VM with local SQL Express install, ran into this issue and found this thread. I saw the recommendation to configure the backup via the host directly instead of vCenter, and that elliminated the problem with the backup. However, I can't restore it. It would not let me restore to original location saying it did not exist. So I selected restore to a new location, but can't get past the "resource pool" as it doesn't find one (since vCenter is not running).

Then I tried restore to new location or with different settings. That will appear to work, but if your VMDK files are on different datastores you need to run multiple restores while knowing where each file needs to go.

Given that, what is the proper method for backup and restore of your vCenter VM?

[Vitaliy S.]

Hi Kevin,

As you've correctly stated you should backup vCenter Server through a standalone host connection. As to the restore process, then I'm not sure I understand what the issue is since you do have an ability to restore different disk to the different datastores through the same entire VM restore wizard.

Thanks!

[cffit]

I do believe there are issues in this scenario too. I have vCenter on one server and the vCenter database on a SQL server. I can backup all my servers fine with VEEAM, but the SQL server that has the vCenter database I have to do individually. It backs up, but I'm not surprised that Kevin is having issues restoring it becasue when I try to do a SureBackup job with this SQL server, it fails and says it can't find the ID. I have a case open for that as of now with no resolution. I figure running a SureBackup job was similar to doing a restore of the VM, so I the issues may be related.

[cffit]

I've noticed that if I choose to backup a VM by choosing it through the host instead of vCenter, if I migrate the VM to another host and then back to the original host, the backup seems to break. I do this when I do VMWare upgrades and have to migrate the VM to another host temporarily. Is this a known issue or do I have a problem?

Also, once a job is setup, how can I go in and tell if a VM is being backed up through vCenter or if it is being backed up thorugh the ESXi host?

[Vitaliy S.]

I've noticed that if I choose to backup a VM by choosing it through the host instead of vCenter, if I migrate the VM to another host and then back to the original host, the backup seems to break.

That's expected, as every time the VM is registered on the host, its moref ID is also changed.

Also, once a job is setup, how can I go in and tell if a VM is being backed up through vCenter or if it is being backed up thorugh the ESXi host?

If you still have standalone hosts added to the backup console, you can use this script to identify how your VM is added to the job > Connection via vCenter vs via ESXi Host (root)

[cffit]

Does it work to backup vCenter and the DB server that hosts the vCenter DB through the vCenter option in VEEAM now? Or is it still necessary to back them up through the individual ESXi host then? If it's still necessary to back them up via host, that means backups will break anytime the VMs are moved and they need to be re-setup then? How do most people get around this?

[veremin]

Does it work to backup vCenter and the DB server that hosts the vCenter DB through the vCenter option in VEEAM now?

Starting from version 7, direct host connection isn't required any longer, as vCenter databases are automatically excluded from VSS processing. You can take a look at this topic for further clarification.

Thanks.

[cffit]

Thanks for the direction in your last post.

When I run the powershell script, it lists all the VMs in the job, but the column for Object.Host.Type is empty for all VMs.

[veremin]

Can you confirm that VMs are added to the job as individual objects not as the part of container? What product version you're using? Thanks.

[cffit]

I'm using v7 with the R2 patch. Each of my jobs has individual VMs added to it via vCenter except for a handful like the vcenter server and the db server that has the vcenter db.

[cffit]

As a test, I just now manually added a VM to a job by adding it through an ESXi host and NOT vCenter. When I run the powershell command it lists all the VMs but the column with $_.Object.Host.Type is blank for both vCenter added VMs and ESXi host added VMs.

Actually now after trying, if I just use the first two lines of the code it works. The last part of the script must either need some tweaking or I'm missing something.

Thanks!

[Vitaliy S.]

Ok, thanks for the feedback. Anyway, since there is no need to use direct connection to the ESXi hosts any more, I believe you can reconfigure your job with vCenter Server VM and remove individual hosts from the backup console.

[veremin]

Actually now after trying, if I just use the first two lines of the code it works. The last part of the script must either need some tweaking or I'm missing something.

Not sure what exactly can be changed in the script, as everything seems to be working fine in my test lab. For the purpose of sheer interest you can leave first two lines (where the variables get assigned) and input $Objects.Host in the console in order to see what exact properties this variable has - whether or not .Type is among those shown.

Thanks.