Veeam v7- backing up vCenter's remote SQL VSS writer error

[MPD]

I have heard that new in Veeam version 7 is the ability to exclude the vCenter database during backup of the vCenter SQL instance. In Veeam versions past, we needed to work around the issue of VSSControl failing to freeze the database actively locked by vCenter by targeting the ESXi server instead of vCenter. This broke vMotion of the job, obviously.

My question is, if you are NOT using the SQL Express instance for vCenter, and instead using a remote MS 2008 R2/2012 SQL server, can you backup the SQL server VM using Veeam in version 7? I have given the Veeam service account full permissions to the SQL server VM (version 7 checks the registry?) but the job fails. I'd like to avoid using SQL Express with vCenter, and be able to consistently backup only a single, consolidated Veeam, vCenter SQL server.

I am using EqualLogic storage, and thought I could have Veeam backup snapshots of the storage volumes, but the EqualLogic doesn't use persistent initiator naming so when the snapshots are taken, the iSCSI initiator will not reconnect to the new names. Major bummer.

Thank you for your help, in advance.

[Gostev]

Yes, you can.

[MPD]

Is there a supported method to back up the remote SQL server using Application Aware Image Processing when the SQL server holds both the Veeam and vCenter databases?

[Gostev]

This should work out of box, unless something broke in v7 around SQL interaction during application-aware processing of our own SQL VM. By now, I have not heard any issues reported.

[MPD]

OK, I will check my setup, as I am seeing the same behavior as version 6.5. Thank you, sir.

EDIT: I just saw that vCenter 5.5 isn't on version 7's compatibility list. My mistake.

[MPD]

I reverted to vCenter 5.1 (ESXi 5.1) and installed a fresh VM of Veeam Enterprise version 7 with today's Veeam patch 1. I have removed the jobs and vCenter server from Veeam, then added the Veeam service account to the SQL server's Local Administrator group to provide registry access as referenced in this post: Backup Veeam and vCenter on same server

I am receiving the VSS timeouts when backing up the vCenter/Veeam SQL server as in version 6.5. I am using a backup proxy also installed in a VM that utilizes the MS iSCSI Initiator combined with Equallogic HIT for MPIO. All other test backups are performing snappy save the vCenter/Veeam SQL server instance.

Are there any registry tweaks for Veeam version 7 for specifying the vCenter database location? I understand that I can add the standalone ESXi host but it will break vMotion of the SQL server, which I'd like to avoid. I have been reading that version 7 fixes the VSS issues, but I am not finding anything in the version 7 documentation as to how the fix was implemented

Thanks all. Version 7 is awesome- great work!

[veremin]

vCenter remote databases should be, indeed, excluded from VSS processing durig vCenter VM backup, as long as sufficient rights to the remote server registry are granted to the account under which vCenter is added to the backup console.

If you’ve followed recommended step and still experience the said issue, kindly, contact our support team and let them investigate it directly, since the deep log investigation seems to be required.

Thanks.

[MPD]

I will look into my configuration and contact support. Thank you.

[veremin]

It will be also appreciated, if you provide the support ticket ID, so that, we can follow the case and update the topic with the found resolution. Thanks.

[MPD]

I have opened case 00460133 titled "Version 7 vCenter Remote SQL Server Backup, VSS Timeout". Perhaps my log files will help

[MPD]

I have been working with Veeam to try an remedy this, but haven't made much progress:

1. Added Veeam Backup user to SQL server Local Administrators - Job fails
2. Added Veeam Backup user to vCenter server Local Administrators - Job fails
3. Manually added vCenter database to Veeam's database exclusions (required Veeam registry edit. VSS is adding the database itself) - Job fails
4. Added exceptions to Windows Firewall on Proxy, Veeam Server, vCenter, SQL - Job fails
5. Turned off Windows Firewall on Proxy, Veeam Server, vCenter, SQL - Job fails
6. Added vCenter server by IP address rather than DNS name - Job fails
7. Use Domain Administrator account - Job fails
8. Recreate Veeam database - Job fails
9. Add standalone ESXi host to job rather than vCenter server - Job works as required
10. Add standalone ESXi host to job and revert changes 1-7 - Job works as required

I am not giving up quite yet. I'd like vMotion to work for my SQL server AND use AAIP for SQL if possible

[mconte]

Same problem here. Support case 00465251.
vCenter 5.1 and its MSSQL server are on distinct VMs.
MSSQL server backup works with "VMware Tools quiescence". If I enable "application-aware backup", job fails with "VSSControl: Failed to freeze guest, wait timeout" error.
I tried to give domain administration rights to the vCenter account used by Veeam to manage the vCenter.
I tried also to give domain admin rights to the "application-aware" domain user.
I also tried to re-import vCenter (and re-create every job) using a domain account with domain admin privileges.

The only way to use application-aware backup is by adding standalone ESXi host, but that's not really a solution, since it doesn't work with vmotion.

[Gostev]

Thanks for including support case ID.

[mconte]

Problem solved!

As suggested by Veeam support (case #00465251), I enabled database exclusions with this registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\EnableDBExclusions = 1 (DWORD).

Then I added vCenter database ("VCDB") to database exclusions (Main menu -> Database exclusions -> Add).
2 exclusions are needed: one using VM IP, one using VM windows name ("mssql1"). With only IP exclusion, backup job still fails.
Now I have application-aware backup working with vCenter DB virtual machine.

One big question: is the VCDB database properly protected now? is it guaranteed to be *always* in consistent and recoverable state when you use database exclusion?

I'm asking because I don't see any 18264 event ("database backed up") for VCDB database, while I still have them for other DBs.
If I don't use application-aware backup, but only "VMware Tools quiescence", I see a 18264 event for each database. Is "VMware Tools quiescence" a better choice for backing up a VM which hosts a vCenter DB? What's the official Veeam best practice about it?

PS: maybe this topic would deserve a detailed KB entry.

[Gostev]

SQL is still fully aware that it is being backed up, which is the most important part. You are simply excluding specific databases from the actual freeze process. Now, all transactional databases, including SQL Server, are fully protected from data loss in case of power loss (which is analogous to crash-consistent backup), that's actually the very reason behind transaction logs, so your data will be safe. The actual recovery (first start up after restore) may take a bit longer due to SQL having to replay transaction log to commit unfinished transactions, but this is the only drawback I can think of.

[mconte]

Let me recapitulate.

With application-aware backup I can have VSS processing and log truncation for every DB, but NOT for vCenter DB.
With "VMware Tools quiescence" I can have VSS processing for every DB, vCenter DB included. The only missing feature is log truncation (missing fro every DB).
Since vCenter DB is the most important thing to protect in my scenario, isn't "VMware Tools quiescence" better suited for the job? Am I missing something?

[Gostev]

I cannot comment on whether or not "VMware Tools quiescence" is better suited for the job, because without source code access I cannot know the internals of this technology, how exactly does it interact with Microsoft VSS, and whether it performs all the required steps and does it correctly.

Originally, we created our own logic for VSS integration, because "VMware Tools quiescence" was the #1 source of support requests, and we could not really troubleshoot or support our customers with those issues. It was in later versions when we also added application-aware processing functionality on top of our VSS integration.

[MPD]

We found a temporary work around in my case

A remote SQLSERVER.internal.domain vCenter database entry is being added to Veeam in its Database Exclusions menu (registry edit required to make the menu appear under Veeam 7's options). The logs state that instead of looking for the vCenter MS SQL database at SQLSERVER.internal.domain, Veeam is looking for a truncated SQLSERVER hostname (not SQLSERVER.internal.domain). When we added the non-FQDN hostname to the database exclusions, everything is working correctly. Veeam thinks that this is incorrect behavior, the original exclusion should work fine, and there may be an easy permanent fix somewhere that doesn't require the manual exclusion

TLDR; Work-around- add the hostname only to Veeam's database exclusions, then re-run your job

[tfloor]

Hi,

Since veeam v7 there is an new feature called transparant vcenter database backup .
i thought.

So i have an Windows 2008r2 SQL Server that has the vcdb (vcenter database) on it.
I have Veeam V7R2, and when backup the sql server it failed very fast with the following:

Unable to release guest: Error: VSSControl: Failed to freeze guest, wait timeout
Error: VSSControl: Failed to freeze guest, wait timeout

Before v7 i backupped the sql server by adding the esxi host as a seperate host (outside vcenter), that worked. But i want to use the way without that workaround.

I have also set the EnableDBexclusions, and i see the "vcdb" database is on the list with the short servername before it. i added the same rule with the ip adress also, and retried the job.

Failed again.

Any help is appreciated.

Thanks

Tristan

[foggy]

Tristan, if nothing in this thread helps, I recommend contacting support to review job debug logs. Thanks.

[tfloor]

Tristan, if nothing in this thread helps, I recommend contacting support to review job debug logs. Thanks.

I think i'm going todo that. Because indeed nothing in this thread helps me...

[tfloor]

Solution Found!

The database exclusion name is case-sensitive.
Since my database for vcenter was fully capitalized, the automatic exclusion made by VSS was not correct, maybe a bug. (not fully capitalized).

Changed it to fully capitalized and works like a charm

[foggy]

Tristan, glad you've nailed it, thanks for sharing!

[tfloor]

Tristan, glad you've nailed it, thanks for sharing!

thanks to tier3 veeam support
Case # 00484747

[cffit]

So there are a lot of fixes for this thrown around so far. It seems the registry edit to exclude the VEEAM databases is the most important. Are any of the other things listed throughout this thread required? Like making certain accounts admins on the SQL server? Or is it just the DB exclusions via the registry?

Thanks!

[Vitaliy S.]

Hi Christensen, it is just db exclusion via registry, no other things are required. Thanks!

[michaelryancook]

This post helped us as well. Just FYI if you have mulitple vCenters on one SQL server the Veeam job only excludes the vCenter database the SQL server is within. We had a second vCenter with it's DB on the same SQL server. The two vCenter's are in linked mode and VSS based backups failed until we manually added the second DB to the database exclusions using the process defined in this thread.

[velk22]

I've searched the forum, and everything I see posted is for v8.x. I have a vCenter vm that whose database is on a shared SQL instance. 2012DB and servers across the board.
Based on what I have seen, I have to run a backup of the vCenter server from INSIDE a host directly because VSS will not freeze the database...and it has to be a special job probably without the VSS option on that job. Great. However, what about if the database is on another shared server? I've read about how VBR v8 will "know to skip" the VSS part of vCenter. Is that when it's a local SQL DB only? How exactly would I configure the Jobs to capture vCenter and its database on another server? I assume there would be 2 jobs...one for vCenter which can't use VSS, one for the shared DB server without VSS + enabling the registry option inside veeam to skip specific DBs. Am I on the right track? Seems to be quite alot of work, to backup vCenter. I would think Veeam had a much better solution worked out to backup one of the most important systems.

[foggy]

Brian, vCenter DB is automatically skipped from VSS processing, you do not need to back it up via the host directly anymore and it can be backed up by a job with AAIP enabled (starting v7, AFAIR). That includes cases with remote DB (provided rights to the remote server registry are sufficient, please see above for details).

[velk22]

Alexander- Thanks for the response- the question is (and maybe some of the secret sauce here) since its remote SQL, it just "knows" what database is used for vCenter? Fine to me if that's the answer. I'm guessing the table structure is known so Veeam could identify it pretty easily in the process pre-snapshot.