Host-based backup of VMware vSphere VMs.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Veeam v7- backing up vCenter's remote SQL VSS writer error

Post by MPD »

I have heard that new in Veeam version 7 is the ability to exclude the vCenter database during backup of the vCenter SQL instance. In Veeam versions past, we needed to work around the issue of VSSControl failing to freeze the database actively locked by vCenter by targeting the ESXi server instead of vCenter. This broke vMotion of the job, obviously.

My question is, if you are NOT using the SQL Express instance for vCenter, and instead using a remote MS 2008 R2/2012 SQL server, can you backup the SQL server VM using Veeam in version 7? I have given the Veeam service account full permissions to the SQL server VM (version 7 checks the registry?) but the job fails. I'd like to avoid using SQL Express with vCenter, and be able to consistently backup only a single, consolidated Veeam, vCenter SQL server.

I am using EqualLogic storage, and thought I could have Veeam backup snapshots of the storage volumes, but the EqualLogic doesn't use persistent initiator naming so when the snapshots are taken, the iSCSI initiator will not reconnect to the new names. Major bummer.

Thank you for your help, in advance.
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by Gostev »

Yes, you can.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD »

Is there a supported method to back up the remote SQL server using Application Aware Image Processing when the SQL server holds both the Veeam and vCenter databases?
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by Gostev »

This should work out of box, unless something broke in v7 around SQL interaction during application-aware processing of our own SQL VM. By now, I have not heard any issues reported.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD »

OK, I will check my setup, as I am seeing the same behavior as version 6.5. Thank you, sir.

EDIT: I just saw that vCenter 5.5 isn't on version 7's compatibility list. My mistake.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD »

I reverted to vCenter 5.1 (ESXi 5.1) and installed a fresh VM of Veeam Enterprise version 7 with today's Veeam patch 1. I have removed the jobs and vCenter server from Veeam, then added the Veeam service account to the SQL server's Local Administrator group to provide registry access as referenced in this post: Backup Veeam and vCenter on same server

I am receiving the VSS timeouts when backing up the vCenter/Veeam SQL server as in version 6.5. I am using a backup proxy also installed in a VM that utilizes the MS iSCSI Initiator combined with Equallogic HIT for MPIO. All other test backups are performing snappy save the vCenter/Veeam SQL server instance.

Are there any registry tweaks for Veeam version 7 for specifying the vCenter database location? I understand that I can add the standalone ESXi host but it will break vMotion of the SQL server, which I'd like to avoid. I have been reading that version 7 fixes the VSS issues, but I am not finding anything in the version 7 documentation as to how the fix was implemented

Thanks all. Version 7 is awesome- great work!
veremin
Product Manager
Posts: 20413
Liked: 2302 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by veremin » 1 person likes this post

vCenter remote databases should be, indeed, excluded from VSS processing durig vCenter VM backup, as long as sufficient rights to the remote server registry are granted to the account under which vCenter is added to the backup console.

If you’ve followed recommended step and still experience the said issue, kindly, contact our support team and let them investigate it directly, since the deep log investigation seems to be required.

Thanks.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD »

I will look into my configuration and contact support. Thank you.
veremin
Product Manager
Posts: 20413
Liked: 2302 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by veremin »

It will be also appreciated, if you provide the support ticket ID, so that, we can follow the case and update the topic with the found resolution. Thanks.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD »

I have opened case 00460133 titled "Version 7 vCenter Remote SQL Server Backup, VSS Timeout". Perhaps my log files will help
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD » 1 person likes this post

I have been working with Veeam to try an remedy this, but haven't made much progress:

1. Added Veeam Backup user to SQL server Local Administrators - Job fails
2. Added Veeam Backup user to vCenter server Local Administrators - Job fails
3. Manually added vCenter database to Veeam's database exclusions (required Veeam registry edit. VSS is adding the database itself) - Job fails
4. Added exceptions to Windows Firewall on Proxy, Veeam Server, vCenter, SQL - Job fails
5. Turned off Windows Firewall on Proxy, Veeam Server, vCenter, SQL - Job fails
6. Added vCenter server by IP address rather than DNS name - Job fails
7. Use Domain Administrator account - Job fails
8. Recreate Veeam database - Job fails
9. Add standalone ESXi host to job rather than vCenter server - Job works as required
10. Add standalone ESXi host to job and revert changes 1-7 - Job works as required

I am not giving up quite yet. I'd like vMotion to work for my SQL server AND use AAIP for SQL if possible
mconte
Service Provider
Posts: 11
Liked: 3 times
Joined: Oct 18, 2013 7:51 am
Full Name: Michele
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by mconte » 1 person likes this post

Same problem here. Support case 00465251.
vCenter 5.1 and its MSSQL server are on distinct VMs.
MSSQL server backup works with "VMware Tools quiescence". If I enable "application-aware backup", job fails with "VSSControl: Failed to freeze guest, wait timeout" error.
I tried to give domain administration rights to the vCenter account used by Veeam to manage the vCenter.
I tried also to give domain admin rights to the "application-aware" domain user.
I also tried to re-import vCenter (and re-create every job) using a domain account with domain admin privileges.

The only way to use application-aware backup is by adding standalone ESXi host, but that's not really a solution, since it doesn't work with vmotion.
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by Gostev » 1 person likes this post

Thanks for including support case ID.
mconte
Service Provider
Posts: 11
Liked: 3 times
Joined: Oct 18, 2013 7:51 am
Full Name: Michele
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by mconte » 2 people like this post

Problem solved!

As suggested by Veeam support (case #00465251), I enabled database exclusions with this registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\EnableDBExclusions = 1 (DWORD).

Then I added vCenter database ("VCDB") to database exclusions (Main menu -> Database exclusions -> Add).
2 exclusions are needed: one using VM IP, one using VM windows name ("mssql1"). With only IP exclusion, backup job still fails.
Now I have application-aware backup working with vCenter DB virtual machine.

One big question: is the VCDB database properly protected now? is it guaranteed to be *always* in consistent and recoverable state when you use database exclusion?

I'm asking because I don't see any 18264 event ("database backed up") for VCDB database, while I still have them for other DBs.
If I don't use application-aware backup, but only "VMware Tools quiescence", I see a 18264 event for each database. Is "VMware Tools quiescence" a better choice for backing up a VM which hosts a vCenter DB? What's the official Veeam best practice about it?

PS: maybe this topic would deserve a detailed KB entry.
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by Gostev » 2 people like this post

SQL is still fully aware that it is being backed up, which is the most important part. You are simply excluding specific databases from the actual freeze process. Now, all transactional databases, including SQL Server, are fully protected from data loss in case of power loss (which is analogous to crash-consistent backup), that's actually the very reason behind transaction logs, so your data will be safe. The actual recovery (first start up after restore) may take a bit longer due to SQL having to replay transaction log to commit unfinished transactions, but this is the only drawback I can think of.
mconte
Service Provider
Posts: 11
Liked: 3 times
Joined: Oct 18, 2013 7:51 am
Full Name: Michele
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by mconte »

Let me recapitulate.

With application-aware backup I can have VSS processing and log truncation for every DB, but NOT for vCenter DB.
With "VMware Tools quiescence" I can have VSS processing for every DB, vCenter DB included. The only missing feature is log truncation (missing fro every DB).
Since vCenter DB is the most important thing to protect in my scenario, isn't "VMware Tools quiescence" better suited for the job? Am I missing something?
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by Gostev »

I cannot comment on whether or not "VMware Tools quiescence" is better suited for the job, because without source code access I cannot know the internals of this technology, how exactly does it interact with Microsoft VSS, and whether it performs all the required steps and does it correctly.

Originally, we created our own logic for VSS integration, because "VMware Tools quiescence" was the #1 source of support requests, and we could not really troubleshoot or support our customers with those issues. It was in later versions when we also added application-aware processing functionality on top of our VSS integration.
MPD
Novice
Posts: 8
Liked: 3 times
Joined: Oct 02, 2013 7:10 pm
Location: Portland, OR
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by MPD » 2 people like this post

We found a temporary work around in my case

A remote SQLSERVER.internal.domain vCenter database entry is being added to Veeam in its Database Exclusions menu (registry edit required to make the menu appear under Veeam 7's options). The logs state that instead of looking for the vCenter MS SQL database at SQLSERVER.internal.domain, Veeam is looking for a truncated SQLSERVER hostname (not SQLSERVER.internal.domain). When we added the non-FQDN hostname to the database exclusions, everything is working correctly. Veeam thinks that this is incorrect behavior, the original exclusion should work fine, and there may be an easy permanent fix somewhere that doesn't require the manual exclusion

TLDR; Work-around- add the hostname only to Veeam's database exclusions, then re-run your job
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

[MERGED] Vcenter database backup Failed to freeze guest

Post by tfloor »

Hi,

Since veeam v7 there is an new feature called transparant vcenter database backup .
i thought.

So i have an Windows 2008r2 SQL Server that has the vcdb (vcenter database) on it.
I have Veeam V7R2, and when backup the sql server it failed very fast with the following:

Unable to release guest: Error: VSSControl: Failed to freeze guest, wait timeout
Error: VSSControl: Failed to freeze guest, wait timeout


Before v7 i backupped the sql server by adding the esxi host as a seperate host (outside vcenter), that worked. But i want to use the way without that workaround.

I have also set the EnableDBexclusions, and i see the "vcdb" database is on the list with the short servername before it. i added the same rule with the ip adress also, and retried the job.

Failed again. :(

Any help is appreciated.

Thanks

Tristan
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by foggy »

Tristan, if nothing in this thread helps, I recommend contacting support to review job debug logs. Thanks.
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by tfloor »

foggy wrote:Tristan, if nothing in this thread helps, I recommend contacting support to review job debug logs. Thanks.
I think i'm going todo that. Because indeed nothing in this thread helps me...
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by tfloor » 1 person likes this post

Solution Found!

The database exclusion name is case-sensitive.
Since my database for vcenter was fully capitalized, the automatic exclusion made by VSS was not correct, maybe a bug. (not fully capitalized).

Changed it to fully capitalized and works like a charm
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by foggy »

Tristan, glad you've nailed it, thanks for sharing!
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by tfloor » 1 person likes this post

foggy wrote:Tristan, glad you've nailed it, thanks for sharing!
thanks to tier3 veeam support
Case # 00484747
cffit
Veteran
Posts: 338
Liked: 35 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by cffit »

So there are a lot of fixes for this thrown around so far. It seems the registry edit to exclude the VEEAM databases is the most important. Are any of the other things listed throughout this thread required? Like making certain accounts admins on the SQL server? Or is it just the DB exclusions via the registry?

Thanks!
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by Vitaliy S. »

Hi Christensen, it is just db exclusion via registry, no other things are required. Thanks!
michaelryancook
Expert
Posts: 116
Liked: 14 times
Joined: Nov 26, 2013 6:13 pm
Full Name: Michael Cook
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by michaelryancook »

This post helped us as well. Just FYI if you have mulitple vCenters on one SQL server the Veeam job only excludes the vCenter database the SQL server is within. We had a second vCenter with it's DB on the same SQL server. The two vCenter's are in linked mode and VSS based backups failed until we manually added the second DB to the database exclusions using the process defined in this thread.
velk22
Novice
Posts: 7
Liked: never
Joined: Apr 26, 2016 1:49 pm
Full Name: Brian V
Contact:

[MERGED] Backup of vCenter Database on VBR9 U1

Post by velk22 »

I've searched the forum, and everything I see posted is for v8.x. I have a vCenter vm that whose database is on a shared SQL instance. 2012DB and servers across the board.
Based on what I have seen, I have to run a backup of the vCenter server from INSIDE a host directly because VSS will not freeze the database...and it has to be a special job probably without the VSS option on that job. Great. However, what about if the database is on another shared server? I've read about how VBR v8 will "know to skip" the VSS part of vCenter. Is that when it's a local SQL DB only? How exactly would I configure the Jobs to capture vCenter and its database on another server? I assume there would be 2 jobs...one for vCenter which can't use VSS, one for the shared DB server without VSS + enabling the registry option inside veeam to skip specific DBs. Am I on the right track? Seems to be quite alot of work, to backup vCenter. I would think Veeam had a much better solution worked out to backup one of the most important systems.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by foggy » 1 person likes this post

Brian, vCenter DB is automatically skipped from VSS processing, you do not need to back it up via the host directly anymore and it can be backed up by a job with AAIP enabled (starting v7, AFAIR). That includes cases with remote DB (provided rights to the remote server registry are sufficient, please see above for details).
velk22
Novice
Posts: 7
Liked: never
Joined: Apr 26, 2016 1:49 pm
Full Name: Brian V
Contact:

Re: Veeam v7- backing up vCenter's remote SQL VSS writer err

Post by velk22 »

Alexander- Thanks for the response- the question is (and maybe some of the secret sauce here) since its remote SQL, it just "knows" what database is used for vCenter? Fine to me if that's the answer. I'm guessing the table structure is known so Veeam could identify it pretty easily in the process pre-snapshot.
Post Reply

Who is online

Users browsing this forum: No registered users and 30 guests