Discussions specific to the VMware vSphere hypervisor
Post Reply
milfner
Influencer
Posts: 20
Liked: 1 time
Joined: Feb 16, 2011 7:51 am
Contact:

VSS issues with vCenter and Veeam

Post by milfner » 1 person likes this post

Hi,

I am having trouble backing up my virtual environment running Veeam on a VM that contains vCenter Server. Let me give you a breif run down of my environment before describing the issue.

I have a single ESXi 5 Server (HP ProLian ML-350 G6) with over 800GB of internal disk storage (RAID-5/SAS), which i use for my primary and only single data store for VMware virtual machines. It is all local storage on this server, no shared storage SAN.

I have 5 VM's (SBS, SQL, TS, XP machine and a vCenter Server).

The vCenter server is installed with a local instance of SQL Server Express. Veeam 6 is also installed on this server with a seperate instance of SQL Server Express. I have found out from some research that adding vCenter server in Veeam to process the VM's is not a good idea because the database freezes up during the snapshot and instead it is recommended to add the ESXi Host directly to Veeam and then add the VM's through the ESXi host for processing, so the vCenter server is not used in Veeam for processing. I have done this and added all of my 5 VM's to a job.

I have created a backup depository, which is a 8TB (RAID-5) Q-NAP NAS connected over ethernet.

My backup proxy settings are all set to 'Automatic'. Just on this, should it always be set to automatic for processing mode? What processing mode is best suited to my situation? Virtual Appliance Mode or Network Mode?

After running the job several times, every job failed, however, 2 out of the 5 VM's passed. These were the TS and an XP Machine. The other three that failed were SBS, SQL and vCenter server, all of which use high I/O load for databases and I assume this is related to VSS failing.

The errors i get for these jobs vary but they all seem to be pointing to VSS. Here are some of the errors.

Code: Select all

4/01/2012 11:40:28 AM :: Failed to prepare guest for hot backup. Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec
4/01/2012 11:56:08 AM :: Failed to prepare guest for hot backup. Error: VSSControl: -2147024809 Backup job failed.
Discovery phase failed.
Cannot enumerate volumes that should be added to the snapshot set.
Cannot enumerate volumes that contain components specified by the processed writer. Writer name: [SqlServerWriter], class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. instance ID: [{58ecd4d2-b67b-4465-af8b-2dcd840acb52}].
Cannot obtain a volume mount point for the volume that contains the specified file. File path: [\\sql\Pr
4/01/2012 2:03:25 PM :: Task failed unexpectedly
The above errors all occured from using Backup Proxy "Automatic" settings, when i changed this to "Network" processing mode, it completed the first VM successfully (which had always been completing successfully previously), but then it got stuck on removing snapshot for 24hrs before moving on to the next VM (SBS, which was known to fail). I had to end the job prematurely.

Please advise what I am doing wrong with my particualr setup.

Reagards,

Matt.

dellock6
Veeam Software
Posts: 5929
Liked: 1746 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: VSS issues with vCenter and Veeam

Post by dellock6 »

You cannot use VSS on vCenter invoking VSS freeze from vCenter itself, read this kb article:

http://www.veeam.com/kb_articles.html/KB1051

Regards,
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

milfner
Influencer
Posts: 20
Liked: 1 time
Joined: Feb 16, 2011 7:51 am
Contact:

Re: VSS issues with vCenter and Veeam

Post by milfner »

dellock6 wrote:You cannot use VSS on vCenter invoking VSS freeze from vCenter itself, read this kb article:

http://www.veeam.com/kb_articles.html/KB1051

Regards,
Luca.
Solution: There is a way to work-around this by adding the ESX host that is managing this vCenter VM into the Veeam Backup console as a standalone host by its IP address. This allows you to communicate directly with the host, bypassing vCenter communication for operations in the job.

I have done this.

Gostev
SVP, Product Management
Posts: 26700
Liked: 4276 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS issues with vCenter and Veeam

Post by Gostev »

The above issue is different, and most likely has to deal with some 3rd party VSS components installed on the VM in question (for example, backup agents from another backup software). They extend VSS framework in the incorrect manner, and make generic VSS calls which are unaware about VSS extensions start failing.

What is the output of the following commands (on a VM that you are trying to backup):

Code: Select all

vssadmin list providers
vssadmin list writers

milfner
Influencer
Posts: 20
Liked: 1 time
Joined: Feb 16, 2011 7:51 am
Contact:

Re: VSS issues with vCenter and Veeam

Post by milfner »

i ran those commands on our SBS, which is one of the VM's failing. Here is the output:

vssadmin list providers:
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

vssadmin list writers:
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {202b65fe-8541-48a8-b2b9-afd8d999c914}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {b9b38a77-d3ed-4ad8-9134-98d2b1fa5834}
State: [1] Stable
Last error: No error

Writer name: 'SharePoint Services Writer'
Writer Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Instance Id: {7569037c-c3f5-4246-8e84-dea58570fa46}
State: [1] Stable
Last error: No error

Writer name: 'FSRM Writer'
Writer Id: {12ce4370-5bb7-4c58-a76a-e5d5097e3674}
Writer Instance Id: {7c659cfd-1f1a-46a5-bf02-127e5d9f644a}
State: [1] Stable
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {7495ad21-6a6a-464e-8397-75491466d578}
State: [1] Stable
Last error: No error

Writer name: 'NPS VSS Writer'
Writer Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
Writer Instance Id: {94f1fdfc-8864-4ced-97bf-9c7be11a653b}
State: [1] Stable
Last error: No error

Writer name: 'SPSearch4 VSS Writer'
Writer Id: {35500004-0201-0000-0000-000000000000}
Writer Instance Id: {84a9811c-5ce3-4ff3-bc2f-6801982ded3a}
State: [1] Stable
Last error: No error

Writer name: 'TS Gateway Writer'
Writer Id: {368753ec-572e-4fc7-b4b9-ccd9bdc624cb}
Writer Instance Id: {46442205-fc3b-4431-9206-6ca358226982}
State: [1] Stable
Last error: No error

Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {68b5d785-7557-452f-97aa-d92c71932efc}
State: [1] Stable
Last error: No error

Writer name: 'TermServLicensing'
Writer Id: {5382579c-98df-47a7-ac6c-98a6d7106e09}
Writer Instance Id: {38c012be-123c-41b9-862b-552486ebb48d}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {12c9f301-877f-44d8-83f1-2375ea6001ed}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {267af405-f6d8-4b8c-b79b-1dbb52fd22c0}
State: [1] Stable
Last error: No error

Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {6a976af1-f09c-4daf-86f8-021a05dde349}
State: [1] Stable
Last error: No error

Writer name: 'FRS Writer'
Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
Writer Instance Id: {402147d5-3799-4543-a31b-3eb2d6e91e65}
State: [1] Stable
Last error: No error

Writer name: 'Certificate Authority'
Writer Id: {6f5b15b5-da24-4d88-b737-63063e3a1f86}
Writer Instance Id: {242220af-f880-4889-b445-fdf9a5da7ccb}
State: [1] Stable
Last error: No error

Writer name: 'Dhcp Jet Writer'
Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
Writer Instance Id: {e14e9c78-e2d4-4755-8b67-f6b923521f39}
State: [1] Stable
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {3466dfe1-cf43-46d8-84a9-ad9db428a145}
State: [1] Stable
Last error: No error

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {2ae78d10-ff49-442c-9627-b909f10efc6b}
State: [1] Stable
Last error: No error

Writer name: 'MSSearch Service Writer'
Writer Id: {cd3f2362-8bef-46c7-9181-d62844cdc0b2}
Writer Instance Id: {fe617a0f-0eaa-4c87-a219-e31c85b67225}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {958accc4-317e-4afd-95b2-a1ca6bd61c86}
State: [1] Stable
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {e74b74fd-8e5b-4a04-840b-698309ded1ea}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {bc170608-97ec-4b1e-b991-c5265d967361}
State: [1] Stable
Last error: No error

milfner
Influencer
Posts: 20
Liked: 1 time
Joined: Feb 16, 2011 7:51 am
Contact:

Re: VSS issues with vCenter and Veeam

Post by milfner »

just an additional note.

I tried running the job without VSS, while also stopping all vCenter Services on the vCenter server as well as removing the vCenter VM from the backup job, so the 4 other VM's would run directly through the ESXi Hypervisor for this job.

This time the job would terminate unexpectedly and cause veeam to crash (not respond) for a specific period of time. You can also see from the following diagram that the network is working while the Veeam job is working, but then something happens and it just stops suddenly causing Veeam to not responsd. This scenario has happened several times WITHOUT using VSS. When VSS is enabled i get the issue from the original post.

http://ft.trillian.im/typera/64sz1Ra21-77jjyfd48u.jpg

Gostev
SVP, Product Management
Posts: 26700
Liked: 4276 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS issues with vCenter and Veeam

Post by Gostev »

OK, please open a support case for further investigation. Thanks.

NEM4EVER
Novice
Posts: 7
Liked: never
Joined: Jan 12, 2012 4:50 pm
Full Name: Ulrik Holm Nielsen
Contact:

Re: VSS issues with vCenter and Veeam

Post by NEM4EVER »

Had the same problem and found a solution here:
http://social.technet.microsoft.com/For ... f70ad8177/

Maybe its helpful :-)

milfner
Influencer
Posts: 20
Liked: 1 time
Joined: Feb 16, 2011 7:51 am
Contact:

Re: VSS issues with vCenter and Veeam

Post by milfner »

Which post exactly got it working for you? was it this...

1. Open the command prompt as administrator.
2. Go to C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\BIN
3. Execute the following command:

PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

After completing the above command restart the server and try to perform the backup operation again.

milfner
Influencer
Posts: 20
Liked: 1 time
Joined: Feb 16, 2011 7:51 am
Contact:

Re: VSS issues with vCenter and Veeam

Post by milfner »

Thank you very much NEM4EVER, as you have helped resolve my troublesome VSS error for one of my VM's in my job (SBS). The problem was caused by Sharepoint Foundation SP1 being installed. I ran the following commands which resolved the SBS VM failing. Now i have 4/5 VM's successful. There is still an SQL Server VM that is failing with a VSS issue.

http://blogs.technet.com/b/sbs/archive/ ... 0-sp1.aspx

Also, All of the above is for a new install of Veeam V5 on a seperate physical machine, as from my original post of having Veeam V6 on one of the VM's in the job (vCenter server), i tried running the job after this fix, but i keep getting an unexpected termination on the job. So the VSS error is fixed, but Veeam 6 is crashing.

Iv decided to stick with Veeam V5, as 4/5 of the VM's are working on this. Once i have resolved the issue with the 5th VM, i will then move back to Veeam V6 on the vCenter VM to try and resolve the unexpected termination error i am getting.

We are making progress. thanks.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 22 guests