VSS issues with vCenter and Veeam

[milfner]

Hi,

I am having trouble backing up my virtual environment running Veeam on a VM that contains vCenter Server. Let me give you a breif run down of my environment before describing the issue.

I have a single ESXi 5 Server (HP ProLian ML-350 G6) with over 800GB of internal disk storage (RAID-5/SAS), which i use for my primary and only single data store for VMware virtual machines. It is all local storage on this server, no shared storage SAN.

I have 5 VM's (SBS, SQL, TS, XP machine and a vCenter Server).

The vCenter server is installed with a local instance of SQL Server Express. Veeam 6 is also installed on this server with a seperate instance of SQL Server Express. I have found out from some research that adding vCenter server in Veeam to process the VM's is not a good idea because the database freezes up during the snapshot and instead it is recommended to add the ESXi Host directly to Veeam and then add the VM's through the ESXi host for processing, so the vCenter server is not used in Veeam for processing. I have done this and added all of my 5 VM's to a job.

I have created a backup depository, which is a 8TB (RAID-5) Q-NAP NAS connected over ethernet.

My backup proxy settings are all set to 'Automatic'. Just on this, should it always be set to automatic for processing mode? What processing mode is best suited to my situation? Virtual Appliance Mode or Network Mode?

After running the job several times, every job failed, however, 2 out of the 5 VM's passed. These were the TS and an XP Machine. The other three that failed were SBS, SQL and vCenter server, all of which use high I/O load for databases and I assume this is related to VSS failing.

The errors i get for these jobs vary but they all seem to be pointing to VSS. Here are some of the errors.

Code: Select all

4/01/2012 11:40:28 AM :: Failed to prepare guest for hot backup. Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec
4/01/2012 11:56:08 AM :: Failed to prepare guest for hot backup. Error: VSSControl: -2147024809 Backup job failed.
Discovery phase failed.
Cannot enumerate volumes that should be added to the snapshot set.
Cannot enumerate volumes that contain components specified by the processed writer. Writer name: [SqlServerWriter], class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. instance ID: [{58ecd4d2-b67b-4465-af8b-2dcd840acb52}].
Cannot obtain a volume mount point for the volume that contains the specified file. File path: [\\sql\Pr
4/01/2012 2:03:25 PM :: Task failed unexpectedly

The above errors all occured from using Backup Proxy "Automatic" settings, when i changed this to "Network" processing mode, it completed the first VM successfully (which had always been completing successfully previously), but then it got stuck on removing snapshot for 24hrs before moving on to the next VM (SBS, which was known to fail). I had to end the job prematurely.

Please advise what I am doing wrong with my particualr setup.

Reagards,

Matt.

[dellock6]

You cannot use VSS on vCenter invoking VSS freeze from vCenter itself, read this kb article:

http://www.veeam.com/kb_articles.html/KB1051

Regards,
Luca.

[milfner]

You cannot use VSS on vCenter invoking VSS freeze from vCenter itself, read this kb article:

http://www.veeam.com/kb_articles.html/KB1051

Regards,
Luca.

Solution: There is a way to work-around this by adding the ESX host that is managing this vCenter VM into the Veeam Backup console as a standalone host by its IP address. This allows you to communicate directly with the host, bypassing vCenter communication for operations in the job.

I have done this.

[Gostev]

The above issue is different, and most likely has to deal with some 3rd party VSS components installed on the VM in question (for example, backup agents from another backup software). They extend VSS framework in the incorrect manner, and make generic VSS calls which are unaware about VSS extensions start failing.

What is the output of the following commands (on a VM that you are trying to backup):

Code: Select all

vssadmin list providers
vssadmin list writers

[milfner]

i ran those commands on our SBS, which is one of the VM's failing. Here is the output:

vssadmin list providers:
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

vssadmin list writers:
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {202b65fe-8541-48a8-b2b9-afd8d999c914}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {b9b38a77-d3ed-4ad8-9134-98d2b1fa5834}
State: [1] Stable
Last error: No error

Writer name: 'SharePoint Services Writer'
Writer Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Instance Id: {7569037c-c3f5-4246-8e84-dea58570fa46}
State: [1] Stable
Last error: No error

Writer name: 'FSRM Writer'
Writer Id: {12ce4370-5bb7-4c58-a76a-e5d5097e3674}
Writer Instance Id: {7c659cfd-1f1a-46a5-bf02-127e5d9f644a}
State: [1] Stable
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {7495ad21-6a6a-464e-8397-75491466d578}
State: [1] Stable
Last error: No error

Writer name: 'NPS VSS Writer'
Writer Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
Writer Instance Id: {94f1fdfc-8864-4ced-97bf-9c7be11a653b}
State: [1] Stable
Last error: No error

Writer name: 'SPSearch4 VSS Writer'
Writer Id: {35500004-0201-0000-0000-000000000000}
Writer Instance Id: {84a9811c-5ce3-4ff3-bc2f-6801982ded3a}
State: [1] Stable
Last error: No error

Writer name: 'TS Gateway Writer'
Writer Id: {368753ec-572e-4fc7-b4b9-ccd9bdc624cb}
Writer Instance Id: {46442205-fc3b-4431-9206-6ca358226982}
State: [1] Stable
Last error: No error

Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {68b5d785-7557-452f-97aa-d92c71932efc}
State: [1] Stable
Last error: No error

Writer name: 'TermServLicensing'
Writer Id: {5382579c-98df-47a7-ac6c-98a6d7106e09}
Writer Instance Id: {38c012be-123c-41b9-862b-552486ebb48d}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {12c9f301-877f-44d8-83f1-2375ea6001ed}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {267af405-f6d8-4b8c-b79b-1dbb52fd22c0}
State: [1] Stable
Last error: No error

Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {6a976af1-f09c-4daf-86f8-021a05dde349}
State: [1] Stable
Last error: No error

Writer name: 'FRS Writer'
Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
Writer Instance Id: {402147d5-3799-4543-a31b-3eb2d6e91e65}
State: [1] Stable
Last error: No error

Writer name: 'Certificate Authority'
Writer Id: {6f5b15b5-da24-4d88-b737-63063e3a1f86}
Writer Instance Id: {242220af-f880-4889-b445-fdf9a5da7ccb}
State: [1] Stable
Last error: No error

Writer name: 'Dhcp Jet Writer'
Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
Writer Instance Id: {e14e9c78-e2d4-4755-8b67-f6b923521f39}
State: [1] Stable
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {3466dfe1-cf43-46d8-84a9-ad9db428a145}
State: [1] Stable
Last error: No error

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {2ae78d10-ff49-442c-9627-b909f10efc6b}
State: [1] Stable
Last error: No error

Writer name: 'MSSearch Service Writer'
Writer Id: {cd3f2362-8bef-46c7-9181-d62844cdc0b2}
Writer Instance Id: {fe617a0f-0eaa-4c87-a219-e31c85b67225}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {958accc4-317e-4afd-95b2-a1ca6bd61c86}
State: [1] Stable
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {e74b74fd-8e5b-4a04-840b-698309ded1ea}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {bc170608-97ec-4b1e-b991-c5265d967361}
State: [1] Stable
Last error: No error

[milfner]

just an additional note.

I tried running the job without VSS, while also stopping all vCenter Services on the vCenter server as well as removing the vCenter VM from the backup job, so the 4 other VM's would run directly through the ESXi Hypervisor for this job.

This time the job would terminate unexpectedly and cause veeam to crash (not respond) for a specific period of time. You can also see from the following diagram that the network is working while the Veeam job is working, but then something happens and it just stops suddenly causing Veeam to not responsd. This scenario has happened several times WITHOUT using VSS. When VSS is enabled i get the issue from the original post.

http://ft.trillian.im/typera/64sz1Ra21-77jjyfd48u.jpg

[Gostev]

OK, please open a support case for further investigation. Thanks.

[NEM4EVER]

Had the same problem and found a solution here:
http://social.technet.microsoft.com/For ... f70ad8177/

Maybe its helpful

[milfner]

Which post exactly got it working for you? was it this...

1. Open the command prompt as administrator.
2. Go to C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\BIN
3. Execute the following command:

PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

After completing the above command restart the server and try to perform the backup operation again.

[milfner]

Thank you very much NEM4EVER, as you have helped resolve my troublesome VSS error for one of my VM's in my job (SBS). The problem was caused by Sharepoint Foundation SP1 being installed. I ran the following commands which resolved the SBS VM failing. Now i have 4/5 VM's successful. There is still an SQL Server VM that is failing with a VSS issue.

http://blogs.technet.com/b/sbs/archive/ ... 0-sp1.aspx

Also, All of the above is for a new install of Veeam V5 on a seperate physical machine, as from my original post of having Veeam V6 on one of the VM's in the job (vCenter server), i tried running the job after this fix, but i keep getting an unexpected termination on the job. So the VSS error is fixed, but Veeam 6 is crashing.

Iv decided to stick with Veeam V5, as 4/5 of the VM's are working on this. Once i have resolved the issue with the 5th VM, i will then move back to Veeam V6 on the vCenter VM to try and resolve the unexpected termination error i am getting.

We are making progress. thanks.