VSS Errors, again

[ehrnst]

Hello,

I am not sure this is a veeam related error, or something that can be fixed inside Windows. Therefor, i do not currently have a support ID. Anyway, here is my VSS issue.

Code: Select all

19.03.2014 14:23:28 :: Unable to release guest. Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x80042302].]

This is a very large MS SQL server, with around 2000 DB's in various sizes (VM has 4 CPU's and 32GB of memory). I have tried numerous things to fix this issue, "everything" from expanding the system partition, set shadow copy space to unlimited and to increase the number of SQL workers. None of these seems to help. In Windows logs, i see the following entry's, one for each database it manages to back up:

Event ID 18264, MSSQLSERVER:
Database backed up. Database: ZF10125, creation date(time): 2011/12/08(15:12:32), pages dumped: 322, first LSN: 45:7580:37, last LSN: 45:7597:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'{22A2A2B9-02C1-4C11-ABDB-BFE521E9B9EE}1828'}). This is an informational message only. No user action is required.

Before these two Errors is displayed:

Event ID 18264, VSS:
Volume Shadow Copy Service error: Unexpected error calling routine XML document is too long. hr = 0x80070018, The program issued a command but the command length is incorrect.
.

Operation:
Writer Modifying Modifying Backup Document

Context:
Execution Context: Requestor
Writer Instance ID: {80F9C8FA-5CE3-4E93-BFC1-8E2ED5DCFED5}

Event ID 18264, VSS:

Volume Shadow Copy Service error: Unexpected error calling routine IXMLDOMNode::replaceChild. hr = 0x80070057, The parameter is incorrect.
.

Operation:
Processing Writer Data
Executing Asynchronous Operation

Context:
Current State: DoSnapshotSet

I am happy to try out the ideas this community have. If Veeam wan't me to open a case, i am more than happy to do so.

/Martin

[foggy]

Have you tried to check the VSS writer state by running "vssadmin list writers"?

And yes, I would recommend to open a case for this.

[Vitaliy S.]

Can you also tell me what SQL Server version you're running on this VM?

[ehrnst]

Have you tried to check the VSS writer state by running "vssadmin list writers"?

And yes, I would recommend to open a case for this.

Yes, none failed

Code: Select all

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {4f27186e-4f6a-4636-803a-f712777e659c}
   State: [11] Failed
   Last error: Retryable error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {c0e4bd23-67f3-44ba-ba91-e41dc796ab6d}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {fe9052d8-09af-4279-96f2-f2e9317486fc}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {48d08800-3fef-43fe-bf78-ea765b7eef04}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {d311ef13-9b98-44f9-8e98-1c813c461572}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {3b670166-918d-4de5-ac1a-bb7009812655}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {8a640de6-49d1-48da-bfbf-c7aa898582fd}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {c127a01b-ede6-4b88-ae31-da2da86b5163}
   State: [5] Waiting for completion
   Last error: No error

Can you also tell me what SQL Server version you're running on this VM?

This particular server runs MS SQL 2012 SP1

[veremin]

What about the failed state of SQL Writer? Can it be related to the problems you're experiencing?

Code: Select all

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {4f27186e-4f6a-4636-803a-f712777e659c}
   State: [11] Failed
   Last error: Retryable error

Thanks.

[ehrnst]

Oh, sorry. I did not notice that one today. I can try to restart the service, but i booted the whole server yesterday.

Unfortunately, still the same error
20.03.2014 10:23:35 :: Unable to release guest. Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x80042302].]

[Vitaliy S.]

Have you tried creating a VSS snapshot manually? Does it work?

[ehrnst]

Yes, created a shadow copy on each disk right now. No problem at all

[Vitaliy S.]

In this case I would suggest continue troubleshooting this issue with our technical team directly, as most likely we would need a WebEx session to see it live.

[ehrnst]

As requested, support case # 00534296

[adrobyazko]

As requested, support case # 00534296

Hello ehrnst,
I have the same issue (about 1500 DBs).
Could you describe results support case # 00534296?
Thanks.

[foggy]

According to the case notes, Martin was going to move databases to a new server, looking for the resolution. I suggest you opening a case for your own investigation, since the reasons of the issue could not be necessarily the same.