Replica Failing

[emf1967]

Hello.

I setup a replication job that stalls at 99% before failing with the following:

Failed to update VM registration Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_POST_RESTORE'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to perform post-restore steps. Failed to finalize restore session. Failed to process request to process post restore steps. Failed to execute post restore command. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_POST_RESTORE'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to perform post-restore steps. Failed to finalize restore session. Failed to process request to process post restore steps. Failed to execute post restore command.

I actually have another replication job that was created on Friday that was running, but now is also failing with the same error. I was unable to find any information on this error.

Using Veeam 7.x with Server 2012 Datacenter.

Case # 00437663

Thanks

Ed

[foggy]

Ed, do you see any related messages in the Application event log? What is the result of the "vssadmin list writers" command execution?

[emf1967]

When run on the guests I am trying to replicate:

Code: Select all

nysvfp02
-----------
Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {43fc68b4-bab6-40e8-809e-c77f7b755410}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {3e228bd9-915d-4ced-809b-716b8516096d}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {72d855b9-ab65-4ad6-bf8c-b9602c4f1dce}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {fa6fb00a-266c-4e96-b0b0-cff2635128c6}
   State: [1] Stable
   Last error: No error

Writer name: 'Dedup Writer'
   Writer Id: {41db4dbf-6046-470e-8ad5-d5081dfb1b70}
   Writer Instance Id: {b3e066e1-728e-449f-871f-bca7efa59821}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {c205e629-421c-44ad-b315-46edb59230d2}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {b382bd4c-def2-4fe4-967e-d22760f37840}
   State: [1] Stable
   Last error: No error

nysvad02
-----------

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {7d60b3eb-ce3e-47f1-b5ad-fcb7f5d20785}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {a35d161e-1e71-4626-85c6-532b5a5205f7}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {8622b3b4-6efd-4ee0-92b7-e92b6d5cc999}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {a2f0abcc-50a1-4135-a9b6-ca3d353f1f21}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {69d54b46-3dfd-4c83-95ea-d427c60a10a7}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {064d9e07-ddc0-43d0-af5a-2005d1b854d7}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {a0d0b395-2c3c-47c6-a8d3-1d088ca7e7c3}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {df03852b-2ca8-4c11-add0-f6272e0fddf6}
   State: [1] Stable
   Last error: No error

No errors in the Event Logs.

Thanks

Ed

[foggy]

Ok, let's wait for what support can suggest after reviewing the logs you've provided in the case.

[emf1967]

Update:

On Friday (9/6) night I rebooted both of my Hyper-V server (my production server - where the VMs are located, and my DR server - where the Replica jobs are run).

After rebooting both servers, the existing replica jobs completed, and I was able to add additional replica jobs which also completed. All were showing as stopped VMs in Hyper-V manager on the DR server. I then disabled the replica jobs until I could finish the proper scheduling, which I did Saturday night. Replicas were scheduled to run every 15 minutes from everyday from 3am to 6pm - leaving time for backups during the other hours. Still tweaking my scheduling, as I don't need that big a window for backups most nights.

When the replica jobs ran this morning at 3am, all failed with the VSS_WS_FAILED_AT_POST_RESTORE.

I uploaded the requests VSSADMIN logs from both servers this morning.

Thanks

Ed

[emf1967]

Update:

Last night before I went home I tried doing a gpupdate /force on DRSVHV01 - The host running the replica jobs. Once I did that, all the replicas completed. They have been updating on a 15 minute schedule without fail since last night.

I will note that I did try this before (some point last week) and it didn't work, so I am not sure if this is going to continue to work or not.

Thanks

[emf1967]

Another update:

Replica jobs are failing again. Everything was working fine up until 10:00am. All replicas after that started to fail with the same "VSS_WS_FAILED_AT_POST_RESTORE" error.

Running gpupdate /force did not fix the issue.

Waiting for support to get back to me - been 24 hrs since I sent them all the logs requested, and the issue was bumped up to the next level of support.

Ed

[Eric.Verreault]

On your host server drill down the event viewer to the Hyper-V VMMS and see which server is causing the issue. Once you find out run the vssadmin list writers on the problematic server. Let us know the result of that.

[emf1967]

On your host server drill down the event viewer to the Hyper-V VMMS and see which server is causing the issue. Once you find out run the vssadmin list writers on the problematic server. Let us know the result of that.

Already posted VSSADMIN LIST WRITER for both servers earlier in this message thread. I did send the Hyper-V VMMS log to Support.

The problem, I think, appears to be rights related. I re-ran GPUPDATE /FORCE and it did fix the issue (took a few minutes and one cycle of replication but everything started to work again).

I remember reading something about an issue with a certain right not updating the way it should in Server 2012 and running gpupdate /force was a quick fix. There was another recommendation about moving Hyper-V servers into their own OU and not inheriting any policies - but before I do that, I want to confirm that it would help.

Thanks

Ed

[eakinsb]

Are there any updates on this issue?

I am having a similar issue, I am getting the same error attempting to restore a VM. The files restore successfully, but then errors registering with Hyper-V VSS writer.

Thanks

[emf1967]

For us, the issue appears to be related to this:

http://support.microsoft.com/kb/2779204

We tried Option #2 in the KB and solved the issue.

It has been over 24 hrs since we implemented the fix / workaround and we have not had a replication failure.

Note that this is a Windows Server 2012 issue and not Veeam.

Hope this helps.

[eakinsb]

Thanks for your quick reply, I will check out the KB article now. I agree this is a Server 2012 issue, but I have had no luck finding a solution so I thought I might try here.

Thanks again