Host-based backup of VMware vSphere VMs.
Post Reply
mbra
Novice
Posts: 4
Liked: never
Joined: Dec 11, 2016 6:50 am
Full Name: Mike Bradley
Contact:

Local and replica jobs failing

Post by mbra »

Ran into an issue, was hoping someone could assist with. Currently have a case open with Veeam, but not sure how active they are on the weekends. Case # 02000719.

We have 2 servers that are giving the following errors on local and replica jobs:

Failed to prepare guest for hot backup. Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec
Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec

Followed by:

Failed to prepare guest for hot backup. Error: VSSControl: -2147212522 Backup job failed. Discovery phase failed. Cannot start empty snapshot set. VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316
Error: VSSControl: -2147212522 Backup job failed. Discovery phase failed. Cannot start empty snapshot set. VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316

The 2 servers in question are Windows 2008 R2. Both at one point had the Idera CDP agent installed. Both were migrated to a new host and the vmware hardware version was upgraded. Both also had a couple of successful backups before this error started. Disabling the "Application aware' setting has got us working as a temporary workaround until we figure out the cause.

No scheduled shadow copies, listing the vss writers shows all normal, listing provider looks normal. Re-registered all items, reinstalled VMware tools, no change. Found no traces left of any old backup items. Running out of ideas, hopefully someone will have more information.

Have also posted items from the event logs I am seeing below:

Log Name: System
Source: volsnap
Date: 12/9/2016 5:05:20 PM
Event ID: 67
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: removed
Description:
The shadow copy of volume \\?\Volume{removed} being created failed to install.

Log Name: System
Source: Service Control Manager
Date: 12/9/2016 5:01:00 PM
Event ID: 7011
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: removed
Description:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VeeamVssSupport service.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Local and replica jobs failing

Post by Mike Resseler »

Mike,

Good to see that you have a support case. I'm sure that they will be able to fix this for you. Please let us know the resolution if they find it.

From the information you show, and from the experience in the past, I would start my troubleshooting with the Idera CDP. The reason is "VSS_E_SNAPSHOT_SET_IN_PROGRESS" as this (in most cases) comes because there are multiple tools taking backups (and interfering with each other) or a leftover is still somewhere doing something (could be totally hidden as a background task)

Mike
mbra
Novice
Posts: 4
Liked: never
Joined: Dec 11, 2016 6:50 am
Full Name: Mike Bradley
Contact:

Re: Local and replica jobs failing

Post by mbra »

We have your jobs set to run 2 minutes later after a failure. We generally only see "VSS_E_SNAPSHOT_SET_IN_PROGRESS" right after the first one fails.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Local and replica jobs failing

Post by Mike Resseler »

If a snapshot without application aware processing works, then probably there is an issue with one of the in-guest writers. Any special application running on those servers?
mbra
Novice
Posts: 4
Liked: never
Joined: Dec 11, 2016 6:50 am
Full Name: Mike Bradley
Contact:

Re: Local and replica jobs failing

Post by mbra »

One is a DC with just normal roles on it. The other is an Exchange server.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Local and replica jobs failing

Post by Mike Resseler »

Do you already have more information from support?
mbra
Novice
Posts: 4
Liked: never
Joined: Dec 11, 2016 6:50 am
Full Name: Mike Bradley
Contact:

Re: Local and replica jobs failing

Post by mbra »

Unfortunately, was told it seems to be a Microsoft issue and doesn't seem like they can assist much more. Figured I would do a final post here and see if you guys had any additional thoughts.

Main issue seems to be with shadow copies. When trying to create a manual shadow copy, it erros out after about 20 minutes with:

The following errors are all that seems to show up in the event logs during this process: Error 0x80042306. The Shadow copy provider had an error. Check the system and Application event logs for more information.

Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{26309595-dc42-11e1-9ff4-806e6f6e6963} - 0000000000000140,0x0053c008,000000000029D540,0,000000000029E550,4096,[0]). hr = 0x80070079, The semaphore timeout period has expired.

The shadow copy of volume \\?\Volume{26309595-dc42-11e1-9ff4-806e6f6e6963} being created failed to install.

Going into device manager while trying to create a shadow copy, Under 'Storage volume shadow copy' I see 'Generic volume shadow copy' with a ! next to it. It reads 'Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Local and replica jobs failing

Post by Mike Resseler »

Mike

Do you have the possibility to remove / add the hardware again? (With a rescan). In many cases a code 19 can be fixed like that.

So:
* Uninstall the "device"
* Do "Scan for Hardware changes"

If that doesn't work, you could try (but that is already a bigger risk) you could restart to "last known good configuration" which could load your OS with the missing registry information. But try the above first :-)

Mike
Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 31 guests