Local and replica jobs failing

VMware specific discussions

Local and replica jobs failing

Veeam Logoby mbra » Sun Dec 11, 2016 7:05 am

Ran into an issue, was hoping someone could assist with. Currently have a case open with Veeam, but not sure how active they are on the weekends. Case # 02000719.

We have 2 servers that are giving the following errors on local and replica jobs:

Failed to prepare guest for hot backup. Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec
Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec

Followed by:

Failed to prepare guest for hot backup. Error: VSSControl: -2147212522 Backup job failed. Discovery phase failed. Cannot start empty snapshot set. VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316
Error: VSSControl: -2147212522 Backup job failed. Discovery phase failed. Cannot start empty snapshot set. VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316

The 2 servers in question are Windows 2008 R2. Both at one point had the Idera CDP agent installed. Both were migrated to a new host and the vmware hardware version was upgraded. Both also had a couple of successful backups before this error started. Disabling the "Application aware' setting has got us working as a temporary workaround until we figure out the cause.

No scheduled shadow copies, listing the vss writers shows all normal, listing provider looks normal. Re-registered all items, reinstalled VMware tools, no change. Found no traces left of any old backup items. Running out of ideas, hopefully someone will have more information.

Have also posted items from the event logs I am seeing below:

Log Name: System
Source: volsnap
Date: 12/9/2016 5:05:20 PM
Event ID: 67
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: removed
Description:
The shadow copy of volume \\?\Volume{removed} being created failed to install.

Log Name: System
Source: Service Control Manager
Date: 12/9/2016 5:01:00 PM
Event ID: 7011
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: removed
Description:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VeeamVssSupport service.
mbra
Novice
 
Posts: 4
Liked: never
Joined: Sun Dec 11, 2016 6:50 am
Full Name: Mike Bradley

Re: Local and replica jobs failing

Veeam Logoby Mike Resseler » Mon Dec 12, 2016 6:06 am

Mike,

Good to see that you have a support case. I'm sure that they will be able to fix this for you. Please let us know the resolution if they find it.

From the information you show, and from the experience in the past, I would start my troubleshooting with the Idera CDP. The reason is "VSS_E_SNAPSHOT_SET_IN_PROGRESS" as this (in most cases) comes because there are multiple tools taking backups (and interfering with each other) or a leftover is still somewhere doing something (could be totally hidden as a background task)

Mike
Mike Resseler
Veeam Software
 
Posts: 3151
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Local and replica jobs failing

Veeam Logoby mbra » Tue Dec 13, 2016 2:36 am

We have your jobs set to run 2 minutes later after a failure. We generally only see "VSS_E_SNAPSHOT_SET_IN_PROGRESS" right after the first one fails.
mbra
Novice
 
Posts: 4
Liked: never
Joined: Sun Dec 11, 2016 6:50 am
Full Name: Mike Bradley

Re: Local and replica jobs failing

Veeam Logoby Mike Resseler » Tue Dec 13, 2016 6:21 am

If a snapshot without application aware processing works, then probably there is an issue with one of the in-guest writers. Any special application running on those servers?
Mike Resseler
Veeam Software
 
Posts: 3151
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Local and replica jobs failing

Veeam Logoby mbra » Wed Dec 14, 2016 3:26 am

One is a DC with just normal roles on it. The other is an Exchange server.
mbra
Novice
 
Posts: 4
Liked: never
Joined: Sun Dec 11, 2016 6:50 am
Full Name: Mike Bradley

Re: Local and replica jobs failing

Veeam Logoby Mike Resseler » Wed Dec 14, 2016 6:18 am

Do you already have more information from support?
Mike Resseler
Veeam Software
 
Posts: 3151
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Local and replica jobs failing

Veeam Logoby mbra » Sat Dec 17, 2016 8:39 pm

Unfortunately, was told it seems to be a Microsoft issue and doesn't seem like they can assist much more. Figured I would do a final post here and see if you guys had any additional thoughts.

Main issue seems to be with shadow copies. When trying to create a manual shadow copy, it erros out after about 20 minutes with:

The following errors are all that seems to show up in the event logs during this process: Error 0x80042306. The Shadow copy provider had an error. Check the system and Application event logs for more information.

Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{26309595-dc42-11e1-9ff4-806e6f6e6963} - 0000000000000140,0x0053c008,000000000029D540,0,000000000029E550,4096,[0]). hr = 0x80070079, The semaphore timeout period has expired.

The shadow copy of volume \\?\Volume{26309595-dc42-11e1-9ff4-806e6f6e6963} being created failed to install.

Going into device manager while trying to create a shadow copy, Under 'Storage volume shadow copy' I see 'Generic volume shadow copy' with a ! next to it. It reads 'Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)
mbra
Novice
 
Posts: 4
Liked: never
Joined: Sun Dec 11, 2016 6:50 am
Full Name: Mike Bradley

Re: Local and replica jobs failing

Veeam Logoby Mike Resseler » Mon Dec 19, 2016 7:31 am

Mike

Do you have the possibility to remove / add the hardware again? (With a rescan). In many cases a code 19 can be fixed like that.

So:
* Uninstall the "device"
* Do "Scan for Hardware changes"

If that doesn't work, you could try (but that is already a bigger risk) you could restart to "last known good configuration" which could load your OS with the missing registry information. But try the above first :-)

Mike
Mike Resseler
Veeam Software
 
Posts: 3151
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler


Return to VMware vSphere



Who is online

Users browsing this forum: Yahoo [Bot] and 11 guests