Backup 4.0 jobs failing with quiesced snapshot error

[vmmatty]

I'm using Veeam Backup 4.0 against a VI3 environment consisting of 7 ESX hosts. Most of my backups have been successful since we first the environment up and running last week. Last night, however, I had 6 or 7 VMs fail with the following error:

Creating a quiesced snapshot failed because the (user-supplied) custom pre-freeze script in the virtual machine exited with a non-zero return code.

Despite what this error says I'm not using VCB to backup the servers. I am using the standard Network Backup method against all of my virtual machines. To make it even more confusing, none of the servers have the VMware Tools VSS driver installed so it can't even quiesce the VMs if it wanted to. If I disable the option to use VMware Tools quiescing in the backup job then the job runs. Can't I leave that option enabled even if the driver isn't installed?

I got successful backups of all of these virtual machines last week, and the only thing that has changed is that the ESX hosts and all virtual machines were rebooted for a SAN firmware upgrade this past weekend. Any thoughts/suggestions on where to start with this one?

[Gostev]

Check if your VMs has existing snapshots, this may cause this error for Linux VMs (I am assuming you are using Linux VMs, which is why you are not leveraging Veeam VSS integration).

[vmmatty]

No, these are Windows 2008 (mostly) virtual machines. I'm not using the Veeam VSS driver because I inherited this environment and didn't want to start installing things into the guests.

None of the VMs had any existing snapshots - that was the first thing I checked. Aside from installing the Veeam driver or disabling quiescing altogether, any other thoughts? I'm trying to figure out why we had a bunch of failures when the backups worked previously. No settings were changed on the guests or in the backup job itself.

[Gostev]

Veeam VSS does not require persistent driver/agent, and using it would resolve your issue.

If it does not work for you, then we can help you to confirm that this is VMware quiesce issue that can be observed without Veeam Backup installed, so that you can open support case with VMware. To do this, try to perform a snapshot of affected VM using MOB (Managed Object Browser) with quiesce option enabled. You should connect using IP address to your VC or ESX host (it depends on how this VM is added to your backup job), and create the snapshot. It may not be easy if you never done this before, so if you need any instructions of how to make a snapshot using MOB, please contact us at support@veeam.com.

[vmmatty]

Thanks Gostev, using the Veeam VSS option did actually allow the backup to proceed on one of the servers that failed. I left the "Enable VMware Tools quiesce" option enabled as well, which was previously causing the backups to fail.

Just to be clear - is it safe to use the Veeam VSS option for all guests? The documentation states the following:

The Enable VMware tools quiescence option is selected by default. It is recommended to use it unless you are backing up VMs with highly transactional applications (such as Exchange and SQL) — in this case, there is a risk of the application process failure.

I have Exchange, SQL, and some other transactional servers. Is it safe to use the Veeam VSS option on all of these servers, or should I be selectively enabling it only on certain servers? If you have both the VMware Tools option enabled and the Veeam VSS option enabled, which one does it actually use?

[Gostev]

Yes, technically Veeam VSS is single best and safest choice for every Windows based server. Our VSS integration implementation is currently most complete and most advanced on the image-level backup market. We have a number of existing threads on this forum discussing it and how it compares to VMware VSS and VMware quiescence, please review those if you are interested in details.

Besides, Veeam VSS is our own code that we fully own/support without having to refer you to VMware Support as in case of VMware quiescence issues like you have encountered.

[vmmatty]

Thanks for your help Gostev. I'll enable Veeam VSS integration for all of my servers,including Exchange and SQL, and see how the backups run tonight.

[vmmatty]

Unfortunately this didn't go as smoothly as I had hoped. After making the change to use the Veeam VSS integration we ran into an issue backing up our SQL cluster. It appears that the actual backup completed, as I can see in the Realtime stats that it backed up 40GB out of 40GB. The "Delete Snapshot" action actually failed (though there is no longer any snapshot), and at the same time the SQL cluster failed as well. The backup job itself then failed because it couldn't talk to the Veeam database.

I was able to get a successful backup of both SQL cluster nodes prior to using the Veeam VSS integration so I suspect it might be playing a role. The actual Veeam job log only shows that it lost connectivity to the SQL server, which is explained by the fact that the cluster failed during the snapshot deletion process.

Any suggestions?

[Gostev]

Are you trying to backup the same database where Veeam Backup DB resides?

[vmmatty]

Yes, it is the SQL server where the Veeam database resides. I was able to get a successful backup of the server prior to using Veeam VSS, so I'm not sure what the issue is.

I've opened up a ticket with support at this point. Thanks for your help with this, hopefully support can resolve for me.

[Gostev]

I think it is known issue - VSS freezes the backend SQL used by the job and this creates a deadlock (SQL cannot process the Veeam Backup job's request). But may be our support is aware about some workarounds.

[vmmatty]

Thanks Gostev. I could see it being a known issue that would cause the backup job to fail (based on the situation you describe), but I wouldn't expect both SQL cluster nodes to crash. I'll see what they say and will post back here if they reveal anything worth sharing with everyone.

Thanks again for all of your help.

[damduq]

I'm not sure that I should continue this topic or create a new one.
I'm always getting quiescing issues with 2 Win 2003 server + MS SQL.
I'ts both the case with Sync and Vss mechanisms.
I made some tests directly with vCenter 4 snapshots :
Each time I make a quiesced Snapshot including memory it works fine.
Each time I make it without memory, it fails. (I made the test 10 times...)

Of course it's a VMware issue, but quiescing mechanisms are quite similar.

Any comment ?

[Gostev]

Damien, are you saying you have the same issue with Veeam VSS enabled, or is this for VMware VSS or VMware SYNC only?

[damduq]

Hello Gostev

In fact I'm experiencing quiescing issues in both side :
In Veeam, with standard vmtools quiescing I get this error
'Creating snapshot
CreateSnapshot failed, vmRef "vm-283", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True"
Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.'

With Veeam VSS quiescing, I get this error
'Creating snapshot
VSSFreezer: -2147467259 Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x8004230f].'

In vSphere if I ask for a snapshot without memory VM recording
and quiescing of guest on, it fails with time limit for holding error.

If I ask the same snapshot, just adding Memory vm recording, it works...

It's puzzling me !

Damien

[Gostev]

Sounds like both VMs have some VSS framework issue.

Try running the following two commands in the VM and see if output contains any errors:

Code: Select all

vssadmin list writers
vssadmin list providers

Also, is there any errors in Event Log on these VMs?

[damduq]

Thank's for your help... Here is what I get :

vssadmin list writers :

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {455302fb-88f7-4dd9-8312-2369eb2a4f68}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {03db0e77-9623-41a5-abbd-15885fa2e016}
State: [7] Failed
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {42cac798-1e4e-4360-bf93-fde16b3f7f04}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {45f66e83-856e-4179-82fd-4b6e3dc7efe8}
State: [1] Stable
Last error: No error

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {fc98b96b-71de-4cb1-9652-22fbe4a7fd6c}
State: [1] Stable
Last error: No error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {5adb1170-bff9-49a8-b40a-c32833eabe90}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {f9c15544-bb7f-468c-9215-0cf5649d867d}
State: [7] Failed
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {ffe8f770-2f65-4547-96b7-162726a64db7}
State: [7] Failed
Last error: No error


vssadmin list providers :
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Provider name: 'VMware Snapshot Provider'
Provider type: Software
Provider Id: {564d7761-7265-2056-5353-2050726f7669}
Version: 1.0.0

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

For the event manager I got a VSS error message in application section
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {564d7761-7265-2056-5353-2050726f7669}. Routine details EndPrepareSnapshots({a262ffa4-b265-4665-9908-bca360523747}) [hr = 0x80004005].
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

and an event manager system error :
Application popup: VeeamGuestAgent.exe - Application Error : The instruction at "0x7c82a776" referenced memory at "0x0073005a". The memory could not be "read".
Click on OK to terminate the program
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

[Gostev]

I can see there is an issue with SQL and IIS VSS writers, they are in the failed state. This is what is likely causing the issue.

[damduq]

Ok, I've rechecked the vss services.
After a reboot of the vm, all the writers were ok :

C:\Documents and Settings\Administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {0ca9ee81-8fa2-4db9-b382-a30467ed93a7}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {442cf149-4260-4276-a686-0652cf0465c7}
State: [1] Stable
Last error: No error

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {631a8591-abbf-473e-beda-310ab0fae50d}
State: [1] Stable
Last error: No error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {67bbe644-1416-4dbe-9715-f32f512bade6}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {823eeb96-4c2e-4388-8bf2-3459a5801e0d}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {941723f1-e333-4639-8d06-105c6dfe25b1}
State: [1] Stable
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {46bc633b-855e-48c0-a4e1-f0475129339a}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {cc0b12bb-03a2-43a1-92ca-2512dd9481be}
State: [1] Stable
Last error: No error


Then I tried to start a Veeam Backup, without vmtools quiescing and With Veeam VSS quiescing.

The Backup failed with this message
Creating snapshot
VSSFreezer: -2147467259 Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x8004230f].

The event log for Application gives me a VSS error
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {564d7761-7265-2056-5353-2050726f7669}. Routine details EndPrepareSnapshots({895efcf3-4e18-418a-b5e8-7215f304cdda}) [hr = 0x80004005].


After this message, if I check the writers, several of them are now on status Failed.

C:\Documents and Settings\Administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {0ca9ee81-8fa2-4db9-b382-a30467ed93a7}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {442cf149-4260-4276-a686-0652cf0465c7}
State: [7] Failed
Last error: No error

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {631a8591-abbf-473e-beda-310ab0fae50d}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {823eeb96-4c2e-4388-8bf2-3459a5801e0d}
State: [7] Failed
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {941723f1-e333-4639-8d06-105c6dfe25b1}
State: [7] Failed
Last error: No error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {67bbe644-1416-4dbe-9715-f32f512bade6}
State: [7] Failed
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {46bc633b-855e-48c0-a4e1-f0475129339a}
State: [7] Failed
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {cc0b12bb-03a2-43a1-92ca-2512dd9481be}
State: [7] Failed
Last error: No error

I've searched a bit in VMware and Microsoft Websites, but didn't found/understand any valuable help...

[Vitaliy S.]

Hello Damien,

That's definitely seems to be a Microsoft VSS issue, I've done some search based on your posts and output and have found Microsoft's hotfix, try applying that fix and see whether it changed anything for you or not:

http://support.microsoft.com/kb/833167

Thank you!

[damduq]

Hello
I agree with you ! It really seems to be a Microsoft VSS issue.
But I was already up to date with my 2003 server, and the patch has confirm it during installation.
I need to investigate a bit more, but it takes time...
Anyway thank's for your help. Damien.
Nota : I've finally bought veem Backup.

[Gostev]

Congratulations now you can officially use our support for all those issues

[aqualityplace]

Hi

Sorry to hijack this thread did you manage to fix your problem we have a customer with a simular problem - here is the output and a copy of one of the errors we found in the event log. I have applied the following vss patch http://support.microsoft.com/kb/940252 still no joy. I have a call open with veeam but sometimes the forums are just as helpful - if I get a solution from veeam I will post the fix

Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: 08/04/2010
Time: 22:34:05
User: N/A
Computer: XXXXXXX
Description:
Application popup: VeeamGuestAgent.exe - Application Error : The instruction at "0x7c82a774" referenced memory at "0x0020006c". The memory could not be "read".

Click on OK to terminate the program

This output was provided by the customer before the vss patch was applied

Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

C:\Documents and Settings\alisala>Vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {2c4ce85a-ff43-4b40-82b1-5a9fdda6ca4f}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {fabe43b9-35c5-4903-b870-3f62e599fce2}
State: [5] Waiting for completion
Last error: No error

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {b396fc4a-1197-4a89-8a23-da88b0a61fc2}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {7bc57179-2c03-4211-97d9-aaea92a514f0}
State: [1] Stable
Last error: No error

Writer name: 'ADAM (VMwareVCMSDS) Writer'
Writer Id: {dd846aaa-a1b6-42a8-aaf8-03dcb6114bfd}
Writer Instance Id: {460a7764-92f7-4779-851c-084d6a18eb15}
State: [5] Waiting for completion
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {7c996667-34b7-4a76-8bc5-759237567814}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {67402982-3c19-445e-baa2-7e2ee119cb34}
State: [5] Waiting for completion
Last error: No error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {39209a15-e227-4fd2-8acd-11457bdd94c8}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {dea21f22-3ef2-4df1-908f-f42d1afe03d7}
State: [1] Stable
Last error: No error

Writer name: 'Removable Storage Manager'
Writer Id: {5d3c3e01-0297-445b-aa81-a48d7151e235}
Writer Instance Id: {d32d6c58-0e60-48a1-b2f8-42e05aee8fe3}
State: [1] Stable
Last error: No error


C:\Documents and Settings\alisala>Vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

[Gostev]

It is not the same issue. What version of Veeam Backup is this?