We upgraded one of our Veeam instances to 9.5 U3 from U2 on Friday 23rd March and we have been having troubles ever since.
Background:
- Environment is vCenter + ESXi 6.5
- We have done this upgrade in 2 other datacenters with same version and same setup, with no errors.
- Veeam Backup 9.5 U3 with Backup Server, 1 Local Proxy and 1 Offsite Proxy
- Repositories are CIFS under NetApp FAS Storage, using Storage Snapshot
- Failures / behaviors are:
- :: Error: Failed to prepare VM for processing: Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [. Freeze timeout occurred. Total freeze time: 60 seconds].
- :: Unable to release guest. Error: VSSControl: Failed to freeze guest over network, wait timeout
- :: Failed to prepare guest for hot backup. Error: VSSControl: Freeze state is none
- :: Failed to prepare guest for hot backup. Error: VSSControl: -2147212302 Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{aa75981d-11e1-4745-a867-7ee1aa3e403c}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].
- Most of these errors take ages to appear in the Veeam console, some even hours before they throw an error
- Processing of backups stops at "Queued for processing" and doesn't progress from there
- After a job gets "stuck" or starts throwing errors, we get SOAP errors for other environments such as monitoring and scripts trying to connect to vCenter. This does NOT happen when Veeam jobs are not running.
- Checked Veeam Logs and
Tests and relevant information:
- Jobs with no guest processing work ok
- Jobs with guest processing using vmware quiescing instead of VSS work ok, but VMs freeze and we receive 1-2 minute ping/outage every time we launch a backup, so that's not an option
- vCenter has been restarted several times, the second a VM fails it starts giving problems to connect via SOAP from our monitoring system and other applications (powerscript / powershell)
- Upgraded and reinstalled vmtools in several affected VMs, same errors
- Tested creating a manual snapshot from vcenter with quiescing then relaunching, solved the error in some VMs, but reocurrs after a few tries.
- Tried with different sets of credentials for vCenter, up to full domain administrator.
Obviously we can't launch all of our jobs without app aware as we need logs truncated and app awareness for SQL, AD, an other critical environments, so we are stuck at this moment.
Any help would be appreciated! We don't know what else to test, and we we are just doing random testing at this point which is frustrating. If anyone has experienced anything similar we would gladly appreciate any pointers or help
Out ticket with Veeam support is #02692012.
Best Regards
Nissy