VSS writers constanly failing during backup

[TwoTabs]

Veeam Support - Case # 03375512

For the past week we have been experiencing failed backup's due to A VSS writer issue. I have restarted the physical machine and have restarted services. Nothing seems to be fixing the issue. This started happening after the latest update. I also ran shadowdisk and it didnt come up with any problems. The Microsoft Hyper v VSS writer always come back failed with a non retryable error

This is just an example of one of them.

Code: Select all

1/28/2019 10:48:23 AM :: Queued for processing at 1/28/2019 10:48:23 AM  
1/28/2019 10:48:23 AM :: Required backup infrastructure resources have been assigned  
1/28/2019 10:48:28 AM :: VM processing started at 1/28/2019 10:48:28 AM  
1/28/2019 10:48:28 AM :: VM size: 127.0 GB (28.3 GB used)  
1/28/2019 10:48:28 AM :: VM is now in the required state for processing  
1/28/2019 10:48:30 AM :: Preparing to create snapshot  
1/28/2019 10:49:01 AM :: Using guest interaction proxy FFLD-Venyu.fairfield.com (Same subnet)  
1/28/2019 10:49:04 AM :: Inventorying guest system  
1/28/2019 10:49:08 AM :: Subscribing to guest processing components  
1/28/2019 10:49:12 AM :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
1/28/2019 10:51:44 AM :: Transaction logs will not be truncated  
1/28/2019 10:51:54 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
1/28/2019 10:51:54 AM :: Task has been rescheduled  
1/28/2019 10:52:17 AM :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
1/28/2019 10:54:42 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
1/28/2019 10:54:42 AM :: Queued for processing at 1/28/2019 10:54:42 AM  
1/28/2019 10:54:44 AM :: Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
1/28/2019 10:54:44 AM :: Network traffic verification detected no corrupted blocks  
1/28/2019 10:54:44 AM :: Processing finished with errors at 1/28/2019 10:54:44 AM

[nielsengelen]

Best would be to let support assist you here ad vss related errors can be very specific per case. They will have more insight into full logs and may ask you to perform some additional tasks. Thanks.

[Mike Resseler]

Hi John,

Niels is correct about working with the engineers. The VSS_WS_FAILED_AT_PREPARE_SNAPSHOT mostly means that inside the guest VM some writers are called (or prepared) and fail. Is it always the same backup job? And the same VM(s)? Or on a specific host?

[ianbutton1]

Hi,
A workaround that I used after discovering this happening on our Hyper-V guests was to create a batch-file that stopped & restarted all the VSS services on the failed servers
1. Either manually - on any guests showing the above message in Veeam GUI logs, restart 3 Windows services – “Hyper-V Volume Shadow Copy Requestor”, “Microsoft Software Shadow Copy Provider”, and “Volume Shadow Copy”; then retry the job. See also Veeam KB2041.
2. Or a quick way to stop & start these services on a list of servers is to run CMD.EXE (as administrator) on VBR server, then run the batch-file (below) with syntax “sc1 failedserver1 failedserver2 failedserver3 ...”

Haven't seen the problem lately, though - perhaps a Windows update resolved the problem, or upgrading to 9.5 U3

Batch file below. It's pretty crude, but it worked! Good luck!

Ian

Code: Select all

REM SC1.bat - IB 13/7/2017 
REM Resets shadow copy services on targets, for retries of Veeam Hyper-V jobs
REM Syntax is "sc1 server1 server2 ..."
@echo off
cls
if "%1"=="" goto error
:start
if "%1"=="" goto end
echo.
echo Processing server %1
echo -------------------------------------
echo Stopping 3 services - please wait (up to a minute) .....
REM stop Hyper-V Volume Shadow Copy Requestor service
echo 1 vmicvss . . .
sc \\%1 stop vmicvss > nul
REM stop Microsoft Software Shadow Copy Provider service
echo 2 swprv . . .
sc \\%1 stop swprv > nul
REM stop Volume Shadow Copy service
echo 3 vss . . .
sc \\%1 stop vss > nul

echo Restarting 3 services - please wait (up to a minute) .....
REM start Hyper-V Volume Shadow Copy Requestor service
echo 1 vmicvss . . .
sc \\%1 start vmicvss > nul
REM start Microsoft Software Shadow Copy Provider service
echo 2 swprv . . .
sc \\%1 start swprv > nul
REM start Volume Shadow Copy service
echo 3 vss . . .
sc \\%1 start vss > nul
echo Now please wait 20 seconds for restarts to finish, then ...
pause

echo Checking services are running - please wait (up to a minute) .....
sc \\%1 query vmicvss
sc \\%1 query swprv
sc \\%1 query vss
echo If all 3 show "State : 4 Running", then Veeam backup of server %1 should work.
echo =======================================
pause

shift
goto start
cls

:end
Echo Now retry failed Veeam job
pause
goto :EOF

:error
Echo No server name entered - syntax is "sc1 server1 server2 ..."
pause