VSS Snapshot failures on several agent backups with S1

[Jeff M]

We have been having VSS Snapshot failures on agent based backups. Case #04230270 Microsoft VSS issue. We think we have narrowed it down to the culprit being the SentinelOne Endpoint Security program running on these servers which VSS snapshots fail. Our security team has tried the Veeam recommended exclusions which have not worked They have been in contact with SentinelOne who have also recommended exclusions which also have not worked. The only thing that has worked is completely shutting down all SentinelOne services on the server before job starts and VSS Snapshot will succeed and the backup will complete.
Here is the kicker, I can and have created a standalone volume level jobs for several of these failing servers which work every time regardless of exclusions for S1 or not. However, only one volume per job will work. So I have to create a job for each volume before VSS Snapshot will succeed. I cant place 2 volumes C:\ and D:\ in the same volume level job or VSS Snapshot fails. Just like the normal Full backup fails every time.
Any Ideas anyone? Volume level job runs the same as far as I can tell. Both process VSS Snapshot first then proceed. Another weirdness is If the server is rebooted prior to normal B&R job with S1 running, the job succeeds, VSS snapshot succeeds 1 time only. Fails every time thereafter.

[Dima P.]

Hello Jeff,

Other users had some issues with the Sentinel One solution but all the problems were resolved by setting up the correct exclusions / rules on the Sentinel One's end. The described behavior looks unexpected but since the issue disappears when you disable the endpoint protection software, I'd suggest to keep working with the Sentinel One support team. Let us know if you need any assistance from our end. Cheers!

[Jeff M]

Well, why does a volume level job of the same drive succeed with no exclusions in place at? As long as its just c:\ and no other drives in the volume level job, if any other drive is included VSS fails just like the normal job. However a volume level of all other drives will succeed. So basically (As a workaround) these agent based backups require 2 volume level jobs to back up 1 server. One job for C:\ one job for all the other drives. That works every time. We have not yet been able to get exclusions for S1 working reliably.
I thought we had it but some failed again tonight.

[johnwatson]

Veeam Error:
Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a transient error. If the backup process is retried, the error may not reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.


Solution:

Open an elevated command prompt
1. Navigate to the Sentinel Agent folder (ie. cd "C:\Program Files\SentinelOne\Sentinel Agent 21.7.6.1219")
2. Run the following command: sentinelCtl.exe config -p agent.vssConfig.agentVssWriters -k "agent passphrase" -v false
a. Your agent passphrase is obtained from Sentinelone dashboard select the computer and go to Action and click on Show Passphrase

Verification:
1. Run the following command: sentinelCtl.exe config agent.vssConfig.agentVssWriters

To enable the VSS writers again once SentinelOne is patched:
1. Run the following command: sentinelCtl.exe config -p agent.vssConfig.agentVssWriters -k "agent passphrase" -v true

[mark.odonnell]

https://euce1-swprd2.sentinelone.net/do ... -unitrends

sentinelctl config -p agent.vssConfig.agentVssWriters -v false -k "passphrase"
sentinelctl config -p agent.vssConfig.enableResearchDataCollectorVssWriter -v false -k "passphrase"

still investigating my issue, found this thought I would share.