-
- Enthusiast
- Posts: 34
- Liked: 3 times
- Joined: Jan 13, 2015 4:31 am
- Full Name: Jeffrey Michael James
- Location: Texas Tech Univ. TOSM Computer Center, 8th Street & Boston Avenue, Lubbock, TX 79409-3051
- Contact:
VSS Snapshot failures on several agent backups with S1
We have been having VSS Snapshot failures on agent based backups. Case #04230270 Microsoft VSS issue. We think we have narrowed it down to the culprit being the SentinelOne Endpoint Security program running on these servers which VSS snapshots fail. Our security team has tried the Veeam recommended exclusions which have not worked They have been in contact with SentinelOne who have also recommended exclusions which also have not worked. The only thing that has worked is completely shutting down all SentinelOne services on the server before job starts and VSS Snapshot will succeed and the backup will complete.
Here is the kicker, I can and have created a standalone volume level jobs for several of these failing servers which work every time regardless of exclusions for S1 or not. However, only one volume per job will work. So I have to create a job for each volume before VSS Snapshot will succeed. I cant place 2 volumes C:\ and D:\ in the same volume level job or VSS Snapshot fails. Just like the normal Full backup fails every time.
Any Ideas anyone? Volume level job runs the same as far as I can tell. Both process VSS Snapshot first then proceed. Another weirdness is If the server is rebooted prior to normal B&R job with S1 running, the job succeeds, VSS snapshot succeeds 1 time only. Fails every time thereafter.
Here is the kicker, I can and have created a standalone volume level jobs for several of these failing servers which work every time regardless of exclusions for S1 or not. However, only one volume per job will work. So I have to create a job for each volume before VSS Snapshot will succeed. I cant place 2 volumes C:\ and D:\ in the same volume level job or VSS Snapshot fails. Just like the normal Full backup fails every time.
Any Ideas anyone? Volume level job runs the same as far as I can tell. Both process VSS Snapshot first then proceed. Another weirdness is If the server is rebooted prior to normal B&R job with S1 running, the job succeeds, VSS snapshot succeeds 1 time only. Fails every time thereafter.
Jeff M
Data Center Operations
Technology Operations & Systems Management
Texas Tech University System
jeff.james@ttu.edu
Data Center Operations
Technology Operations & Systems Management
Texas Tech University System
jeff.james@ttu.edu
-
- Product Manager
- Posts: 14720
- Liked: 1705 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: VSS Snapshot failures on several agent backups with S1
Hello Jeff,
Other users had some issues with the Sentinel One solution but all the problems were resolved by setting up the correct exclusions / rules on the Sentinel One's end. The described behavior looks unexpected but since the issue disappears when you disable the endpoint protection software, I'd suggest to keep working with the Sentinel One support team. Let us know if you need any assistance from our end. Cheers!
Other users had some issues with the Sentinel One solution but all the problems were resolved by setting up the correct exclusions / rules on the Sentinel One's end. The described behavior looks unexpected but since the issue disappears when you disable the endpoint protection software, I'd suggest to keep working with the Sentinel One support team. Let us know if you need any assistance from our end. Cheers!
-
- Enthusiast
- Posts: 34
- Liked: 3 times
- Joined: Jan 13, 2015 4:31 am
- Full Name: Jeffrey Michael James
- Location: Texas Tech Univ. TOSM Computer Center, 8th Street & Boston Avenue, Lubbock, TX 79409-3051
- Contact:
Re: VSS Snapshot failures on several agent backups with S1
Well, why does a volume level job of the same drive succeed with no exclusions in place at? As long as its just c:\ and no other drives in the volume level job, if any other drive is included VSS fails just like the normal job. However a volume level of all other drives will succeed. So basically (As a workaround) these agent based backups require 2 volume level jobs to back up 1 server. One job for C:\ one job for all the other drives. That works every time. We have not yet been able to get exclusions for S1 working reliably.
I thought we had it but some failed again tonight.
I thought we had it but some failed again tonight.
Jeff M
Data Center Operations
Technology Operations & Systems Management
Texas Tech University System
jeff.james@ttu.edu
Data Center Operations
Technology Operations & Systems Management
Texas Tech University System
jeff.james@ttu.edu
-
- Influencer
- Posts: 11
- Liked: 9 times
- Joined: Feb 16, 2021 7:42 pm
- Full Name: John Watson
- Contact:
Re: VSS Snapshot failures on several agent backups with S1
Veeam Error:
Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a transient error. If the backup process is retried, the error may not reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.
Solution:
Open an elevated command prompt
1. Navigate to the Sentinel Agent folder (ie. cd "C:\Program Files\SentinelOne\Sentinel Agent 21.7.6.1219")
2. Run the following command: sentinelCtl.exe config -p agent.vssConfig.agentVssWriters -k "agent passphrase" -v false
a. Your agent passphrase is obtained from Sentinelone dashboard select the computer and go to Action and click on Show Passphrase
Verification:
1. Run the following command: sentinelCtl.exe config agent.vssConfig.agentVssWriters
To enable the VSS writers again once SentinelOne is patched:
1. Run the following command: sentinelCtl.exe config -p agent.vssConfig.agentVssWriters -k "agent passphrase" -v true
Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a transient error. If the backup process is retried, the error may not reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.
Solution:
Open an elevated command prompt
1. Navigate to the Sentinel Agent folder (ie. cd "C:\Program Files\SentinelOne\Sentinel Agent 21.7.6.1219")
2. Run the following command: sentinelCtl.exe config -p agent.vssConfig.agentVssWriters -k "agent passphrase" -v false
a. Your agent passphrase is obtained from Sentinelone dashboard select the computer and go to Action and click on Show Passphrase
Verification:
1. Run the following command: sentinelCtl.exe config agent.vssConfig.agentVssWriters
To enable the VSS writers again once SentinelOne is patched:
1. Run the following command: sentinelCtl.exe config -p agent.vssConfig.agentVssWriters -k "agent passphrase" -v true
-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Nov 02, 2023 9:05 am
- Full Name: mark odonnell
- Contact:
Re: VSS Snapshot failures on several agent backups with S1
https://euce1-swprd2.sentinelone.net/do ... -unitrends
sentinelctl config -p agent.vssConfig.agentVssWriters -v false -k "passphrase"
sentinelctl config -p agent.vssConfig.enableResearchDataCollectorVssWriter -v false -k "passphrase"
still investigating my issue, found this thought I would share.
sentinelctl config -p agent.vssConfig.agentVssWriters -v false -k "passphrase"
sentinelctl config -p agent.vssConfig.enableResearchDataCollectorVssWriter -v false -k "passphrase"
still investigating my issue, found this thought I would share.
Who is online
Users browsing this forum: No registered users and 15 guests