SplunkForwarder service prevent the deletion of VeeamVssSupport service

[Quentin]

Hello there!

I recently encountered a strange issue: some Windows Server VM with AAIP enabled failed with the following error: 'Error: Unable to perform installation. Unable to register and start service. Attempt 10 of 10. Cannot register management service. Service name: [VeeamVssSupport]. Cannot create service. Machine: [C2023TRVWWD]. Service name: VeeamVssSupport. Service binary path: C:\Windows\VeeamVssSupport\VeeamGuestHelper.exe. Service account: []. Cannot create service. Win32 error: The specified service has been marked for deletion. Code: 1072 '

The strange thing here is that this error did not occur on the same VM or at every backup. Sometimes a retry solved the issue, but sometimes a reboot was needed.
I didn't find anything relevant in the Veeam or Windows Event logs.

It turns out that the SplunkForwarder service is the culprit.

When I logged into a VM after the backup ended, the VeeamVssSupport service was still present but stopped. I tried 'sc delete VeeamVssSupport', but received an error message saying the service was marked for deletion.
Then I found this Reddit post: https://www.reddit.com/r/sysadmin/comme ... e_without/. Since the issue was similar, I had to establish what those VMs had in common.
The only thing they had in common was the SplunkForwarder service, so I tried restarting it and it worked. However, when the VeeamVssSupport service is marked for deletion and I restart the SplunkForwarder service, the Veeam service is deleted.

I am now searching for a way to prevent this. I asked the Splunk team if the VeeamVssSupport service could be excluded from Splunk analysis, but I won't receive an answer until next year.
Is there a way to restart the Splunk service with Veeam before or after the backup?

Thanks for reading!
Regards,

[Gostev]

Sure thing, we provide pre- and post-job script functionality to enable users to do things like that.

[Quentin]

Hi Gostev,

As I understand it, the pre- and post-job scripts run on the VBR server, not on every VM in the job. I may have missed something about it.
The pre-freeze and post-thaw scripts don't fit either, as they are executed during the backup process.

I tried a post-thaw script that starts a PowerShell child process which wait for the VeeamVssService to stop and then restart SplunkForwarder service, but the Powershell process dies when Veeam disconnect the service account.

Regards,

[Gostev]

For VMs we offer pre-freeze and post-thaw scripts which are executed before and after VM snapshot creation respectively.

[Origin 2000]

As I understand it, the pre- and post-job scripts run on the VBR server, not on every VM in the job. I may have missed something about it.
The pre-freeze and post-thaw scripts don't fit either, as they are executed during the backup process.

No. Veeam copied the scripts into the VM GuestOS an execute them. If the right return code(not sure about that) is given that VM snapshot will be triggert.

The original pre-freeze and post-thaw feature was introduced with VADP ages ago and where automaticly triggert when you place the script into one of two different locations which were hardcoded by VMware. This changed later where all scripts are executed in the given directory ("C:\Programme\VMware\VMware Tools\backupScripts.d") so you need to implement some logic based on a argument which is automaticly added.

In the good old times (2008/9) we stop a SAP with OracleDB within a VM to get a consistent backup of the System and we start it after the snapshot.

Regards,
Joerg

[Quentin]

Hi Joerg,

I think there has been a misunderstanding. When the backup on a VM ends, the VeeamVssSupport service gets stuck in the Service Control Manager with the 'marked for deletion' flag.
This is caused by the SplunkForwarder service opening a handle on it without closing it before the next backup. I tried to restart the Splunk service using Veeam mechanisms outside the backup window, as during the execution of the pre-freeze and post-thaw scripts, the VeeamVssSupport service isn't 'marked for deletion' yet.

This screenshot might help: https://imgur.com/a/IoEbfFF
The commands were executed while no backup was running on this VM.

Regards,
Quentin