Veeam Satellite Process is not cleaned up

[tpayne]

I apologize, that was the wrong case number, here is the correct one:

2724946

Thank you.

[Vitaliy S.]

Just had a discussion with our QA team on this problem. The hotfix which is available through our support team addresses the majority of situations when this issue is observed. If you still have the same problem after applying the fix, then a new ticket has to be created. This is needed to review latest logs to find an acceptable solution to the problem.

[StoneFritz]

ID# 02729989

[arun.kr]

Hello,

I have faced satellite process related issue twice.

First time - scheduled backup jobs started to fail. We had logged the case (Case # 03051798 ) with VEEAM. support suggested rebooting the server. One thing that we noticed there were lots of satellite services were active.

Second time - while restoring the VM. we got error satellite service not started. this time we didn't open case. simply rebooted the server and error gone away. but within 4 hours i can see over 50 satellite processes running.

Should i open a case and ask for fix, you are talking about in the thread.

Thanks
Arun

[Vitaliy S.]

Yes, please do and let me know your case ID. BTW, rebooting the backup server should not be excepted as a solution.

[Talom]

I discovered the bug 2 years ago.
Why has the hotfix not yet been added to Veeam and still needs to be requested from support?
I'm just curious.

[veremin]

It's already incorporated in Update 3a. So, there is no need to request it separately any longer. Thanks.

[MariusN]

It's already incorporated in Update 3a. So, there is no need to request it separately any longer. Thanks.

I am not able to confirm this - we have the same probem with 3a -> satelitte processes not cleaning up.
We use simple powershell scripts to send stats to grafana for visualizing metrics. This scripts are creating the veeam satelitte processes which won't be closed.

So if you reboot the System everything is fine, but after a certain amount of time the backups and even copy Jobs will fail with the following messages:

Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

The following is used:
https://github.com/jorgedlcruz/veeam_gr ... -stats.ps1

Only solution is to disable the Scheduler for running this scripts or reboot the Server after 1 day for the moment.

[Vitaliy S.]

Marius, there could be various reasons for the satellite processes not being cleaned up. Can you contact our technical team and let them review the reason for this behavior?

[edv-compas]

since a few days, we have the same problem on an installation.

[Vitaliy S.]

Peter, please contact our technical support team for further investigation.

[edv-compas]

opened an case: 03118470

[edv-compas]

support told that this an known issue and will resolved with update 4
there are 2 workarounds:
https://www.veeam.com/kb1909
or
make an scheduled task with: taskkill /f /im Veeam.Backup.Satellite.exe

[masonit]

Couple of days ago we upgraded to 9.5 3a and today I was unable to login to VBR with console. It just failed with "veeam failed to start satellite process". Found this thread and we had 96 (!) Veeam.Backup.Satellite processes running. Stopped them and after that was able to login. Looks like backups were unaffected. Have never had this problem before. Started with 3a. Every hour we collect stats with powershell. Could this be a part of the problem?

\Masonit

[MariusN]

What I can say for the Moment I think its a bug mentioned already in this thread..
there are 2 workarounds:
https://www.veeam.com/kb1909
or
make an scheduled task with: taskkill /f /im Veeam.Backup.Satellite.exe

I have the same side-effect running statscollection with powershell as posted above. As soon as you disable the powershell scripts you won't run into this issue anymore.

[doum]

What I can say for the Moment I think its a bug mentioned already in this thread..
there are 2 workarounds:
https://www.veeam.com/kb1909
or
make an scheduled task with: taskkill /f /im Veeam.Backup.Satellite.exe

I have the same side-effect running statscollection with powershell as posted above. As soon as you disable the powershell scripts you won't run into this issue anymore.

I had the problem too this night.

but if you run the taskkill, when do you run it ?
isn't there a risk of killing a task in use by a backup?

[veremin]

Before scheduling any activity it wouldn't harm to reach our support team; this should help to isolate the issue to specific root cause and get access to the solution needed. Thanks.

[mriksman]

I'm having this issue also. Has a hotfix been released?

It started when I started using Powershell scripts yesterday. My script uses this;

Code: Select all

      $connectVeeam = Connect-VBRServer
      Get-VBRBackupSession | Export-Clixml "$pathxml\backupsessiontemp.xml"
      Get-VBRJob | Export-Clixml "$pathxml\backupjobtemp.xml"
      Get-VBRBackup | Export-Clixml "$pathxml\backupbackuptemp.xml"
      Get-VBREPJob | Export-Clixml "$pathxml\backupendpointtemp.xml"
      $disconnectVeeam = Disconnect-VBRServer

And then everything else simply looks at the generated XML files.

The excess Veeam.Backup.Satellite process killed my backup server last night!

[mriksman]

My issue seems to be because two different scripts run at the same time - each performing a Connect-VBRServer.

When I disable either one, then it runs fine.

[Regnor]

I've noticed that if I run the Guest Credentials Test many times for a huge number of VMs in a Veeam Deployment, that after some time the Veeam Console shortly disconnects and a new Satellite process is started. After that regularly a new process appears but only for the user session, which ran the tests. After closing the Backup Console and killing all processes the problem doesn't reoccur.

[Vitaliy S.]

Anyone who is experiencing this problem, please contact our support team to investigate the problem and receive a hotfix (if available for the particular case). Thanks!