Veeam services not starting first time on v13

[kevin.boddy]

Hi,

Is anyone else seeing this issue where your Veeam services do not start on the first try, particularly the Veeam Backup Service?
If we keep trying to start the services they eventually start after 20-30 minutes.
I'm seeing two .NET runtime errors and then a Veeam service start error in the Windows application log.
There are no Veeam logs generated until the service starts only this Dotxxxx.tmp file present in the Veeam logs folder.

This happens on multiple backup servers with different operating systems and patch levels. AV exclusions in place. The only common denominator is Veeam installed.
Also happens on servers just running a few Veeam services like the Veeam PowerShell Management Service.

Code: Select all

Category: Veeam.Backup.Service
EventId: 0

Failed to wait for Identity service process to be initialized: timed out

Category: Veeam.Backup.Service
EventId: 0

   at Veeam.Backup.ServiceLib.CIdentityServiceProcess.TryToWaitForServiceInitializedEvent(GlobalEvent initializedEvent, IManagedExecProcess process)
   at Veeam.Backup.ServiceLib.CIdentityServiceProcess.StartServiceProcessAndWaitForInitializationEvent()
   at Veeam.Backup.ServiceLib.CIdentityServiceProcess.StartServiceProcessIfIsNotRunning()
   at Veeam.Backup.ServiceLib.CIdentityService.Start()
   at Veeam.Backup.ServiceLib.CVbServiceImpl..ctor(CVbEnvironment env, IServiceShutdownLifetime shutdownLifetime, CancellationToken term)
   at Veeam.Backup.Service.CVbrStartup.OnStart()

Service cannot be started. System.Exception: Failed to wait for Identity service process to be initialized: timed out
   at Veeam.Backup.ServiceLib.CIdentityServiceProcess.TryToWaitForServiceInitializedEvent(GlobalEvent initializedEvent, IManagedExecProcess process)
   at Veeam.Backup.ServiceLib.CIdentityServiceProcess.StartServiceProcessAndWaitForInitializationEvent()
   at Veeam.Backup.ServiceLib.CIdentityServiceProcess.StartServiceProcessIfIsNotRunning()
   at Veeam.Backup.ServiceLib.CIdentityService.Start()
   at Veeam.Backup.ServiceLib.CVbServiceImpl..ctor(CVbEnvironment env, IServiceShutdownLifetime shutdownLifetime, CancellationToken term)
   at Veeam.Backup.Service.CVbrStartup.OnStart()
   at Veeam.Backup.Core.Service.CWindowsServiceLifetime.OnStart(String[] args)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

#07929117

[tinto1970]

yes I noticed it too in my lab. I have to manually restart some services after a reboot.

I am also experiencing a issue with the update to lastest build: the setup does not "see" that an Update is possibile, it only offers the Modify (remove) option

[kevin.boddy]

Apparently this is a known issue the was raised on the community forums with the RTM version of v13.

Veeam engineer suggests running
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control" -Name ServicesPipeTimeout -Value 60000 -Type DWord
to increase the timeout.

Microsoft says this is a workaround and to research the problem.

So far I've only been able to get it to work by increasing the timeout to 5 minutes.

[petesteven]

Do your servers meet the necessary system requirements? In my experience, startup and service problems tend to occur when the server has insufficient resources.

[DaStivi]

You can review disk latency in Task Manager or monitor disk queue length using Performance Monitor (PerfMon). High values may indicate that your system is overloaded.
Additionally, antivirus (AV) scanners can sometimes cause similar issues by interfering with I/O operations.
Another possible reason could be network or firewall restrictions. If your Internet firewall is blocking or delaying Certificate Revocation List (CRL) checks, these operations may take significantly longer than expected.

[Gustav]

Do your servers meet the necessary system requirements?

A valid question.
I guess that quite a few are not aware of the requirement of V13 for 8 cores minimum, making an upgrade from V12 non-trivial in many scenarios.

[david.matthewson]

I too found this and my current workaround is to change all the Veeam services to 'Delayed Start' on the Windows server to allow the box to start other NT Services first and then, when the flurry is over, start Veeam.

You can alter the default 'delay' with this reg key.

Create the following registry key via the Windows Registry Editor.

HKLM\SYSTEM\CurrentControlSet\Control\AutoStartDelay

It is of type REG_DWORD. Set the value of the key to the desired delay in milliseconds (e.g. 60000 for one minute). You will need to reboot for the change to take effect. Remember this change affects all Automatic (Delayed Start) services.

Perhaps not ideal but it works!

[aschaupp]

I had to do both.
Increase assigned memory to the virtual machine and delay service starts - it is not yet perfect but without registry it sits for around 3-5 minutes before you can access the console due to not all services being up

[kevin.boddy]

Definitely not a resource issue. 24x cores and 160GB of memory. No disk IO issues. CPU is hardly used when these services try to start. This looks more like a Veeam+.NET issue to me.
Happens on all our servers. AV and EDR exclusions are in place.

You can see this DotNetZIP-xxxxx.tmp file is created in the log folder when the service is trying to start and this grows to about +2.5GB and once the service starts this file disappears and normal Veeam logging commences.
If you watch task manager for the service it's memory usage grows very slowly until it finally starts.