Veeam.Backup.Manager.exe STARTLOGSEXPORT consuming all available memory

[cip2013]

Hi Everyone,

I could use a little help if anyone has come across this issue. We upgraded 2 of our VBRs from version 11 to version 12 and are now seeing a lot of issues with failed jobs, lost communication, slow console response and lots of errors reported by VeeamOne. The two servers that we upgraded are in two different locations and one of them is only backing up 3 systems. When I run task manager, I see that there is a command running consuming almost all the memory and not releasing it. The command line for the process is "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Manager.exe" "STARTLOGSEXPORT". I am not exporting any logs when this process is running. If I kill this process everything goes back to normal, but every day this process launches and starts consuming all the memory again. I am holding off on upgrading the remaining servers until we get this sorted. I do have a case open and they are looking into it, but I think it might be going in the wrong direction. They are asking me to add more memory to the server (which is on currently on order). Mind you these servers were running fine for a long time with version 11. The only thing "non-standard" about these servers is that we have the Crowdstrike Endpoint agent installed on them, but we do have all the required exclusions added to the policies and I just verified that none were missing.

Thank you.

[Mildur]

Hi Len

We cannot guess solution for such issues effectively over the forum.
Please share the case number with us. With the case number I can check the case for you.

Best,
Fabian

[cip2013]

Hi Fabian,

I understand. I did work with TS last night and we did find the cause of this issue so I will let everyone know what happened in case anyone else runs into this.

So, we found that something was kicking off a log collection job every day at 6:00am. The tech I was working with indicated that this isn't something that is done within VB&R. I mentioned that we do use VeeamOne and maybe that was where it was coming from. Sure enough, there was a agent job scheduled from VeeamOne to collect logs for reporting purposes from the VBR server. That being said, it should not consume all the memory when running the log collection. The log collection process was using 60GB of RAM at one point. We have several VBRs and only the two that were upgraded to Veeam12 are experiencing the issue. I forwarded the logs to support and they are looking into the possible cause for the memory issue. Not sure if it's on the VeeamOne side or the VBR at this point. Strange thing is that if you manually export the logs on the VBR it works fine. For now, I disabled the log collection in VeeamOne and we were able to get all our backups and copy jobs to run properly last night. I would suggest you do the same if you run into this issue as a workaround until we get a final resolution. I will update this post with more info when I get it.

[Mildur]

Hi Len

Thank you for the update.
I have moved this topic to VeeamOne forums.

It may be possible, that your environment is affected by the same issue as discussed here:
post484108.html#p484108

Best,
Fabian

[RomanK]

Hello Len

Veeam ONE VID feature is using PowerShell cmdlet like Get-VBRServer | Export-VBRLogs -FolderPath...
The agent never runs something like "Veeam.Backup.Manager.exe" "STARTLOGSEXPORT" however, this could be initiated by the PowerShell cmdlet itself.

Could you please share the case # in this thread? It would be interesting to know the current state of the troubleshooting.

Thanks

[mzammit]

Had the same issue and opened a case # 07220813

Their solution was the following:

"In order to overcome this issue please create the following registry key on the VBR server :

Path : HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name : ExportLogArchiveMode
Type: REG_DWORD
Value : 3

This value will make Veeam to use the old compression logic for Veeam logs collection.

The issue should be fixed in one of the future patches"

You need to reboot the server for registry entry to take effect.

[Gostev]

I believe this was fixed in 12.1.2... remembering seeing something about this in the release notes.

[stahly]

Hello,

We have the same problem with our VBR server: 100% CPU and 100% memory utilization. (10 Core / 64 GB RAM). Every day from 07:00 to 08:00. Exactly at the time when you want to check the backups...

My questions:
- Is it possible to change the time? At 04:00 at night, this high utilization would not be so dramatic.
- Is the registry entry confirmed?

Unfortunately, I cannot install the update to 12.1.2 because the remote components are not updated during the update in our environment. That's why I had to go back to the version before last.

Thank you!

[RomanK]

Hello stahly,

Yes, it is possible to change the time just follow this guide steps.
In short: Monitor client > Veeam Backup & Replication node > Veeam ONE Agents tab > select agent > schedule action



The registry key is provided by our support to switch the method as the first step of troubleshooting. There are no guarantees that it will help in your particular case. Opening a dedicated support case is always advised.

Thanks

[stahly]

Thank you @RomanK
I will try it!