[v12] Jobs are timingout to start since upgrade

[keiranshelden]

HI All,

Case #05987817

We upgraded from v11a latest patch last week on 3 VBR servers (All different vSphere environments) and since then, for all 3, after approximately 2.5hrs uptime, jobs start to time out while trying to start (this includes offload and tiering jobs) and the GUI starts to becoming very slow. The server itself is fine and I am able to navigate Explorer, and other sections of the server fine.

The symptoms:
When moving in the console, whether to select a function, category or even edit a job, it takes a long time to load.
Restarting the backup service will timeout and not stop the service.
Rebooting the server gets stuck on stopping the Veeam backup service and can take about 10 minutes.
Closing the console won't happen and is instant from task manager
Opening console (when in this state) can take over half an hour and then timeout with error message advising it's taking longer than usual
SOBR Offload and Archive tiering jobs fail

interesting though, if a job is already in progress, it is not affected. for example, if a job started at 8pm and finishes at the 11pm, The next job might be scheduled for 9pm and it fails to start, but the first job still completes at 11pm successfully. But if I reboot and manually run that 9pm job, it succeeds.

I watch the resources and memory is around 75% utilised during processing and CPU is about 50%.

The environment:
Server 2019 DC
4 vCPU
16Gb RAM
SSD

Thank you.

[karsten123]

What else did you change? Did you migrated to postgresql?

[keiranshelden]

No other changes apart from updating the components along with the Wizard. We did not migrate to PostgreSQL this time.

[karsten123]

strange. did you set av excludes according to kb1999?

[keiranshelden]

We are currently testing tonight for a memory heap issue. We found the event ID 243. Per the Veeam Support Engineer request, we have applied https://www.veeam.com/kb1909

As we have 3 servers doing the same thing, we have applied to 2 of the servers and then we will let the jobs run per schedule, we will then do some further testing in the morning to confirm.

Thank you.

[PetrM]

Hello,

It looks like an issue related to VeeamBackup database performance, my guess is that one of the SQL stored procedures frequently called by one of our services takes longer than expected after upgrade. You can check the most expensive queries but the best action plan would be to let our engineers working on this problem.

Thanks!

[DJakoby]

@keiranshelden,

Check and see if there are multiple instances of the Veeam.Backup.Satelite.exe process. I upgraded two different VBR instances and both are experiencing what you describe. I found that it was because of these processes multiplying numerous times and not closing themselves out. As a workaround, I ended up creating an hourly Scheduled Task using "Taskkill /im veeam.backup.satellite.exe /f", which has temporarily resolved the issue. I actually found this fix from an old forum post, that talks about the same issue back in v9.5: powershell-f26/veeam-satellite-process- ... 36078.html

[Gostev]

In case of V12, the issue is most likely caused by the usage of N-Central.

[DJakoby]

@Gostev,
Is there a known issue with the usage of N-Central?

[Gostev]

Yes, a few customers have posted about this issue previously... in short, N-Central is not compatible with V12 and should not be used until N-Able adds official V12 support on their end, because as it stands right now N-Central actively interferes with normal VBR operation (just as you are seeing).

[DJakoby]

Hi Gostev,
Thank you!
Is this incompatibility documented anywhere in the v12 documentation?
I checked the Release Notes and could not find any reference to it.

[Mildur]

Hi Duane

We don't track, test or document the compatibility of our software with third party monitoring software. It is the monitoring vendor's responsibility to do so.

I recommend you refer to N-Able system requirements:
https://documentation.n-able.com/N-cent ... onitor.htm

Veeam Backup and Replication 6.5, 7, 8, or 9.x installed

They mention that Veeam PowerShell Snap-in (Version 10 or earlier) must be installed. So I assume they also support Veeam Backup and Replication V10. Best you ask N-Able support about that statement.

From our side, we have a long topic about this case where you can find a statement from Veeam.

Best,
Fabian

[keiranshelden]

In case of V12, the issue is most likely caused by the usage of N-Central.

Thanks @Gostev - That's spot on. We got to this conclusion yesterday, although initially thought to be Arctic Wolf agent, but a quick test of that confirmed it was something else and the Veeam Support Engineer advised that they had seen N-Able do similar, in which we also use, so we disabled it and the VBR server is still working well some 13 hrs later.

I have also had N-Able cause issues sometime ago with the Cloud Connect Gateway Service and VSPC.

A team member also came across this thread advising on how to remove the monitoring of the Veeam Service from N-Able. Once we had more information we knew what to look for.

veeam-backup-replication-f2/console-won ... 5942672104

Thanks all for your help.

[bjdboyer]

Does anyone use Logic Monitor to monitor Veeam? We have N-Able but don't use it to monitor the Veeam jobs, but we use Logic Monitor. Anyone know if this same incompatibility exists with Logic Monitor and V12?

Bill Boyer

[IgnaceD]

To be honest. Last thing to do is check N-Able documentation, they don't update their stuff at all.
And their support is kinda useless aswell. Whenever someone over here logs a case with them it ends up at my desk to give them the solution...

I'm using N-Central to monitor Veeam and managed to workaround this.
First off if for some reason an engineer insists on using Veeam One (why use it if you're using N-Central or vice versa), make sure they know not to update to V12 if it's installed on the same server.
Change the regkey and reboot from https://www.veeam.com/kb1909
Change the intervall in which N-Central checks on the monitor service, 15 minutes is already a bit too much of a good thing to only every x hours.

This will allready be good for 98% of the problems in my case.

I've got one server who kept giving issues for that one I created a simple monitor within N-Central that checks the amount of Veeam.Sattelite processes as user Djakoby mentioned in this topic and kills them off if goes beyond a certain number.

Ideal? Certainly not but I'm afraid this will be a case of Veeam blaming N-Central and N-Central blaming Veeam... which doesn't help anybody so better to have a look and make it enjoyable/workable for yourself.

[StevenMeier]

We have this exact issue , but we do not have Ncentral installed on our backup Server, never have.

We have all the same issues as the logger of the forum post.

I have logged a case but still waiting on first response.

Case Number is 06020076.

[StevenMeier]

So I am now unto my 3 or 4 week of this issue.

Case #06020076

Support seem to have no idea and are not making much atempt to solve.
Since I have no Ncentral Agent (N-able) on the box they appear lost)
I supplied a full sql Backup for them....provided job logs.
I stated I was concerned to see very many veeam.sateliite process in taskmanager as well as backup manager process also in task manager.....spawning over 3 days.....then I have to restart the server after 3 days and manually retry and resync jobs and the server is good for 3 days.
I have applied sql timeouts in registry as requested because the jobs were timing out was the error (903 secs).
After many emails with grasping at straws the veeam support said please apply veeam 12 latest patch...which I did...... then we get a new issue. this error message

5/05/2023 11:42:47 AM Warning Failed to enumerate NAS backup metadata files Details: Agent: Failed to process method {MessageChannel.EstablishConnectionByRoles}: Unsupported NAS protocol version. Message version: 12.0.0.151 (201326743), Required version: 12.0.0.150 (201326742)
Now we have Refs volumes on our landing zone backups but we also have hpe 5250 storeOnce devices which do not use ReFS but get the same error.

Luckily I took a snap shot as I will roll back the patch.....
Veeam this is a mess.....why am I having no decent effort put into solving my issue ?
This is a failure on Veeam support and I suggest QA .
We never had issues on Veeam 11......wish I could go back to that now.

[Mildur]

Hi Steven

I will check your case with our support management.

Best,
Fabian