Console wont Load and backups fail after V12 update (N-able N-central)

[yzfr6]

Updated from V11 to V12 and I have a daily occurrence of this for the past week. The console hangs after entering credentials \MFA and eventually times out. Rebooting or restarting the services "fixes" it but it happens again each day. I have a case open 05887625 but I haven't heard back after it was escalated to V12 support. Is anyone else seeing this behavior?

[Mildur]

Hi Chris

I updated your contract number to your correct case number (05887625).
Have you used the Talk to a Manager button for case escalation?
https://www.veeam.com/kb2320

Best,
Fabian

[DaStivi]

hi,
just double check your timesettings and times between your mfa/authenticator device and the server time is correct, had issues with timesync and that caused issues that the mfa token was not accepted anymore... but basically ended in an error message and wasn't able to logon at all to the console.. didn't hang but maybe it helps you..

[yzfr6]

Its not an authenticator issue, it accepts the auth it just hangs after. No backups are running either when this happens it seems to lock up all of Veeam.

[LickABrick]

With one of our customers we also had some weird issues:
- Jobs would not run (no progress shown whatsoever)
- Basically all kinda tasks like testing AAP credentials would also show no progress at all
- Tons of Veeam.Backup.Sattelite processes in the task manager
- Slow / unresponsive Veeam console

After a reboot / restart of Veeam services either one job or a couple of jobs would be successful but after a little while the issue would return.

The issue (up until now) looks to be fixed after doing a repair of V12 (running installer and using the repair option).

[yzfr6]

That sounds exactly like the issue I'm encountering. I'll try a repair install to see if that helps thanks!!

[yzfr6]

Ran a repair install and still had the issue. For now I've set the server to reboot after each job is run and that seems to allow the backups to actually run but it still failed on our hourly storage snapshots this morning so it seems the issue isnt from running a backup job itself but occurs when the server is idle and then prevents the jobs from running.

[Dima P.]

Hello folks,

Thank you for your feedback! The issue is being investigated, stay tuned.

[LickABrick]

@yzfr6, do you by any chance use N-Able? I just remembered I also disabled the 'Automation Manager' service (part of N-Able) at the same time I did the Veeam repair. The Automation Manager also does some monitoring of Veeam jobs.

[yzfr6]

We do have N-Able and I just disabled the Automation Manager and rebooted. I'll see if that helps

[Alex B]

We are also see the above reported problem.

After a restart everything works fine to start with but it does not take long before everything grinds to a halt and requires a reboot. We do also have N-Able installed

[Dima P.]

Thank you for the heads up folks! Passed this information to our QA team!

[yzfr6]

I can confirm at this point no issues all day since I disabled the Automation Manager service from N-Able.

[Alex B]

I can also confirm that after disabling Automation Management Agent all scheduled backup jobs ran overnight with no failures and the B&R Console remains responsive.

[JeremiahDDS]

I have the same one issue and confirm that it's related to N-Able and the automation manager powershell scripts. I had opened a case with Veeam and N-Able. This is something N-Able will have to resolve as there using a deprecated command in there Veeam monitoring script.

[EddyMandemaker]

Hello,
I think we are in the same boat @ our offices. Case ID "Case # 05894192". Wa also use Nable.
Also I notice event ID 243 'windows heap desktop memory size' error, just 1 before the server is going into 'stuck' mode.
Kind Regards

[rennerstefan]

I have the same one issue and confirm that it's related to N-Able and the automation manager powershell scripts. I had opened a case with Veeam and N-Able. This is something N-Able will have to resolve as there using a deprecated command in there Veeam monitoring script.

Hi, could you please send me both the Veeam as well as the N-Able ticket ID by direct message?
Thanks

[Egor Yakovlev]

We have a confirmation of a problem with co-existence with N-able agent from our QA labs.
Investigating root cause.

/Stay tuned.

[westCPS]

Hi!

We upgrade yesterday our system from v11a to v12 and have the same issue.
Disable n-able Services on this Server (Windows Server 2019 Std. latest Patches) and reboot is a working workaround.
Should we also open a case and send you the logs? Or do you have enough cases with this issue

Best regards
Werner

[Egor Yakovlev]

Hi Werner,

Yes, please do open the case - that helps us estimating the global coverage for a problem.
You can refer to this thread in a ticket as well.

/Thanks!

[LickABrick]

My case number was #05887186, the case was closed with the solution being the repair of the Veeam installation as at the time I thought this was the thing that fixed the issue.

[westCPS]

Hi!

Sure our Case is #05911910 We uninstall n-able to be sure that without all works fine.

best regards
Werner

[bart@itc.be]

Just disable the veeam monitoring function in Ncentral, this solves the issue

[dharmandds]

The problem is the deprecated cmdlet and the way Veeam responds when it is used. It launches a process when you call the cmdlet and then it doesn't end. While it is the N-Central monitoring script that uses the cmdlet, the way the Veeam process hangs seems to be a bit weird. We have had issues every time there has been a major new Veeam release (9 to 10, 10 to 11, and now 11 to 12) because Veeam changes things to add improved functionality or reporting and it takes N-Able a little while to update their monitoring scripts. If you manage Veeam and N-Central, your best bet is to not be an early adopter of new major Veeam releases so that other people can figure out the bugs for you before you upgrade.

[Mildur]

Hello Darrin

Thanks for sharing.
It's always recommended to check compatibility lists and requirements before updating a product to a new version. Not only for N-Able and Veeam. Also for other third party integrations with Veeam.

I believe N-Central only states compatibility with v9 and earlier versions of Veeam:
https://documentation.n-able.com/N-cent ... onitor.htm

Best,
Fabian

[IgnaceD]

Hi,

First time poster but saw similar behaviour at some of the enviroments we manage.
At the moment I'm still running a last test but it looks like this is a combination of Veeam V12 and Veeam One 12 running on the same device in combination with N-Central/N-Able.

So either don't upgrade Veeam One at the moment.
But for those who have already done so:
- Follow the regkey from: https://www.veeam.com/kb1909
- Change the scanning interval within N-Central from every 5 minutes to just every few hours.

I've got 3 enviroments where we have updated to V12 and these are the current situations:
All have N-Central for backup monitoring and have the regkey from above in place.

- One with Veeam V12 and no update of Veeam One: no issues at all
- One with Veeam V12 and Veeam One 12 with no further changes or adjustments: same issue - I've now disabled Veeam One there for testing and so far no more issues
- One with Veeam V12 and Veeam One 12 with regkey and Veeam One 12 running but with changed scanning interval within N-Central: no more issues.

Hope that this can help others.

[Egor Yakovlev]

Thanks for sharing!
Case is still under investigation and every detail helps.

[IgnaceD]

Forgot to mention.
Within N-Central I've also added a check of the Veeam Backup Service and automaticly restart it when it goes into failure.

Main workaround seems to be to first add the regkey and reboot and change the interval period of the Veeam backup within N-Central.
Or don't update Veeam One if you also have that running on the same device as VBR.

[YouGotServered]

We have what could be a similar issue in multiple Veeam environments after V12 upgrade. VeeamOne is not installed, but we do have SolarWinds N-Central on all of our backup servers. Our symptoms are backup consoles that take decades to load and every job / storage operation (like putting a SOBR extent into maintenance, backup jobs, etc) times out after "900ms" as shown in the console. We will probably be yanking the Veeam monitoring service template from N-Central altogether since we don't rely on it for our Veeam monitoring needs anyways. Will report back if it helps or not.

[redboom]

hello!

we also have problems with V12 and are also using nable (latest version)

we are getting timeout errors. sometimes the jobs work and then they fail for some reason...

i will try the regkey. will update tomorrow!

but just to clarify, what are you guys disabled exactly (the automation services?)