Linux processes on 11.0.1.1261 (Sep 27, 2021)

[dkvello]

I'm testing the 11a - 11.0.1.1261 (Sep 27, 2021) - build and I find that the Veeam services/processes on Linux Repositories dies after 15-30 minutes, requiring a reboot of the Linux host/repository.
I've tested on Ubuntu 20.04/20.10/21.04.

So, backup-jobs and copy-jobs to Linux repositories always fails with thiw build.

To work around this.
Is there a way to monitor and automatically restart the veeamtransport/veeamimmureposvc services ?

[HannesK]

Hello,
That sounds like a technical issue - dying services are not normal Please provide a support case ID for this issue, as requested when you click New Topic.
Without case number, the topic will eventually be
deleted by moderators.

Best regards,
Hannes

PS: support can only help if you upload logs https://www.veeam.com/kb1832

[dkvello]

I know I'm in danger of being deleted

I'm a partner, I don't have any commercial or trial licenses in my labs - only NFR. So, I can't create support-tickets.

I was more interested in if there's some way to set up Service/Process resiliency and self-healing?
I can't see any Veeam services (sudo service --status-all) registered even though there are veeam processes running.

[Mildur]

only NFR. So, I can't create support-tickets.

But you can do that

https://www.veeam.com/licensing-policy.html

2.5 Support
Support for Customers with the Community Edition/Free License/NFR licensed products is provided on a best-effort basis. For more information, please refer to the Veeam Customer Support Policy.

[Gostev]

Yes!

And in fact it would be greatly appreciated in this particular case, because our QC is unable to reproduce the issue in the same configuration as you shared above. So your logs are our only hope at this time @dkvello

The other way to look at this is we lost the whole day of the entire Linux QC team in trying to unsuccessfully reproduce the issue, which might have been immediately clear if only the posting rules were followed and the support case with debug logs was opened right away. See, we don't just randomly have all these "unpopular" posting rules around technical issues for no good reason. Rather, they've crystalized out of soon 15 years of experience...

[dkvello]

I've created a support-ticket and uploaded the first batch of logs.
I'll upload more logs when I get back to work tomorrow and also post logs from my second environment that have similar failures.

I've had some success in setting the Repository workload to max one task (instead of the default 4) and otherwise limiting concurrent I/O.
These are all Hardened repo's, one in IBM Cloud as well as several on-prem.

[Gostev]

Excellent! Could you share the support case ID that was assigned, so I could forward it directly to the devs?

[dkvello]

I've dumped a bunch of new logs from Lab and customer ++

Case #05058021

[HannesK]

Hello,
thank you. I asked QA to have a look on it.

Thanks for your help,
Hannes

[Gostev]

According to QA this looks to be some Ubuntu-specific (or kernel-specific) issue which could also be dependent on the specific hardware you are using (high number of cores).

But together they are reliably causing some race condition that methodically shuts down our process...
The good news this is not some new regression introduced in V11a, as the offending code did not change since 2013.

Is there any chance you try any other distro? Or a different machine?

P.S. We will be fixing this in any case.

[dkvello]

So, one of the Repositories is an Ubuntu VM in IBM Cloud (Two kernels, 16GB RAM). I'll try to "reload" it to Ubuntu 20.10 and see how that affects it.
It's "partner" repository (Where the production backup ends up and where the IBM Cloud Copy get its data) is a physical server (Lenovo x3650M5 Quad-core) with and IBM Flash System backend - Fiberchannel. That one is a bit more difficult to "roll back".

My Lab environment are all Virutal Ubuntu 21.04 (The repositories are 4 Kernel and 32GB RAM) at the moment and I can live with the issues for now

Lenovo Repository HW
CPU: Quad Core Intel Xeon E5-2637 v4 (-MT MCP-) speed/min/max: 1379/1200/3700 MHz
Kernel: 5.11.0-37-generic x86_64 Up: 2d 1h 04m Mem: 2427.4/256926.4 MiB (0.9%)
Storage: 40.11 TiB (14.6% used)

[HannesK]

I suggest to switch to a supported version

from the user guide (assuming that you use XFS)

Ubuntu 18.04 LTS and 20.04 LTS

[dkvello]

Yes, but for the 11a release it says Ubuntu 21.04 as well which is why I tested upgrading from 20.04

https://www.veeam.com/kb4215
RHEL/CentOS 8.4, Ubuntu 21.04, Debian 11, SLES 15 SP3, OpenSUSE Leap 15.3, Fedora 34 distributions are now supported as guest OS, for the installation of Veeam Backup & Replication components, and for agent-based backup with the Veeam Agent for Linux 5.0.1 (included in 11a). For the complete list of supported distributions depending on the component, please refer to the System Requirements section of the Release Notes document.

[Gostev]

XFS repos have always had different requirements (limited scope of OS comparing to other components) and that is not changing. There's the dedicated section for them in the System Requirements.

[fractalenergy]

@Gostev : Ohhh thnx for the clarification, just rebuilded our repository on XFS with Debian 10 two weeks ago, and we were waiting 11a GA for upgrading to Debian 11 as stated as compatible in the KB. So you confirm we should stick with the original Veeam 11 XFS repository OS recommandation even with 11a?

[Gostev]

In fact 11a expands the list of supported OS versions slightly, it should be the following:

For advanced XFS integration, only the following 64-bit Linux distributions are supported:
CentOS 8.2, 8.3 and 8.4
Debian 10.x, 11
RHEL 8.2, 8.3 and 8.4
SLES 15 SP2, 15 SP3
Ubuntu 18.04 LTS, 20.04 LTS

[fractalenergy]

Okay so Debian 11 now supported, great new. Thanks Gostev for the details.

[dkvello]

Thank You for the quick fix. All is good.

[Gostev]

No problems! Glad we were able to find and hotfix the issue.

But it's pretty telling how the code that worked fine on different versions of all sorts of Linux distributions for 8 years had finally ran into the wall on this specific one due to some subtle changes in it. This is exactly why we specifically validate our code against a limited number of distributions/versions.