linux agent issues with Kernel 4.18.0-425

[sfaits]

Just an FYI for users running RHEL 8 and looking at upgrading to 8.7 and/or kernel version 4.18.0-425.

In my test instances after upgrading to RHEL 8.7 my veeam agent backup starts then hangs at the Creating volume snapshot stage.
The server then killed all of the remote connections and became completely unresponsive to local commands or new remote sessions. My only option was to kill the server and power it back up.

I opened a support ticket (# 05723966) to confirm this was an issue with the new kernel, which they did (and very quickly to).
No ETA on when 4.18.0-425 will be supported by the Linux Agent at the moment.

Just wanted to share what I have learned with others, hope it helps.
Thanks!

[HannesK]

Hello,
thanks for sharing. Yes, RHEL 8.7 is unsupported for now.

Our goal is to support new operating systems within 90 days after GA.

Best regards,
Hannes

[sfaits]

Thank you for that info Hannes.

[ITP-Stan]

Thanks for this info.
I'm affected by this.

[ITP-Stan]

@HannesK

I'm a bit surprised that a regular yum update is considered a new operating system?
Because 8.7 is just a regular update and a newer kernel version, but still not the 5.x branch.

[HannesK]

Hello,
yes, 8.5 is considered different than 8.6 and 8.7 is considered different than 8.6. The kernel version must match a supported kernel.

To get around that, one can use the nosnap package and do file-based backup with optional LVM snapshot or leverage NAS backup (which has license implications, but no kernel dependencies). Even with nosnap, it's still unsupported, but it should work.

Best regards,
Hannes

[ITP-Stan]

Thanks for the information.

[asgard79]

Thanks.
Unfortunately, I've done update the same day where kernel was available...

[sfaits]

Asgard79
If you can suffer the downtime with restarts you could try reverting the kernel to 4.18.0-372.
Kernel reversions have worked for us in the past.
Best of luck.

[dhnws]

Is there a place on the veeam website that I can check for the newest RHEL 8 rpms or maybe some other way to be informed when a new rpm is released?

Thanks

[Mildur]

Hi dhnws

We have kb article on the latest Veeam Agent for Linux support.
Please note a new version can be released as a private build. We will release them to public with new versions of Veeam Backup & Replication. https://www.veeam.com/kb2804

Thanks
Fabian

[ITP-Stan]

I might try to upgrade to 9.0 since that release is supported by Agent for Linux 5.0.2

[dhnws]

Hi dhnws

We have kb article on the latest Veeam Agent for Linux support.
Please note a new version can be released as a private build. We will release them to public with new versions of Veeam Backup & Replication. https://www.veeam.com/kb2804

Thanks
Fabian

Thanks for the info. I will keep an eye on that KB article.

[Mildur]

Private builds of Veeam Agent for Linux require a licensed agent.
Please consider that before doing any upgrade of your RHEL server. The kb article lists which agents are available per support (private builds) and agents available on our download pages (public release).

Best,
Fabian

[ITP-Stan]

Ah, I missed that. The build number is slightly different.

[ITP-Stan]

So I tested with Agent for Linux 6.
It could back-up RHEL/Rocky 8.7 now.
I then upgraded to RHEL/Rocky 9 as I was planning to do for a while.
It immediately upgraded to RHEL/Rocky 9.1 and I was out of luck again, the kernel module was not compatible with 9.1 only 9.0.

So I bit the bullet and reinstalled with LVM partitioning so I could use the nosnap version.
In the user guides it was said that there needs to be free space on the physical extent to do back-ups but it did not mention how much.
I thought 2GB free space for a +/- 38GB volume with low changes would be enough.
However I was wrong, Veeam gave an error about not enough free space.
And since I was using XFS as a filesystem, that can't shrink, I had to start over again and doubled it, which worked.

So kind suggestion to mention the needed free space somewhere in the user guide for nosnap LVM.

[PTide]

Hello,

The amount of space required for LVM snapshots very much depends on how much IO load you have on your system. YMMV, as they say it : )
Advising on how much space one might actually need for LVM snapshot storage would require us to have collected a pretty extensive statistics on snapshot overflows over the past years.
I remember seeing different storage vendors saying number between 10% and 20% for systems that do not experience too much write load. ~4GBs that you set matches the lower boundary.

Thanks!

[ITP-Stan]

Even this suggestion is fine as it gives an indication.

[PTide]

Ok, will add that to the docs.

Thanks!

[aj_potc]

So I tested with Agent for Linux 6.
It could back-up RHEL/Rocky 8.7 now.
I then upgraded to RHEL/Rocky 9 as I was planning to do for a while.
It immediately upgraded to RHEL/Rocky 9.1 and I was out of luck again, the kernel module was not compatible with 9.1 only 9.0.

Sometimes this can be alleviated by booting from an earlier kernel version. So, you allow all packages to be updated to the latest RHEL/Rocky release, but run the system on a slightly earlier kernel version (from 9.0) that Veeam supports.

In my opinion, at least for public-facing servers, it's smart to make sure updates are applied as soon as possible to any exposed services. Waiting several months for Veeam to support a later kernel is not always possible. So, rather than holding back on updating the entire system for the sake of Veeam compatibility, sometimes it's a workable alternative to hold back on just the kernel.

[ITP-Stan]

I agree, but in the upgrade it went straight to 9.1 and there was no option to boot older kernel.
And apparently with Rocky it's no longer possible to downgrade kernels.

In my opinion the nosnap package with LVM is the perfect solution!