Support for RHEL 8.5 [Kernel 4.18.0-348]

[lehaneye]

Greetings,
We have recently started a round of patching our non-prod systems running on RHEL 8 to close vulnerabilities. In the process we updated our content views to the latest support levels and as a result, those systems running RHEL 8.4 were updated to 8.5. This update has brought with it a new kernel for RHEL 8 and we no longer fall inside the supported 4.18.0-305 as the kernel now sits at 4.18.0-348. Wondering when support, including experimental, will be available. Before agent 5.0.1 was released, we had to use 5.0.0.4325 to support our RHEL 8.4 releases so we've been down this road. Looking to see if there is any chatter about further RHEL 8.5 support. Case was open with support to discuss this issue but no definitive answer as of yet, recommend I post something here(#05139930).
Thanks for your time!

[Gostev]

Hello! Just to confirm, do you have any issues with the product right now? Thanks.

[lehaneye]

Hello, thanks for the reply, and yes, any system that has been updated from RHEL 8.4 to 8.5 is running the latest kernel and my attempt to resolve the issue was to first see if migrating to the Agent for Linux 5.0.1 but it did not.

We've only got a handful of systems running RHEL 8.5 and those were previously running Agent 5.0.0.4325 to support the RHEL 8.4. The 8.4 systems get the message

Code: Select all

Failed to execute IOCTL_TRACKING_ADD

but the one system that I upgraded to Agent 5.0.1 is a bit more detailed and first reports

Code: Select all

Module loading operation has returned code [255]

and then

Code: Select all

Failed to load module [veeamsnap] with parameters [zerosnapdata=1 debuglogging=0 snapstore_block_size_pow=14 change_tracking_block_size_pow=18 logdir=/var/log/veeam fixflags=0 logmaxsize=15728640]

So anything running the latest kernel is unable to complete the backup snapshot. Certainly let me know if you're looking for something else more detailed.

[Gostev]

OK. From these errors it sounds like there were some significant breaking changes in this kernel version, so I would not hope for a quick fix here. But let's wait and see what devs find out.

[Gostev]

Good news! Our devs have already prototyped a fix and if all goes well, we should be able to release support for RHEL 8.5 [Kernel 4.18.0-348] as a part of the next cumulative patch for 11a. But these changes are in the very core of the agent, so they trigger a full retest including a long-term stability testing cycle... which will require at least a few weeks to complete.

[lehaneye]

Excellent news! Appreciate the update, I look forward to getting this fix on our systems. Thank you!

[aj_potc]

Good news! Our devs have already prototyped a fix...

Do all systems running kernel 4.18.0-348 require this fix?

The reason I ask is because I've also moved one of my systems to this kernel version, and I was really happy that I didn't see any breakage with Veeam. Everything has been working exactly as it was before the upgrade.

[Gostev]

No. If you use snapshot-less backup mode, then it should not be needed.

[PTide]

Do all systems running kernel 4.18.0-348 require this fix?

Not that I am aware of. We haven't seen any problems with Ubuntu 18.10 (it used 4.18 too, but is EOL already). I also don't recall any other distros using 4.18 (besides RHEL-based).

Anyway, I'd say that any RHEL-based distro using 4.18 will need that fix (until our QA department proves it otherwise in their tests).

Thanks!

[aj_potc]

Interesting. I'm doing snapshot-based backups with CentOS on this kernel version and see no errors at all.

[PTide]

Hmm...could you check whether the trailing part is the same for CentOS and RHEL systems that you have?

Is it 4.18.0-348.2.1.el8_5?

Thanks!

[aj_potc]

Indeed, it appears it's not the same.

I'm running:
348.el8.0.2

I didn't realize that another one was released since 8.5 came out.

I'll be sticking with 348.el8.0.2 for now.

Thank you very much for the clarification!

[KernelKurtz]

Just for info here, I'm running CentOS 4.18.0-348.2.1.el8_5.x86_64 and Veeam snapshots are still working fine w/veeam and kmod-veeamsnap 5.0.1.4493

[no-reply]

Also you can switch to the RHEL EUS repos to stay protected and supported within the release train.

[aj_potc]

Just for info here, I'm running CentOS 4.18.0-348.2.1.el8_5.x86_64 and Veeam snapshots are still working fine w/veeam and kmod-veeamsnap 5.0.1.4493

I've also bumped one of my systems up to kernel 4.18.0-348.2.1.el8_5.x86_64, and Veeam snapshots are still running just fine here. Like yours, my agent version is 5.0.1.4493.

So it would appear that not all systems are affected. I wonder what the difference is...

[PTide]

@aj_potc, @KernelKurtz,

CentOS does not always blindly apply all of RedHat updates (even after RedHat has purchased CentOS as a whole). Sometimes CentOS doesn't get the same micro updates that RedHat gets. If you are after the technical details, then we need to wait for our developers to figure it out.

Thanks!

[KernelKurtz]

Thanks PTide. No worries. Everything still works so I'm certainly not complaining.

Cheers!

[aj_potc]

This may not be groundbreaking info, but I converted some systems over to Rocky Linux 8.5 this past weekend, and Veeam is still running perfectly on them.

The kernel version on these systems is 4.18.0-348.2.1.el8_5.x86_64.

[lehaneye]

Greetings,
With the recent release of Veeam Agent for Linux 5.0.2.4567, I just wanted to give a final update with my scenario running RHEL 8.5 with Kernel 4.18.0-348. I've been running the updated agent for about a week now and the issues I was experiencing with Agent 5.0.1.4493 have been corrected by the new release and our backups are completing successfully. Appreciate all the comments and suggestions provided. Let me know if there is any follow up needed. Thanks!

[sfaits]

lehaneye,
What minor point release of the 4.18.0-348 kernel are you using? Just curious.
Our agent backup broke when we updated to 4.18.0-348.7.1. (linux agent 5.0.2.4567).

Strangely I do have a rhel 8.5 running with 4.18.0-348.7.1. and Linux agent 5.0.2.4567, where the backups actually snap successfully and complete...still trying to figure the differences between the two servers

Thanks!

[PTide]

@sfaits,

Did I get it right - you have two identical RHEL 8.5 machines, both running 4.18.0-348.7.1 and 5.0.2.4567. One machine runs backups just fine, while the other does not.

Is that correct?

Thanks!

[sfaits]

They are identical only in OS and patch level, One has many applications and the other is not running much, mostly a test server.
I am most likely missing something obvious between the two....

[sfaits]

Figured out the issue, User error.
When the agent was installed we (mostly likely I) used the wrong veeam repo package and the the el7 version of the kmodsnap got installed. This worked fine until the kernel update.
Once I removed the agent and installed the correct rhel8 kmodsnap everything started working again.

[PTide]

That's some uplifting news, thank you for getting back to us!

[mark49808]

I'm seeing this problem with a fresh install/update of 8.5.

Install 8.5 from latest ISO
yum update
Install latest veeam components including ueficert

kmod-veeamsnap.x86_64 5.0.2.4567-1.el8 @veeam
veeam.x86_64 5.0.2.4567-1.el8 @veeam
veeam-release-el8.x86_64 1.0.8-1 @System
veeamsnap-ueficert.noarch 5.0.2.4567-1 @System


4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

Code: Select all

[23.02.2022 13:52:05.170] <139896030377728> prtcl  | >>  |Failed to load module [veeamsnap] with parameters [zerosnapdata=1 debuglogging=0 snapstore_block_size_pow=14 change_tracking_block_size_pow=18 logdir=/var/log/veeam fixflags=0 logmaxsize=15728640].

Code: Select all

modprobe veeamsnap
modprobe: FATAL: Module veeamsnap not found in directory /lib/modules/4.18.0-348.12.2.el8_5.x86_64

Tried install/reinstall/reboot. Nothing seems to work.

Edit: found this KB: https://www.veeam.com/kb2804

Seems downgrading to 4.18.0-348.2.1.el8_5 made things work.