Rocky Linux now GA. Veeam support ??? (also AlmaLinux)

[VAXman65]

I built up a Rocky Linux 8.4 system today using the GA code base that is now available. That worked just fine and I was able to install Veeam and get it started. However, once i defined the backup job, it fell over on execution. (Disappointing but not entirely surprising).

Is there a timeframe for Rocky Linux support in Veam?

I also filed Veeam Support - Case # 04874548

FWIW, here is the first sign of trouble in the Job log. I imagine the Veeam code needs a liberal sprinkling with “ || rocky” through the codebase.

[21.06.2021 19:15:24.340] <140649266202368> lpbdeve| ERR |Failed to create DeviceInfoEx for device [/dev/sr0].
[21.06.2021 19:15:24.340] <140649266202368> lpbdeve| >> |No medium found
[21.06.2021 19:15:24.340] <140649266202368> lpbdeve| >> |Failed to probe device [/dev/sr0].
[21.06.2021 19:15:24.340] <140649266202368> lpbdeve| >> |--tr:CRealBlkidAccessor: Failed to probe device [/dev/sr0].
[21.06.2021 19:15:24.340] <140649266202368> lpbdeve| >> |--tr:in virtual lpbdevenumlib::blkid::IBlkidProbe::Ptr lpbdevenumlib::accessor::CRealBlkidAccessor::ProbeDevice(const astr&) const at /mnt/Sources/LpbDevEnumLib/lpbdevenumlib/accessor/impl/RealBlkidAccessor.cpp:38

[Regnor]

Here's a quote from @gostev regarding Rocky Linux:

We will certainly consider adding support in future depending on its actual adoption levels and market share.

So if a certain percentage of Veeam customers is requesting support for Rocky Linux then it will be evaluated.

[aj_potc]

Sadly, Veeam doesn't support RHEL or CentOS 8.4 yet, so this may be the primary reason that it doesn't work on Rocky 8.4.

[Gostev]

I would say, the primary reason is that we're trying to limit the number of distributions our QC needs to test the product against only to the most widely adopted ones. If Rocky will see good adoption, then we will add support for it - likely along with removing support for some other distribution to free up the QC resources for Rocky.

For RHEL 8.4 support specifically, we worked directly with Red Hat and they added some special code in their custom kernel to ensure Veeam will be able to support RHEL 8.4. I don't know if derivative distributions will automatically have the same mod, but without one we won't be able to support anything based on 5.8 or later kernel versions because of the kernel feature loss that previously enabled I/O interception essential for our snapshot and CBT drivers (see the sticky topic). Which means only file-level backup without snapshots would be possible.

[VAXman65]

I was able to confirm that the problem I was having was not with Rocky Linux, but rather with the newer kernel. In this case, the fault was bug-for-bug compatible with CentOS. The same fix worked in both places.

In both CentOS and Rocky Linux, 8.4 went out with kernel 4.18.0-305.3.1. This was the first kernel version of RedHat derivatives that broke volume snapshots with the incorporation of the 5.8 kernel enhancements. For both CentOS and Rocky Linux, I was able to leave the systems overall at the 8.4 version but downgrade the kernel back to kernel-4.18.0-240.22.1. This mixture was my preferred approach for Rocky Linux since it only went GA with 8.4. The alternative is to downgrade across the board to 8.3, which also worked.

So my experience has shown that CentOS 8.4 falls into this same problem and the kernel version that comes with it (at least today) does not have the necessary fixes to work with Veeam. Downgrade of at least the kernel restores functionality. Rocky Linux has proven to be bug for bug compatible and exhibits exactly the same problem and exactly the same solution.

Overall, this has been a positive experience for me, appearing to prove out how well Veeam with work with Rocky on the basis of this edge case where it actually broke!

Longer term, this does point to a CentOS issue and that it may be necessary for Veeam to work with the upstream maintainers to ensure it not only gets into the RedHat distro, but also into the codebase that the RedHat specific version is built from. If this is done, then it would flow automatically into the downstream releases. It may be the case that the fix code is being rolled into only the RedHat commercial version

[PTide]

Hi,

Depending on what your current kernel version is, I would suggest to take a look at this post.
As for pushing it upstream - yes, we have been working with the kernel maintainers.

Thanks!

[Gostev]

Unfortunately, kernel maintainers don't want anything in the kernel that is not consumed by the kernel itself, which makes it extremely challenging to add functionality like I/O interception capabilities to be used by external applications like Veeam or Oracle.

[VAXman65]

With CentOS 8 dying at the end of the year, there will be little need/impetus to address this issue for that distro. CentOS Steam is another matter, but it being upstream of RedHat would seem to offer no solution if the point of entry for the code fix is with RedHat.

Rocky Linux incorporating these changes remains downstream from RedHat on an ongoing basis. In theory, code made part of the mainline build feeding into RedHat commercial release should appear in Rocky. However, if RedHat chose to incorporate the fixes only in their commercial code, it would be missed by Rocky Linux, since it would not be part of the code base passed down. This would be an awfully weird way of doing things - isn't the RedHat-only code their (limited) protected IP? I would not think that the fixes Veeam has been pushing and providing would qualify.

The real question would be, what kernel version is in RedHat 8.4 and does it fail today? If it is working, then are the changes in the codebase passed downstream or are they trapped in the RedHat proprietary code?

[Gostev]

RedHat 8.4 is distributed with good old 4.18 kernel, but also with some stuff from 5.8 kernel backported into it, including the breaking change unfortunately. It's a good question though where those backported changes sit.

[VAXman65]

Is the plan here to hope for modification to the 5.8 kernel and hope that RedHat chooses to backport the fix to the breaking change later on?

Or is the approach to provide Redhat the code directly to fix the breaking change so that it would be included with the backported elements they include in 4.18?

[Gostev]

It's the latter and it's already done. We identified the issue long ago based based on the pre-release 8.4 builds testing, and worked with Red Hat to implement a workaround for their custom kernel. We still need to modify our code slightly to work with that though, so the currently shipping Veeam Agent version won't work with 8.4 yet.

[FromOZ]

I would say, the primary reason is that we're trying to limit the number of distributions our QC needs to test the product against only to the most widely adopted ones. If Rocky will see good adoption, then we will add support for it - likely along with removing support for some other distribution to free up the QC resources for Rocky.

The key point here is that with respect to Rocky Linux we are not talking about yet another/new Linux distribution to have to code for, and assign QA resources to. We are talking about RHEL, because "Rocky Linux is a community enterprise operating system designed to be 100% bug-for-bug compatible with America's top enterprise Linux distribution" (see https://rockylinux.org home page with FAQ on this topic).

Rocky Linux is, as CentOS was before, a downstream build of RHEL. The project started because RedHat shifted CentOS from being downstream to upstream. Of course they already had an upstream, Fedora.

The net result/takeaway is:

1. CentOS is, essentially, dead.
2. Rocky Linux will be the platform CentOS likely most users will be changing to. It was started by Gregory Kurtzer, founder of the CentOS project, he obviously has good credentials.

[Gostev]

If most of the former CentOS users migrate to Rocky Linux, then of course we will have no choice but to test and officially support our products on it. As CentOS had a decent market share before all the recent changes.

[aj_potc]

There already are a couple of replacements for CentOS. Even if Rocky does not end up being the most popular, I think the point that @FromOZ is making is that Rocky isn't a new distribution that you have to support, but rather is intended to be identical to what CentOS was: a downstream build of RHEL.

In other words, if things work as intended, you won't have to make a decision about testing and supporting Rocky. If you support RHEL, then support for Rocky is a given.

[PTide]

Hi,

Well, despite of whatever distro maintainers claim about compatibily, we still test all distros separately, even CentOS and RHEL. We don't announce anything to be supported just because the manufacturer says that it's 100% compatible, because sometimes we catch odd bugs on seemingly identical distros.

Thanks!

[arundas]

CentOS 8 is going out of support soon. AlmaLinux and RockyLinux has already stepped in and released their versions already. But it seems like the latest Linux kernel 4.18.0-305 have issues with most of backup software like Acronis, R1Soft and Veeam.
https://imgur.com/a/8he1GQZ

Acronis (https://kb.acronis.com/content/68870) and R1Soft are providing their customers with workaround for the issue. Will Veeam be providing support for these OS/Kernel ?

[HannesK]

Hello,
and welcome to the forums.

Please see the answers around the same topics above. If you are interested in more details, the sticky topic about kernel 5.8 and later documents the progress on these kernels veeam-agents-for-linux-mac-aix-solaris- ... 70158.html

Best regards,
Hannes

[VAXman65]

Today, I received the kmod-veeamsnap.x86_64 5.0.1.4493-1.el8 on both My CentOS 8.4 and Rocky Linux 8.4 systems. I had been using Veeam, though with my OS kernel trapped at 4.18.0-240.22.1.el8. So I re-enabled kernel updates, got 4.18.0-305.19.1.el8_4 and rebooted.

Running the Veeam backup worked perfectly! CentOS and Rocky Linux both worked exactly the same and both ran the backup to conclusion without complaint. Here is the job summary information my automation retuned for the Rocky Linux run:

Code: Select all

Backup session
   ID: {fba57ddf-7338-4329-aa5d-8f2012f5439c}
   Job name: kristall-all
   Job ID: {93a61224-a702-4624-90e1-00b2da035bab}
   State: Success
   Start time: 2021-10-08 12:21:42
   End time: 2021-10-08 12:22:10

2021-10-08 16:21:42 UTC [info] Job kristall-all started at 2021-10-08 16:21:42 UTC
2021-10-08 16:21:43 UTC [info] Preparing to backup
2021-10-08 16:21:45 UTC [info] Creating volume snapshot
2021-10-08 16:21:46 UTC [info] Starting incremental backup to Repository_1
2021-10-08 16:21:47 UTC [info] Backed up V0uv033 5 GB at 899.3 MB/s
2021-10-08 16:21:52 UTC [info] Backed up sdb 1 MB at 5.3 MB/s
2021-10-08 16:21:53 UTC [info] Backed up V3uv033 480 MB at 179.8 MB/s
2021-10-08 16:21:55 UTC [info] Backing up BIOS bootloader on /dev/sda
2021-10-08 16:21:56 UTC [info] Backed up sda 1 GB at 566.6 MB/s
2021-10-08 16:21:58 UTC [info] Backing up summary.xml
2021-10-08 16:22:00 UTC [info] Releasing snapshot
2021-10-08 16:22:05 UTC [info] Applying retention policy
2021-10-08 16:22:05 UTC [info] Full backup file merge completed successfully
2021-10-08 16:22:08 UTC [info] Executing post-job script
2021-10-08 16:22:10 UTC [info] Post-job script has been executed successfully
2021-10-08 16:22:10 UTC [info] Processing finished at 2021-10-08 16:22:10 UTC

[PTide]

Hi,

What do you use Rocky for? Desktop/Laptop/Server?

Thanks!

[VAXman65]

This was my prototype for migration from CentOS to Rocky. These are server instances running as KVM server VMs. In this case, I am moving my Samba AD domain controllers to the platform.

I have a number of server VMs running CentOS. Now that I have proven out Rocky and have a functioning backup solution, I’ll be migrating a number of CentOS 7 systems to Rocky 8.4

[doublem]

If most of the former CentOS users migrate to Rocky Linux, then of course we will have no choice but to test and officially support our products on it. As CentOS had a decent market share before all the recent changes.

@Gostev

Thinking the other way around. If Veeam supports Rocky Linux, users may be adopting Rocky Linux at a larger scale.

[Robert G.]

We would like to request that AlmaLinux be added to the list of supported operating systems. For the following components:

• Veeam Backup & Replication
  • Backup Proxy Server
  • Backup Repository Server
  • Gateway Server
  • Veeam Agent Computer (Linux)

• Veeam Agent
  • Veeam Agent for Linux

There are already a few threads where this topic has been discussed:

• veeam-agents-for-linux-mac-aix-solaris- ... 75071.html
• https://www.reddit.com/r/Veeam/comments ... for_veeam/

Like MySQL/MariaDB or ownCloud/Nextcloud, I don't think we will see a clear winner of this "successor battle" in the next few years.

• RockyLinux seems to be the winner of the community heart
• AlmaLinux scores with update speed, features, years of experience and professionalism in general

[HannesK]

Hello,
I merged your request to the existing discussion. We count your request +1

Best regards,
Hannes

[saahi]

Our linux guys would love to have support for Almalinux.

[aj_potc]

Unfortunately, CentOS users have got a real chicken-and-egg problem here.

Those of us running CentOS have to migrate to another distro before it goes out of support at the end of this year (just a few short weeks away).

But Veeam wants to wait until they know which OS has the most users before they announce support.

This means that we could switch our production servers to an OS that Veeam decides not to support in the future. I'd really not like to be in that situation.

It would be great if Veeam could give us some indication here, but I know that may not be a realistic expectation.

[PTide]

@aj_potc,

Which distro are you inclining towards? What about RHEL (allows for up to 16 machines in production for free).

Thanks!

[aj_potc]

Probably Rocky Linux, as I've successfully converted a few systems already. But I wouldn't be against Alma; they sound like a professionally run operation that I could trust.

I really don't want to deal with the RHEL subscription model in order to keep systems updated, and I'm pretty close to the 16-system limit already. So this doesn't sound like a good path for us.

Thank you for your interest!

[villapla]

Hi,

We are currently migrating our CentOS 8 servers to AlmaLinux, so we would like to see AlmaLinux as a supported OS for our Veeam infrastructure (Proxy Servers and Repository Servers ---including XFS reflink support---).

We have delayed the migration of our Proxy and Repository servers, but it would be nice to know the Veeam support road-map for RHEL 8 clones as CentOS 8 is about to reach its EOL.

Thanks in advance!

[soehl]

but it would be nice to know the Veeam support road-map for RHEL 8 clones as CentOS 8 is about to reach its EOL

+1

[danswartz]

+1 for me too. I have a small almalinux install that serves up a heartbeat datastore for Vsphere 6.7 in my home lab. I'd love to back it up, but... NOPE!