[LINUX] Kernel 5.16 and later, veeamsnap support

[marcwa19197]

Hello,

i know this is an "big topic" in this forum, i read much threads about it.
I just want to ask for a status for the Kernel 5.8 compatibility. As you might know, Fedora 33 was released a few days ago. Fedora 33 is using Kernel 5.8 as default.

Is there any timeline? If not, i will wait for official compatibility with Fedora 33.
At the moment im using Fedora 32 with an older Kernel, because Kernel 5.8 is not working at the moment.

I read something about an newsletter, how can i subscribe to it?

Thank you very much.

[PTide]

Hi,

Is there any timeline? If not, i will wait for official compatibility with Fedora 33.

Unfortunately there is no timeline for now.

I read something about an newsletter, how can i subscribe to it?

I guess you mean "Word from Gostev"? Just click on a "wrench" icon under the topic name and select "Subscribe"

Thanks!

[marcwa19197]

Thank you for the information!
Im getting
"You are not authorised to read this forum."
if i follow your link..

[PTide]

My bad. Please try "User Control Panel > "Board Preferences" tab > "Weekly Digest Subscription" section > "Subscribe to weekly digest" radio button.

Thanks!

[marcwa19197]

Thanks, but it was already checked.
No Problem - i will just wait for an announcement

So just for high level planning, do you think Kernel 5.8 will be compatible this year? If not, i will check if i can go to Fedora 33 but using Kernel 5.7.

[ascaris]

He had said something about one of the major players in Linux being favorable to the idea of including whatever change was needed in their version of the kernel some time ago, if I recall. Has it been announced which one it was and how that was going?

[Gostev]

Thanks, but it was already checked.
No Problem - i will just wait for an announcement

It's a weekly digest. If you're not getting it, you should check your spam filters.

[Gostev]

Kernel 5.8 will be compatible this year?

In general, kernel 5.8 or later versions will never be supported for an image-level backup. The next supported kernel version is currently unknown, because it is beyond our control. Veeam Agent for Linux depends on the critical functionality that was removed starting from kernel 5.8, which allowed us to perform the I/O filtering for snapshot and changed block tracking purposes. There are currently a few options on the table on how to address this, including upstreaming the actual drivers in question.

Here are all 4 ways of how this may play out:
1. One major distributions told us they are simply not going to include the breaking change into their custom kernel at all.
2. Another major distribution considers to keep including the kernel patch we provided temporarily, until either of the following two possibilities happen.
3. Linux kernel maintainers may decide to upstream our out of tree drivers into the Linux kernel. By now, most work on our side has be done to enable this, and the drivers are in review by the community.
4. Linux kernel maintainers may decide to restore the lost functionality in some shape or form. Reviewing our submission from (3) has sparkled the corresponding conversations between them.

This was the status as of about one week ago, I did not check for updates since then.

[marcwa19197]

Hello Gostev,

thank you very much for that information. At the moment i also receive the Newsletter! Thanks.

So i guess, there is still no update or timeline? Or maybe a Roadmap?

[Gostev]

Here's my understanding - beg a pardon if I don't get certain terms right, as I'm far from the trenches.

Our developers have delivered the patch to restore the missing Linux kernel functionality, but were then told by maintainers that it cannot be accepted into the kernel without any "internal consumer" of its functionality. But, the community quickly determined device-mapper (dm) as the best candidate to become such a consumer.

So in the past month, our developers kept working with kernel maintainers, now on implementing the required changes into dm. However, the joint team is completely stuck at this time because the mission-critical Linux kernel maintainer for dm is currently busy with urgent work at his employer, and cannot spend any time on this until next year.

On a negative news side, the alpha version of RHEL 8.4 appears to have the entire block layer from kernel version 5.8 ported into their custom kernel based on version 4.18. This includes the "breaking change" (the kernel function loss that renders image-level backups impossible). This comes totally unexpected considering how conservative RedHat has been in terms of advancing their kernel version, and unfortunately also at the worst possible moment.

By the way, what is everyone's take on manually patching the kernel with the patch we implemented until it makes it into the tree sooner or later? I do realize it's a lot of work if you have many machines, but might be OK for Linux enthusiasts at least?

[marcwa19197]

Hello Gostev,

thank you very much for this information!

Is there maybe a small instruction how we can patch our kernel manually? Im not an expert on this, but a few steps with commands would be really helpful!
(As it was on Github for Kernel 5.7 https://github.com/mike2307/veeamsnap)

Thank you and have a happy new year!

[Finix]

If device registers submit_bio function in its block_device_operations, it is not hard to support in the current veeam snapshot framework, right? If so, it could at least make much of DM/MD (except multipath) device work.

[PTide]

Is there maybe a small instruction how we can patch our kernel manually? Im not an expert on this, but a few steps with commands would be really helpful!
(As it was on Github for Kernel 5.7 https://github.com/mike2307/veeamsnap)

There is no patch right now as we are currently working on a prototype. Once we are ready to provide such patch I will post links and instructions in this thread.

Thanks!

[KernelKurtz]

Since it looks like the Linux snapshot functionality is going to be broken shortly without this (what exactly are they thinking? - I know someone who uses Acronis in the enterprise which I believe uses the same functionality - seems unlike the Linux kernel maintainers to purposely break something important and widely used like this), I'd be willing to give the patch a try when it is available.

Will watch this space for more info. Many thanks!

[ibschreiber]

Hi folks,
backup on my Ubuntu 2004 stoped working after Kernel was upgraded to the 5.8 branch. Bad, really bad. As Veeam can't do anything about it at this point, the only option was to revert back to 5.6. But I didn't want to loose the ability to use newer Kernels when necessarry. So my solution was to install the 18.04 kernels while still allowing Kernels from 20.04 to be installed. how?

1: Install the older Kernels:

Code: Select all

apt install linux-generic-hwe-18.04

(or any other version, the hwe works fine on my laptop)

2: Install veeam (install the .dep to configure the source in apt, then

Code: Select all

sudo apt install veeam

This fails miserably when dkms tries to built veeamsnap for the 5.8 kernel, so....

3: Go to /usr/src/veeamsnap-x.x.x.xxxx and edit the file dkms.conf
add the line

Code: Select all

BUILD_EXCLUSIVE_KERNEL="^5.4.*"

to it so the package will only be built for the 5.4 kernels.

another friendly

Code: Select all

sudo dpkg-reconfigure veeamsnap

and all should be fine.

4: To set the default kernel to the 5.4, I used grub configure and enabled the option to select the last booted kernel as default.
you can also edit /etc/default/grub and set/add

Code: Select all

GRUB_SAVEDEFAULT="true"
GRUB_DEFAULT="saved"

to it.

That's it. Now you can boot into the 5.4 to have your backup working, boot into 5.8 or any other kernels when needed and the apt update process is only broken when veeam releases a new version of veeamsnap. Just repeat Step 3 and you're good to go.

@VEEAM: As veeamsnap is broken on higher kernels anyway, could you please include BUILD_EXCLUSIVE_KERNEL="^5.4.*" (maybe a bit more generalized) in your next release of the linux version. This would really help!

[marcwa19197]

There is no patch right now as we are currently working on a prototype. Once we are ready to provide such patch I will post links and instructions in this thread.

Thanks!

Thank you very much!
Will also wait and watch this thread for a update.

[HannesK]

@ibschreiber: I merged your post with the existing discussion.

Now you can boot into the 5.4 to have your backup working, boot into 5.8 or any other kernels when needed

I'm not sure how customers should handle that in practice. Who would reboot the system for backup?

[ibschreiber]

Well I'm working using Kernel 5.4. Jus for trying things out I switch to kernel 5.8. For servers this could be a way to get the backup working again. I think it depends on the requirements in each individual case.

[PTide]

For veeamsnap to work on kernels 5.8 and later it is required to add blk_interposer functionality.

This feature is being developed in blk-interposer.v2 branch here:

git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel

One needs only kernel changes (skip the device-mapper)

Therefore, only these two commits are required:

commit 8dddc3dd22db58c690ee2916266b136a1b80875b
Date: Thu Nov 19 17:49:23 2020 +0100
blk_interposer - Block Layer Interposer

commit 71e21b5e3941f4c0cacad47cc119141ca7fd2a66
Date: Fri Dec 11 19:01:19 2020 +0300
block: blk_interposer - change attach/detach logic

If are not into having too much fun with git, you can use the prepared patches from this branch:

0001-blk_interposer-Block-Layer-Interposer.patch
0003-block-blk_interposer-change-attach-detach-logic.patch

or use this cumulative patch:

blk_interposer_cumulative.patch

In order to install a stable vanilla kernel (5.10 for example) you have to download it first:

Code: Select all

 wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.3.tar.xz 

Unpack:

Code: Select all

 tar -xf ./linux-5.10.3.tar.xz 

Apply the patch:

Code: Select all

 cd ./linux-5.10.3
 patch -p1 <../0001-blk_interposer-Block-Layer-Interposer.patch
 patch -p1 <../0003-block-blk_interposer-change-attach-detach-logic.patch

Or apply the cumulative patch:

Code: Select all

 patch -p1 <../blk_interposer_cumulative.patch

Next, configure and build the kernel (as usual), and install it. blk_interposer does not require any additional configuration.

We recommend to build a package (deb or rpm, depending on your packet manager) and install the kernel as a package. For example, "make -j4 deb".

[SShtepa]

Hi all.

There is a solution for the 5.8 kernel, I posted it here https://github.com/veeam/veeamsnap/tree/experimental.
This is experimental code, do not rush to apply it without additional testing on the test infrastructure.

But the solution is temporary. Already for 5.9 we need blk_interposer https://github.com/veeam/veeamsnap/tree ... interposer.

[Gostev]

Our developers have delivered the patch to restore the missing Linux kernel functionality, but were then told by maintainers that it cannot be accepted into the kernel without any "internal consumer" of its functionality. But, the community quickly determined device-mapper (dm) as the best candidate to become such a consumer.

So in the past month, our developers kept working with kernel maintainers, now on implementing the required changes into dm. However, the joint team is completely stuck at this time because the mission-critical Linux kernel maintainer for dm is currently busy with urgent work at his employer, and cannot spend any time on this until next year.

Just to provide an update on the progress, in the end our developers were able to make the required updates to dm on their own, and the dm patch is now in review by the maintainers.

[marcwa19197]

This sounds really good.
Thank you very much Gostev for keeping us informed.

Looking forward to use Veeam Agent with Kernel 5.10+.

[marcwa19197]

Any idea in which kernel version the fix will be included?

Thanks.

[Gostev]

It's not up to us to decide. All we can do at this point is to keep the maintainers satisfied, which is the absolute top priority for our VAL devs. So they've been working hard on issuing all the changes and improvements requested by reviewers for the patch momentarily.

[PTide]

The latest news can be seen here:
https://patchwork.kernel.org/project/li ... ies=427209

[marcwa19197]

Thanks for the information.
Im just asking to know which Kernel i need to install if the patch is available. But i guess this will also be "announced" somehow by Veeam, or?
So i can plan the next reboot / kernel update a little better.

Thanks.

[Gostev]

Yes, of course this will be the big news we will share widely.

[PhoenixHGIC]

Any update on this?

I purchased VEEAM last year, one of the main features was a full backup of our Linux Servers. Unfortunately support now tell me that since 5.8 Kernel we can only perform a file level backup.

Thanks.

[PTide]

Hi,

Which distro are you using?

Thanks!

[Gostev]

Any update on this?

Not in 3 days after my last post

I purchased VEEAM last year, one of the main features was a full backup of our Linux Servers. Unfortunately support now tell me that since 5.8 Kernel we can only perform a file level backup.

Can you just stay with a pre-5.8 based LTS version of your distro until the kernel feature loss that enabled image-level backups is addressed?