Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

[LINUX] Kernel 5.8 and later

Post by marcwa19197 »

Hello,

i know this is an "big topic" in this forum, i read much threads about it.
I just want to ask for a status for the Kernel 5.8 compatibility. As you might know, Fedora 33 was released a few days ago. Fedora 33 is using Kernel 5.8 as default.

Is there any timeline? If not, i will wait for official compatibility with Fedora 33.
At the moment im using Fedora 32 with an older Kernel, because Kernel 5.8 is not working at the moment.

I read something about an newsletter, how can i subscribe to it?

Thank you very much.

PTide
Product Manager
Posts: 5789
Liked: 581 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Fedora 33 - Kernel 5.8

Post by PTide »

Hi,
Is there any timeline? If not, i will wait for official compatibility with Fedora 33.
Unfortunately there is no timeline for now.
I read something about an newsletter, how can i subscribe to it?
I guess you mean "Word from Gostev"? Just click on a "wrench" icon under the topic name and select "Subscribe"

Thanks!

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: Fedora 33 - Kernel 5.8

Post by marcwa19197 »

Thank you for the information!
Im getting
"You are not authorised to read this forum."
if i follow your link..

PTide
Product Manager
Posts: 5789
Liked: 581 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Fedora 33 - Kernel 5.8

Post by PTide »

My bad. Please try "User Control Panel > "Board Preferences" tab > "Weekly Digest Subscription" section > "Subscribe to weekly digest" radio button.

Thanks!

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: Fedora 33 - Kernel 5.8

Post by marcwa19197 »

Thanks, but it was already checked.
No Problem - i will just wait for an announcement :-)

So just for high level planning, do you think Kernel 5.8 will be compatible this year? If not, i will check if i can go to Fedora 33 but using Kernel 5.7.

ascaris
Novice
Posts: 4
Liked: 1 time
Joined: May 08, 2019 6:16 am
Contact:

Re: Fedora 33 - Kernel 5.8

Post by ascaris »

He had said something about one of the major players in Linux being favorable to the idea of including whatever change was needed in their version of the kernel some time ago, if I recall. Has it been announced which one it was and how that was going?

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Fedora 33 - Kernel 5.8

Post by Gostev »

marcwa19197 wrote: Oct 30, 2020 8:07 amThanks, but it was already checked.
No Problem - i will just wait for an announcement :-)
It's a weekly digest. If you're not getting it, you should check your spam filters.

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Fedora 33 - Kernel 5.8

Post by Gostev » 1 person likes this post

marcwa19197 wrote: Oct 30, 2020 8:07 amKernel 5.8 will be compatible this year?
In general, kernel 5.8 or later versions will never be supported for an image-level backup. The next supported kernel version is currently unknown, because it is beyond our control. Veeam Agent for Linux depends on the critical functionality that was removed starting from kernel 5.8, which allowed us to perform the I/O filtering for snapshot and changed block tracking purposes. There are currently a few options on the table on how to address this, including upstreaming the actual drivers in question.

Here are all 4 ways of how this may play out:
1. One major distributions told us they are simply not going to include the breaking change into their custom kernel at all.
2. Another major distribution considers to keep including the kernel patch we provided temporarily, until either of the following two possibilities happen.
3. Linux kernel maintainers may decide to upstream our out of tree drivers into the Linux kernel. By now, most work on our side has be done to enable this, and the drivers are in review by the community.
4. Linux kernel maintainers may decide to restore the lost functionality in some shape or form. Reviewing our submission from (3) has sparkled the corresponding conversations between them.

This was the status as of about one week ago, I did not check for updates since then.

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: Fedora 33 - Kernel 5.8

Post by marcwa19197 »

Hello Gostev,

thank you very much for that information. At the moment i also receive the Newsletter! Thanks.

So i guess, there is still no update or timeline? Or maybe a Roadmap?

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Fedora 33 - Kernel 5.8

Post by Gostev » 2 people like this post

Here's my understanding - beg a pardon if I don't get certain terms right, as I'm far from the trenches.

Our developers have delivered the patch to restore the missing Linux kernel functionality, but were then told by maintainers that it cannot be accepted into the kernel without any "internal consumer" of its functionality. But, the community quickly determined device-mapper (dm) as the best candidate to become such a consumer.

So in the past month, our developers kept working with kernel maintainers, now on implementing the required changes into dm. However, the joint team is completely stuck at this time because the mission-critical Linux kernel maintainer for dm is currently busy with urgent work at his employer, and cannot spend any time on this until next year.

On a negative news side, the alpha version of RHEL 8.4 appears to have the entire block layer from kernel version 5.8 ported into their custom kernel based on version 4.18. This includes the "breaking change" (the kernel function loss that renders image-level backups impossible). This comes totally unexpected considering how conservative RedHat has been in terms of advancing their kernel version, and unfortunately also at the worst possible moment.

By the way, what is everyone's take on manually patching the kernel with the patch we implemented until it makes it into the tree sooner or later? I do realize it's a lot of work if you have many machines, but might be OK for Linux enthusiasts at least?

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by marcwa19197 »

Hello Gostev,

thank you very much for this information!

Is there maybe a small instruction how we can patch our kernel manually? Im not an expert on this, but a few steps with commands would be really helpful!
(As it was on Github for Kernel 5.7 https://github.com/mike2307/veeamsnap)

Thank you and have a happy new year!

Finix
Lurker
Posts: 1
Liked: never
Joined: Dec 28, 2020 8:52 am
Full Name: FinixYan
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by Finix »

If device registers submit_bio function in its block_device_operations, it is not hard to support in the current veeam snapshot framework, right? If so, it could at least make much of DM/MD (except multipath) device work.

PTide
Product Manager
Posts: 5789
Liked: 581 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by PTide » 1 person likes this post

Is there maybe a small instruction how we can patch our kernel manually? Im not an expert on this, but a few steps with commands would be really helpful!
(As it was on Github for Kernel 5.7 https://github.com/mike2307/veeamsnap)
There is no patch right now as we are currently working on a prototype. Once we are ready to provide such patch I will post links and instructions in this thread.

Thanks!

KernelKurtz
Novice
Posts: 9
Liked: 3 times
Joined: Dec 08, 2020 5:20 pm
Full Name: Blair Steenerson
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by KernelKurtz » 1 person likes this post

Since it looks like the Linux snapshot functionality is going to be broken shortly without this (what exactly are they thinking? - I know someone who uses Acronis in the enterprise which I believe uses the same functionality - seems unlike the Linux kernel maintainers to purposely break something important and widely used like this), I'd be willing to give the patch a try when it is available.

Will watch this space for more info. Many thanks!

ibschreiber
Influencer
Posts: 18
Liked: 3 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

[MERGED] Ubuntu 20.04 backup stoped working - solution (kind of)

Post by ibschreiber » 1 person likes this post

Hi folks,
backup on my Ubuntu 2004 stoped working after Kernel was upgraded to the 5.8 branch. Bad, really bad. As Veeam can't do anything about it at this point, the only option was to revert back to 5.6. But I didn't want to loose the ability to use newer Kernels when necessarry. So my solution was to install the 18.04 kernels while still allowing Kernels from 20.04 to be installed. how?

1: Install the older Kernels:

Code: Select all

apt install linux-generic-hwe-18.04
(or any other version, the hwe works fine on my laptop)

2: Install veeam (install the .dep to configure the source in apt, then

Code: Select all

sudo apt install veeam
This fails miserably when dkms tries to built veeamsnap for the 5.8 kernel, so....

3: Go to /usr/src/veeamsnap-x.x.x.xxxx and edit the file dkms.conf
add the line

Code: Select all

BUILD_EXCLUSIVE_KERNEL="^5.4.*"
to it so the package will only be built for the 5.4 kernels.

another friendly

Code: Select all

sudo dpkg-reconfigure veeamsnap
and all should be fine.

4: To set the default kernel to the 5.4, I used grub configure and enabled the option to select the last booted kernel as default.
you can also edit /etc/default/grub and set/add

Code: Select all

GRUB_SAVEDEFAULT="true"
GRUB_DEFAULT="saved"
to it.

That's it. Now you can boot into the 5.4 to have your backup working, boot into 5.8 or any other kernels when needed and the apt update process is only broken when veeam releases a new version of veeamsnap. Just repeat Step 3 and you're good to go.

@VEEAM: As veeamsnap is broken on higher kernels anyway, could you please include BUILD_EXCLUSIVE_KERNEL="^5.4.*" (maybe a bit more generalized) in your next release of the linux version. This would really help!
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by marcwa19197 »

PTide wrote: Dec 29, 2020 1:36 pm There is no patch right now as we are currently working on a prototype. Once we are ready to provide such patch I will post links and instructions in this thread.

Thanks!
Thank you very much!
Will also wait and watch this thread for a update.

HannesK
Veeam Software
Posts: 7084
Liked: 1129 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by HannesK »

@ibschreiber: I merged your post with the existing discussion.
Now you can boot into the 5.4 to have your backup working, boot into 5.8 or any other kernels when needed
I'm not sure how customers should handle that in practice. Who would reboot the system for backup?

ibschreiber
Influencer
Posts: 18
Liked: 3 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by ibschreiber »

Well I'm working using Kernel 5.4. Jus for trying things out I switch to kernel 5.8. For servers this could be a way to get the backup working again. I think it depends on the requirements in each individual case.
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP

PTide
Product Manager
Posts: 5789
Liked: 581 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by PTide »

For veeamsnap to work on kernels 5.8 and later it is required to add blk_interposer functionality.

This feature is being developed in blk-interposer.v2 branch here:

git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel

One needs only kernel changes (skip the device-mapper)

Therefore, only these two commits are required:

commit 8dddc3dd22db58c690ee2916266b136a1b80875b
Date: Thu Nov 19 17:49:23 2020 +0100
blk_interposer - Block Layer Interposer


commit 71e21b5e3941f4c0cacad47cc119141ca7fd2a66
Date: Fri Dec 11 19:01:19 2020 +0300
block: blk_interposer - change attach/detach logic


If are not into having too much fun with git, you can use the prepared patches from this branch:

0001-blk_interposer-Block-Layer-Interposer.patch
0003-block-blk_interposer-change-attach-detach-logic.patch


or use this cumulative patch:

blk_interposer_cumulative.patch


In order to install a stable vanilla kernel (5.10 for example) you have to download it first:

Code: Select all

 wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.3.tar.xz 
Unpack:

Code: Select all

 tar -xf ./linux-5.10.3.tar.xz 
Apply the patch:

Code: Select all

 cd ./linux-5.10.3
 patch -p1 <../0001-blk_interposer-Block-Layer-Interposer.patch
 patch -p1 <../0003-block-blk_interposer-change-attach-detach-logic.patch
Or apply the cumulative patch:

Code: Select all

 patch -p1 <../blk_interposer_cumulative.patch
Next, configure and build the kernel (as usual), and install it. blk_interposer does not require any additional configuration.

We recommend to build a package (deb or rpm, depending on your packet manager) and install the kernel as a package. For example, "make -j4 deb".

SShtepa
Veeam Software
Posts: 1
Liked: 1 time
Joined: Apr 09, 2013 10:12 am
Full Name: Sergei Shtepa
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by SShtepa » 1 person likes this post

Hi all.

There is a solution for the 5.8 kernel, I posted it here https://github.com/veeam/veeamsnap/tree/experimental.
This is experimental code, do not rush to apply it without additional testing on the test infrastructure.

But the solution is temporary. Already for 5.9 we need blk_interposer https://github.com/veeam/veeamsnap/tree ... interposer.

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Fedora 33 - Kernel 5.8

Post by Gostev » 1 person likes this post

Gostev wrote: Dec 12, 2020 9:21 pmOur developers have delivered the patch to restore the missing Linux kernel functionality, but were then told by maintainers that it cannot be accepted into the kernel without any "internal consumer" of its functionality. But, the community quickly determined device-mapper (dm) as the best candidate to become such a consumer.

So in the past month, our developers kept working with kernel maintainers, now on implementing the required changes into dm. However, the joint team is completely stuck at this time because the mission-critical Linux kernel maintainer for dm is currently busy with urgent work at his employer, and cannot spend any time on this until next year.
Just to provide an update on the progress, in the end our developers were able to make the required updates to dm on their own, and the dm patch is now in review by the maintainers.

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by marcwa19197 » 1 person likes this post

This sounds really good.
Thank you very much Gostev for keeping us informed.

Looking forward to use Veeam Agent with Kernel 5.10+.

marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by marcwa19197 »

Any idea in which kernel version the fix will be included?

Thanks.

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by Gostev » 1 person likes this post

It's not up to us to decide. All we can do at this point is to keep the maintainers satisfied, which is the absolute top priority for our VAL devs. So they've been working hard on issuing all the changes and improvements requested by reviewers for the patch momentarily.

PTide
Product Manager
Posts: 5789
Liked: 581 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by PTide » 1 person likes this post


marcwa19197
Influencer
Posts: 24
Liked: 2 times
Joined: Dec 05, 2019 1:30 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by marcwa19197 »

Thanks for the information.
Im just asking to know which Kernel i need to install if the patch is available. But i guess this will also be "announced" somehow by Veeam, or? :-)
So i can plan the next reboot / kernel update a little better.

Thanks.

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by Gostev » 1 person likes this post

Yes, of course this will be the big news we will share widely.

PhoenixHGIC
Lurker
Posts: 2
Liked: never
Joined: Jan 26, 2021 10:54 am
Full Name: Martin Painter-Sims
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by PhoenixHGIC »

Any update on this?

I purchased VEEAM last year, one of the main features was a full backup of our Linux Servers. Unfortunately support now tell me that since 5.8 Kernel we can only perform a file level backup.

Thanks.

PTide
Product Manager
Posts: 5789
Liked: 581 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by PTide »

Hi,

Which distro are you using?

Thanks!

Gostev
SVP, Product Management
Posts: 27722
Liked: 4718 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [LINUX] Kernel 5.8 and later

Post by Gostev »

PhoenixHGIC wrote: Feb 11, 2021 1:19 pmAny update on this?
Not in 3 days after my last post :D
PhoenixHGIC wrote: Feb 11, 2021 1:19 pmI purchased VEEAM last year, one of the main features was a full backup of our Linux Servers. Unfortunately support now tell me that since 5.8 Kernel we can only perform a file level backup.
Can you just stay with a pre-5.8 based LTS version of your distro until the kernel feature loss that enabled image-level backups is addressed?

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 4 guests