Standalone backup agents for Linux and Unix workloads on-premises or in the public cloud
Post Reply
rodney.vanderwoude
Lurker
Posts: 2
Liked: never
Joined: May 26, 2020 8:15 am
Full Name: Rod
Contact:

Veeam slow on new OS support

Post by rodney.vanderwoude »

We run mostly RHEL8.x based systems. In order to meet PCI compliance we tend to have these fully patched and on the latest versions. However time after time new versions of RHEL break veeam backup (linux agent for physical servers). RHEL8.2 for instance is not supported/working.

What is Veeam doing about always been the weakest link in the chain and supporting the latest operating system updates in a timely manner?

And yes i am aware there is a release note (standard veeam response) but keeping our systems out of date to not break Veeam is reason enough to start looking at alternatives.

PTide
Product Manager
Posts: 5561
Liked: 528 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam slow on new OS support

Post by PTide »

Hi,

How fast do you adopt new versions of distros? On day zero, 2 weeks, 1 month?

Thanks!

rodney.vanderwoude
Lurker
Posts: 2
Liked: never
Joined: May 26, 2020 8:15 am
Full Name: Rod
Contact:

Re: Veeam slow on new OS support

Post by rodney.vanderwoude »

Hi

usually within 2 weeks for non-production followed 2 weeks later for production.

In the end, having a dependency on backup software is the last thing people should need to think about, now it needs to be the first.

Take care

Gostev
SVP, Product Management
Posts: 26291
Liked: 4100 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam slow on new OS support

Post by Gostev »

Hey, Rod.

We could use some support from customers like yourself with Red Hat. Most vendors (including the biggest, like Microsoft and VMware) have Veeam in their QC labs, and a part of their QC process to ensure the new product builds and fixes don't "break" it. And when they do - because the changes are necessary - then they give us a heads up in advance, so we have enough time to prepare our product for these changes.

For example, the major legacy API removal VMware executed in vSphere 7 required us to spend almost a year of re-writing most of the low-level code. But we had an advanced warning from them, because they saw this change causing issues in their Veeam test lab - so we started this work over 1 year before vSphere 7 was shipped. And as a result, such a huge change went completely seamless for our customers.

Just imagine if next RHEL release introduces a change of the similar magnitude, and it takes us 6 months to deliver support for it, where your expectations is 2 weeks.

But obviously, for us to get into the vendor's QC labs and test matrix, requires pressure for the vendor's customers themselves. Neither Microsoft nor VMware had us in their labs right away, it was multiple requests of their own customers that got us there.

Thanks!

Ejdesgaard
Service Provider
Posts: 34
Liked: 6 times
Joined: Aug 24, 2012 11:59 am
Contact:

Re: Veeam slow on new OS support

Post by Ejdesgaard »

Hey Gostev,

I think it's a cheap shot to blame the vendor in this case... Here's why.
In order for you to properly support any enterprise distribution, you really need to do at least a some-effort* support on the community distributions that they are based on.
If you do that for RHEL, then you need to have a some-effort* support for Fedora https://en.wikipedia.org/wiki/Red_Hat_E ... ith_Fedora because that will give you a very early warning on any major changes that eventually will be included in RHEL.

RHEL-8 was released on May 7, 2019 and is based on Fedora 28, released on: 2018-05-01.
There you got your early warning with the added benefit that you have a _much_ more diverse(and properly larger) test/QC base then your or Red Hat's internal lab/QC, namely all the daily users of Fedora Workstation (and server)
Just imagine how nice your product would be on Windows and VMware if they had such a diverse test-base where you are able to discuss any given issue openly, due to the lack of nda of various sort.

Ofc. it would be better then nothing if an Enterprise linux shop like Red Hat or Suse got your software in their QC ecosystem, but it would by no means outweigh the benefit you would get if your do a some-effort* support on the distros that those enterprise solutions are based on.

*some-effort: the bar is set higher then what you deliver today, we can still not compile veeamsnap from your experimental branch, let alone get support on it, when on kernel-5.6
https://github.com/veeam/veeamsnap/pull/5/commits even tho a pull request came in 46 days ago to fix the issue

ps.
Your lack of support for kernel-5.6 would have been catched no later then during the merge window of 5.6, that's about 7-10 weeks before release, if you got the veeamsnap kernel module mainlined. and any future bugfixes would be able to be backported to official kernel releases, allowing you to state "we support latest x.y.bugfix kernel release"

PTide
Product Manager
Posts: 5561
Liked: 528 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam slow on new OS support

Post by PTide »

Hi @Ejdesgaard,

The pull request has been reviewed by the development team. Indeed, the changes proposed in the pull request do fix the issue for 5.6, so thank you for sharing your solution. Unfortunately, it also breaks compatibility with older RH 6 kernels that are not EOL yet and that we still support.
In order for you to properly support any enterprise distribution, you really need to do at least a some-effort* support on the community distributions that they are based on.
We do support Fedora and SLES distros. In fact, although Fedora 32 is not listed as supported in VAL v4.0, our development team has been trying it since Day-1. The same holds true for SLES/openSUSE - we keep an eye on the base-distros exactly for the reason that we want to catch as much changes and as early as possible.
<...>that will give you a very early warning on any major changes that eventually will be included in RHEL.
Had RedHat used the same version of kernel as Fedora that would have been great. Unfortunately, that's not the case - we have already encountered RedHat-specific critical issues, for instance this one. And that's not even a minor update.

That is, we have our own testing and security requirements. We understand the need for faster release schedule and we are currently working on it.

Thanks!

Ejdesgaard
Service Provider
Posts: 34
Liked: 6 times
Joined: Aug 24, 2012 11:59 am
Contact:

Re: Veeam slow on new OS support

Post by Ejdesgaard »

The pull request has been reviewed by the development team. Indeed, the changes proposed in the pull request do fix the issue for 5.6, so thank you for sharing your solution. Unfortunately, it also breaks compatibility with older RH 6 kernels that are not EOL yet and that we still support.
Correct me if I'm wrong, but would this issue not be permanently resolved, moving forward, if you mainlined the driver ?
Alternatively, Veeam needs to branch off the driver, such that you have a branch for each kernel version that you want to support... It sounds like a lot more work in the long run then to get it mainlined :)
Had RedHat used the same version of kernel as Fedora that would have been great. Unfortunately, that's not the case - we have already encountered RedHat-specific critical issues, for instance this one. And that's not even a minor update.
I agree that's an annoying bug, but it's less bad then CBT or ReFS corruption bugs that have been creeping up from time to time... bugs do happen and I don't see this as a valid excuse to not have a functioning veeamsnap driver 55 days after a fix pull request have been made.


Credits for the commit that fixes 5.6 must be directed to https://github.com/mike2307 since I had no part in the commit.

Gostev
SVP, Product Management
Posts: 26291
Liked: 4100 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam slow on new OS support

Post by Gostev »

If I understand it correctly, the suggested change cannot be accepted because it breaks compatibility for earlier RH6 kernels which are still under support from Veeam perspective.

However, have you considered forking the driver and just maintaining the forked version yourself? This is the beauty of open source, and is the very reason why Veeam has made these drivers available under GPL. Everyone is free to do with them whatever they like! :D

mike2307
Novice
Posts: 9
Liked: 1 time
Joined: Mar 26, 2019 10:42 am
Contact:

Re: Veeam slow on new OS support

Post by mike2307 »

Hi Gostev and PTide!

Here's the author of the above mentioned pull request (https://github.com/veeam/veeamsnap/pull/5/files) that was supposed to fix compilation issues with kernel version 5.6.x. In the meantime I managed to also incorporate some changes to get version 5.7.x going in addition.

Thanks and lot to your developers at veeam for not giving me ANY (!) feedback in the last 2 months or so and leaving my pull request abandoned till the end of days. :shock:
If I would have gotten any feedback, I would have been able to react to it.

I updated the pull request to use compile switches for all my changes, resulting in no breaks of compatibility. That would have been a no-brainer for your developers as well.

Anyway... Can you please (!) manage that my pull request gets reviewed again and provide some feedback resp. merge it?

Thanks a lot.

PTide
Product Manager
Posts: 5561
Liked: 528 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam slow on new OS support

Post by PTide »

Hi @mike2307,

I have reminded our dev team about your requests, thank you.

Please keep in mind, that the guys in our development team are usually occupied by their primary tasks and therefore can respond slower than you expect them to.

Thanks!

Ejdesgaard
Service Provider
Posts: 34
Liked: 6 times
Joined: Aug 24, 2012 11:59 am
Contact:

Re: Veeam slow on new OS support

Post by Ejdesgaard »

@Gostev
Please clarify how we or other customers will get proper support from the Veeam support team if we follow your advice.

@PTide you need to clarify your statement.
Are you saying that the Veeam Agent Dev team's primary task is something different then fixing catastrophic usability bugs in the software they are responsible for ?
If that's the case then I would really like to know what their primary task is...

@Mike2307 Thanks for your contribution, it's greatly appreciated!
Even more so when there seems to be no dev team for Veeam Agent, within the Veeam corp.

Gostev
SVP, Product Management
Posts: 26291
Liked: 4100 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam slow on new OS support

Post by Gostev »

If you want to get support from the Veeam support team, you should use the official agent, and ensure you're compliant with the official System Requirements for the Linux OS versions you're using.

The primary task for the Veeam Agent dev team is to ship on time periodic updates with new functionality - which includes support for the newest OS versions - without breaking any existing functionality, include the compatibility with the previous OS versions too. Our next release vehicle is right this month, so as you can imagine they've been quite busy indeed.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests