Veeam slow on new OS support

[rodney.vanderwoude]

We run mostly RHEL8.x based systems. In order to meet PCI compliance we tend to have these fully patched and on the latest versions. However time after time new versions of RHEL break veeam backup (linux agent for physical servers). RHEL8.2 for instance is not supported/working.

What is Veeam doing about always been the weakest link in the chain and supporting the latest operating system updates in a timely manner?

And yes i am aware there is a release note (standard veeam response) but keeping our systems out of date to not break Veeam is reason enough to start looking at alternatives.

[PTide]

Hi,

How fast do you adopt new versions of distros? On day zero, 2 weeks, 1 month?

Thanks!

[rodney.vanderwoude]

Hi

usually within 2 weeks for non-production followed 2 weeks later for production.

In the end, having a dependency on backup software is the last thing people should need to think about, now it needs to be the first.

Take care

[Gostev]

Hey, Rod.

We could use some support from customers like yourself with Red Hat. Most vendors (including the biggest, like Microsoft and VMware) have Veeam in their QC labs, and a part of their QC process to ensure the new product builds and fixes don't "break" it. And when they do - because the changes are necessary - then they give us a heads up in advance, so we have enough time to prepare our product for these changes.

For example, the major legacy API removal VMware executed in vSphere 7 required us to spend almost a year of re-writing most of the low-level code. But we had an advanced warning from them, because they saw this change causing issues in their Veeam test lab - so we started this work over 1 year before vSphere 7 was shipped. And as a result, such a huge change went completely seamless for our customers.

Just imagine if next RHEL release introduces a change of the similar magnitude, and it takes us 6 months to deliver support for it, where your expectations is 2 weeks.

But obviously, for us to get into the vendor's QC labs and test matrix, requires pressure for the vendor's customers themselves. Neither Microsoft nor VMware had us in their labs right away, it was multiple requests of their own customers that got us there.

Thanks!

[Ejdesgaard]

Hey Gostev,

I think it's a cheap shot to blame the vendor in this case... Here's why.
In order for you to properly support any enterprise distribution, you really need to do at least a some-effort* support on the community distributions that they are based on.
If you do that for RHEL, then you need to have a some-effort* support for Fedora https://en.wikipedia.org/wiki/Red_Hat_E ... ith_Fedora because that will give you a very early warning on any major changes that eventually will be included in RHEL.

RHEL-8 was released on May 7, 2019 and is based on Fedora 28, released on: 2018-05-01.
There you got your early warning with the added benefit that you have a _much_ more diverse(and properly larger) test/QC base then your or Red Hat's internal lab/QC, namely all the daily users of Fedora Workstation (and server)
Just imagine how nice your product would be on Windows and VMware if they had such a diverse test-base where you are able to discuss any given issue openly, due to the lack of nda of various sort.

Ofc. it would be better then nothing if an Enterprise linux shop like Red Hat or Suse got your software in their QC ecosystem, but it would by no means outweigh the benefit you would get if your do a some-effort* support on the distros that those enterprise solutions are based on.

*some-effort: the bar is set higher then what you deliver today, we can still not compile veeamsnap from your experimental branch, let alone get support on it, when on kernel-5.6
https://github.com/veeam/veeamsnap/pull/5/commits even tho a pull request came in 46 days ago to fix the issue

ps.
Your lack of support for kernel-5.6 would have been catched no later then during the merge window of 5.6, that's about 7-10 weeks before release, if you got the veeamsnap kernel module mainlined. and any future bugfixes would be able to be backported to official kernel releases, allowing you to state "we support latest x.y.bugfix kernel release"

[PTide]

Hi @Ejdesgaard,

The pull request has been reviewed by the development team. Indeed, the changes proposed in the pull request do fix the issue for 5.6, so thank you for sharing your solution. Unfortunately, it also breaks compatibility with older RH 6 kernels that are not EOL yet and that we still support.

In order for you to properly support any enterprise distribution, you really need to do at least a some-effort* support on the community distributions that they are based on.

We do support Fedora and SLES distros. In fact, although Fedora 32 is not listed as supported in VAL v4.0, our development team has been trying it since Day-1. The same holds true for SLES/openSUSE - we keep an eye on the base-distros exactly for the reason that we want to catch as much changes and as early as possible.

<...>that will give you a very early warning on any major changes that eventually will be included in RHEL.

Had RedHat used the same version of kernel as Fedora that would have been great. Unfortunately, that's not the case - we have already encountered RedHat-specific critical issues, for instance this one. And that's not even a minor update.

That is, we have our own testing and security requirements. We understand the need for faster release schedule and we are currently working on it.

Thanks!

[Ejdesgaard]

The pull request has been reviewed by the development team. Indeed, the changes proposed in the pull request do fix the issue for 5.6, so thank you for sharing your solution. Unfortunately, it also breaks compatibility with older RH 6 kernels that are not EOL yet and that we still support.

Correct me if I'm wrong, but would this issue not be permanently resolved, moving forward, if you mainlined the driver ?
Alternatively, Veeam needs to branch off the driver, such that you have a branch for each kernel version that you want to support... It sounds like a lot more work in the long run then to get it mainlined

Had RedHat used the same version of kernel as Fedora that would have been great. Unfortunately, that's not the case - we have already encountered RedHat-specific critical issues, for instance this one. And that's not even a minor update.

I agree that's an annoying bug, but it's less bad then CBT or ReFS corruption bugs that have been creeping up from time to time... bugs do happen and I don't see this as a valid excuse to not have a functioning veeamsnap driver 55 days after a fix pull request have been made.

Credits for the commit that fixes 5.6 must be directed to https://github.com/mike2307 since I had no part in the commit.

[Gostev]

If I understand it correctly, the suggested change cannot be accepted because it breaks compatibility for earlier RH6 kernels which are still under support from Veeam perspective.

However, have you considered forking the driver and just maintaining the forked version yourself? This is the beauty of open source, and is the very reason why Veeam has made these drivers available under GPL. Everyone is free to do with them whatever they like!

[mike2307]

Hi Gostev and PTide!

Here's the author of the above mentioned pull request (https://github.com/veeam/veeamsnap/pull/5/files) that was supposed to fix compilation issues with kernel version 5.6.x. In the meantime I managed to also incorporate some changes to get version 5.7.x going in addition.

Thanks and lot to your developers at veeam for not giving me ANY (!) feedback in the last 2 months or so and leaving my pull request abandoned till the end of days.
If I would have gotten any feedback, I would have been able to react to it.

I updated the pull request to use compile switches for all my changes, resulting in no breaks of compatibility. That would have been a no-brainer for your developers as well.

Anyway... Can you please (!) manage that my pull request gets reviewed again and provide some feedback resp. merge it?

Thanks a lot.

[PTide]

Hi @mike2307,

I have reminded our dev team about your requests, thank you.

Please keep in mind, that the guys in our development team are usually occupied by their primary tasks and therefore can respond slower than you expect them to.

Thanks!

[Ejdesgaard]

@Gostev
Please clarify how we or other customers will get proper support from the Veeam support team if we follow your advice.

@PTide you need to clarify your statement.
Are you saying that the Veeam Agent Dev team's primary task is something different then fixing catastrophic usability bugs in the software they are responsible for ?
If that's the case then I would really like to know what their primary task is...

@Mike2307 Thanks for your contribution, it's greatly appreciated!
Even more so when there seems to be no dev team for Veeam Agent, within the Veeam corp.

[Gostev]

If you want to get support from the Veeam support team, you should use the official agent, and ensure you're compliant with the official System Requirements for the Linux OS versions you're using.

The primary task for the Veeam Agent dev team is to ship on time periodic agent updates with the new functionality - which includes support for the newest OS versions - without breaking any existing functionality, including the compatibility with the previous OS versions too. Our next release vehicle is right this month, so as you can imagine they've been quite busy indeed.

[marcwa19197]

Hi Gostev and PTide!

Here's the author of the above mentioned pull request (https://github.com/veeam/veeamsnap/pull/5/files) that was supposed to fix compilation issues with kernel version 5.6.x. In the meantime I managed to also incorporate some changes to get version 5.7.x going in addition.

Thanks and lot to your developers at veeam for not giving me ANY (!) feedback in the last 2 months or so and leaving my pull request abandoned till the end of days.
If I would have gotten any feedback, I would have been able to react to it.

I updated the pull request to use compile switches for all my changes, resulting in no breaks of compatibility. That would have been a no-brainer for your developers as well.

Anyway... Can you please (!) manage that my pull request gets reviewed again and provide some feedback resp. merge it?

Thanks a lot.

Thank you very much, just used your forked branch to get it working with kernel 5.7.7.
Next time i have to use a newer kernel i will count on you! Also you should prepare for an donate button !

So if i understand correctly this fix will be provided by Veeam this month?

Thank you and best regards

[PTide]

Hi,

That's right - VAL will receive update that adds support for kernel 5.7.7 (for Fedora and openSUSE)

Thanks!

[dovla091]

Veeam version 4.0.0.1961 on Fedora 32 5.7.10 kernel -> from veeam official repository does not work... what now?

[Gostev]

Support for Fedora 32 is included in 10a, see the sticky topic.

[dovla091]

Previously we had an app installed on local machine, so any backup was done locally, not via server. VBR10 requires Windows sever installation and then deploy to clients (plus license). Does this means that we cannot use Veeam agent for linux locally on that specific machine or tha thist new version will be added into linux repository?

How can I deploy then new version 4.0.1 of linux agent?

[premeau]

It does help you avoid the need for a Windows machine (or some other method to deal with the MSI files), but to get 4.0.1 on my unmanaged machines, I do an administrative (network share install) of the VAL MSI from the 10a ISO (after mounted):

Code: Select all

msiexec /a x:\Packages\VALRedist.msi

After doing so, you'd find the packages to install on a standalone machine at <install path>\CommonAppData\Veeam\Agents\val

Ultimately, it is the same client, and if history is any guide, it will be in the repository a bit further down the release process, but it does update and work on unmanaged machines as expeted.

[Ejdesgaard]

In addition to the current production repositories, found here: http://repository.veeam.com/backup/linu ... /7/x86_64/ it would be nice with RTM and early-access repositories in addition.
If nothing else, it would be a really nice "quality-of-life" addition.

[rodney.vanderwoude]

Still no RHEL8.2 support and RHEL8.3 Beta is out..

[Ejdesgaard]

@rodney.vanderwoude
Technically that's not correct, see: https://www.veeam.com/kb3228

You "just" need to:
#1 Download the full ~4GB iso.
#2 find an MSI file within the iso.
#3 Find a Windows env. in order to extact the rpm, see post by @premeau
#4 Add it to an internal custom repository - Not sure if the rpm is properly signed such that the repository.veeam.com key can be used.
#5 Do the normal Linux management workflow

If Veeam managed to update the repository, then it would be a ~0.04GB download, in contrast to the above, and it would fit quite nicely into existing Linux management workflows

I completely agree with you from a practical point of view, no support for RHEL-8.2 atm.

@Veeam please update http://repository.veeam.com/backup/linu ... /8/x86_64/ with the latest bits.

[swiffer]

from the windows forums:

Veeam v10a and Agents are planned to go GA (General Available) on 4th of august if everything is going well (no major errors found until then).
After that, they are available to download from the page.

so -> let's see

[Dima P.]

swiffer,

Already available via website (previously was available for several weeks via support). Cheers!

[swiffer]

Yepp, and now available in the repositories as well. Sadly (i know, Just released) the new Release of veeamsnap does Not Work for 5.8. we already updated the pull request in github but while compiling fine it currently causes weird Crashes on Backup start.

[rodney.vanderwoude]

Merely for info. Veeam Agent for Linux (4.x anyways) will unceremoniously crash 8.4 release of RHEL without warning.

[Mildur]

Yes, RHEL is not supported. Everything can happen with unsupported OS:

Agent 4.0 --> RHEL 6.0 – 8.2
Agent 5.0 --> RHEL 6.0 - 8.3

RHEL 8.4 was released GA 2021-05-18.
Veeam's statement is, to support new OS levels in 90 days after GA date, if there are not to many issues in the development phase. For RHEL, official support can take up until 18th august.

https://helpcenter.veeam.com/archive/ag ... ments.html
https://helpcenter.veeam.com/docs/agent ... tml?ver=50

[rodney.vanderwoude]

Perhaps a useful config option would be to abort the agent on unsupported OS builds? A failed backup due to an unsupported OS version seems slightly better than an unexpected system panic IMO.

[Dima P.]

Hello Rod,

Thanks for the feedback, we will discuss it with the RnD folks!

[dejan.ilic]

More than once, newer OS updates has been breaking backup jobs due to snapshot not being able to be created.
Latest one we have now is RHEL 8.4. Due to internal security requirements we keep as up to date as possible and clients have started being updated to that level. Any Agent based backup on these fails now due to snapshot error.

What I would wish for is a checkbox allowing for snapshotless backup failback for that client if veeam snapshot fail, but still triggering a warning in the system.
That would be a much better situation for us than staying without a backup for up to 90 days until a new agent is released.
Also it would avoid forcing us from moving the clients to another snapshotless jobschedule, leaving us with the same level of protection as the checkbox would allow us to have without any administration/overhead.

[PTide]

Hi,

Thank you for the proposal, noted.

Cheers