Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Issue
Using Veeam Agent for Linux 2.0 or earlier on the latest release of Red Hat Enterprise Linux (RHEL) 7.5 and its derivatives (CentOS 7.5, Oracle RHCK 7.5) causes the operating system to crash during backup snapshot creation.

Cause
veeamsnap kernel module used in Veeam Agent for Linux 2.0 and earlier is not compatible with newly released RHEL 7.5 kernel 3.10.0-862

More info
We are researching the issue in collaboration with RedHat. Status updates will be posted here and in KB2659

13.06.2018 UPDATE: The fix is available via support team.
03.07.2018 UPDATE: New 2.0.1 version that addresses the issue is available via repositories.
cbuchleiter
Lurker
Posts: 1
Liked: never
Joined: Apr 14, 2018 3:31 pm
Full Name: Chris Buchleiter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cbuchleiter »

RHEL 7.4 has been updated with the 3.10.0-862 kernel and is exhibiting the same issue.
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Hi Chris,

Thank you for reporting that!
tsightler
VP, Product Management
Posts: 6034
Liked: 2859 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by tsightler »

Does this happen even if the module is recompiled from source? I know our module uses some ABI calls that are not marked as stable and thus could break between kernel trains in RHEL7.
stevelin
Novice
Posts: 5
Liked: 1 time
Joined: Apr 20, 2018 5:06 pm
Full Name: Steve Linthicum
Contact:

RHEL 7.5

Post by stevelin »

I've been testing veeam linux agent on a RHEL 7.4 physical oracle server in preparation of the Oracle plugin becoming available later this year. Last week we did our normal monthly patch cycle on this dev host and it patched to RHEL 7.5. Now when I try to run a backup through veeam the server reboots. I tried uninstalling and reinstalling and setting the backup up from scratch but get the same behavior. Run a backup, reboot and backup fails.

Anyone else seeing this issue?
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Hi,

This is a known issue, we are working on the fix. Please note that even if you recompile the module from source it won't work on 3.10.0-862

Thanks
GaryLynch
Lurker
Posts: 1
Liked: never
Joined: Apr 05, 2018 2:18 pm
Full Name: Gary Lynch
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by GaryLynch »

I too am experiencing this problem. Have been evaluating the Agent on a server that is due to go into production soon and yesterday I performed some updates and now the server crashes. Do we know how long it will take to fix? This could dictate when our server can go into production or we use alternative backup methods.

Thanks
Gary
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Support for 7.5 will be introduced in the next VAL update which will take at least a month to be released.

Thanks
cameltom
Novice
Posts: 3
Liked: never
Joined: Apr 30, 2018 7:39 am
Full Name: Thomas Gächter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cameltom »

hi all,
any news about update for veeamagent (Linux)???
btw. not just Redhat 7.5 are affectet ..... 7.4 with last kernel-update (specter fix!)too.

regards
hajek
Influencer
Posts: 11
Liked: 1 time
Joined: Aug 01, 2016 4:11 pm
Full Name: Tomas Hajek
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by hajek »

I have 2 systems that are running RHEL 7.5 with Kernel 3.10.0-862.el7.x86_64 and VAL 2.0.0.400, one (Dell PowerEdge R720xd) has been successfully backing up for about 3 weeks and another (Dell PowerEdge R630) for 7 days. I have two other systems, however, (Dell PowerEdge R330's) that exhibit the same crashing behavior which lead me to this post today. Is this expected that some systems work and others don't? Should I submit a support request to include information on working vs. non-working systems? Is it know what the root cause of the issue is? Should I downgrade kernel versions to be on the safe side for my systems that are working (if so what version is safe)?
thanks
stevelin
Novice
Posts: 5
Liked: 1 time
Joined: Apr 20, 2018 5:06 pm
Full Name: Steve Linthicum
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by stevelin »

Support will tell you like they did me, RHEL 7.5 is not supported.
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Hi,
Is this expected that some systems work and others don't?
We've observed such behaviour during tests. For now it can be described as follows:
If the module has been built via dkms on 7.4 system with kernel version < 3.10.0-862, then dkms will fail to rebuild the module after kernel update, BUT weak-updates will kick in and the module might keep working. This is not supported as we don't know potential pitfalls of such approach.

Thanks
kge8889
Novice
Posts: 6
Liked: never
Joined: Jun 12, 2017 1:34 pm
Full Name: Ken Everett
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by kge8889 »

I have also seen this issue on a test server that was recently updated to 7.5. For now we will have to delay updating the production servers until this is resolved which becomes a serious issue in another month. We will have to ask for an exception since the servers can't be patched as it will break backups.
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Hi,

The issue will be addressed in the next release.

Thanks
cameltom
Novice
Posts: 3
Liked: never
Joined: Apr 30, 2018 7:39 am
Full Name: Thomas Gächter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cameltom »

Hi,
and when is the delivery date of this new release?!
We get really in trouble with the auditor ..... because of "old" kernels .... or not existing backups!

regards
tom
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Unfortunately I cannot provide you with any estimates except the one that I've mentioned before. The update will be released as soon as our QA team ensures that everything is in place and work stable.

Thank you
Riverog
Lurker
Posts: 1
Liked: never
Joined: May 17, 2018 2:03 pm
Full Name: Gustavo Rivero
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Riverog »

Its been a month since the error was identified. You stated a month for the fix. A month of no backups is significant in a a business. So there is no date on the fix?
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Hi Gustavo,

In fact neither me nor anyone else has ever stated that the fix will be released within a month. What was stated 3 weeks ago is that "Support for 7.5 will be introduced in the next VAL update which will take at least a month to be released."

Thank you for your understanding
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev »

Just to remind here as well, our commitment is to release support for new platform/OS/kernel versions within 90 days of general availability, with the actual historical average time being 2 months, depending primarily on complexity of required fixes. To avoid ending up in situations such as above (having no backup for weeks due to incompatibility), we recommend not to upgrade to new platform/OS/kernel versions before they are officially supported by the corresponding Veeam product. Thank you!
stevelin
Novice
Posts: 5
Liked: 1 time
Joined: Apr 20, 2018 5:06 pm
Full Name: Steve Linthicum
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by stevelin »

So what does that say about it also not working in the latest updated 7.4 which is fully supported and there is no 90 day wait?!?!
nielsengelen
Product Manager
Posts: 5792
Liked: 1214 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by nielsengelen » 1 person likes this post

Steve, RHEL 7.4 has received the kernel update which causes this issue in the beginning of April. If you don't install this update, RHEL 7.4 systems will continue to work fine. Thanks!
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
ironwolve
Lurker
Posts: 1
Liked: never
Joined: May 21, 2018 8:14 pm
Full Name: B H
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by ironwolve »

Is there maybe a possible workaround? I just migrated to a new nfs server, and experiencing the backup veeam bug now.

Not sure why the client is tied to a kernel or a specific version of redhat/centos.
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by PTide »

Not sure why the client is tied to a kernel or a specific version of redhat/centos.
VAL uses its own kernel module to operate, which depends on kernel's internal structures, one of which got modified.

Thanks
cameltom
Novice
Posts: 3
Liked: never
Joined: Apr 30, 2018 7:39 am
Full Name: Thomas Gächter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cameltom »

Hi,

this i can read in the docu:
Prebuilt binaries require kernel >= 2.6.32-131.0.15 for CentOS / RHEL 6 and kernel >= 3.10.0-123 for CentOS / RHEL 7 to function.
For other kernels please install the veeamsnap module using the source rpm package available at the repository.

i downloaded the veeamsnap from this repository ...... with the same error on dkms build!

It's now 6 weeks since this error is well known ...... please let your customers know, when they can get a solution. In this situation i have to think about a backup change to a other solution.
Just because when "Murphy" visit us and we need a restore to a date after 10. April 2018 (!!!) ..... we just are not able to do it ..... it starts really to be a businesses critical issue.

regards
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev »

Current target is mid-June, we will keep you updated if these timelines change - we're trying to accelerate the release by all means possible, but we cannot sacrifice stability testing in particular.
danswartz
Veteran
Posts: 266
Liked: 30 times
Joined: Apr 26, 2013 4:53 pm
Full Name: Dan Swartzendruber
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by danswartz »

Thanks, appreciate the heads-up. I was painfully bitten when I tried installing and running VAL on 3.10.0-862.3.2.el7.x86_64. Everything (apparently) installed okay, yet when I tried to do a backup, the system panicked and rebooted. Ugh...
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev »

Keep in mind there's a workaround for those who don't really NEED to use the problematic kernel update for whatever reason, and that is to simply go back to the previously used kernel until the new VAL version is released.

You can list all installed kernels using the following command:

Code: Select all

awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg
Then, you should modify "/boot/grub2/grubenv" by changing the saved_entry to match the kernel you want to load by default.
ManOrs
Enthusiast
Posts: 30
Liked: 3 times
Joined: Dec 20, 2016 6:39 am
Full Name: Manuel Orsatti
Location: Italy
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by ManOrs »

if you don't like modifying "/boot/grub2/grubenv" by hand, you can instead use

Code: Select all

grub2-set-default <number>
where <number> is the position on the list obtained with awk (starting the count from 0)
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev »

Thanks Manuel, this certainly seems much less error-prone than what I suggested :D
paul777
Novice
Posts: 9
Liked: never
Joined: Feb 21, 2016 9:07 pm
Full Name: Paul
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by paul777 »

Hi everyone,

We find ourselves in the same boat. Luckily we can wait about 3 more weeks, maybe until after July 4. After that it's back to TSM if not fixed. From reading this, last news was maybe a month from April 27 so hopefully the update/fix is close. If any new news is available please let us know. Thanks!
Locked

Who is online

Users browsing this forum: No registered users and 5 guests