Standalone backup agent for Linux servers and workstations on-premises or in the public cloud
P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » Apr 11, 2018 10:14 am

Issue
Using Veeam Agent for Linux 2.0 or earlier on the latest release of Red Hat Enterprise Linux (RHEL) 7.5 and its derivatives (CentOS 7.5, Oracle RHCK 7.5) causes the operating system to crash during backup snapshot creation.

Cause
veeamsnap kernel module used in Veeam Agent for Linux 2.0 and earlier is not compatible with newly released RHEL 7.5 kernel 3.10.0-862

More info
We are researching the issue in collaboration with RedHat. Status updates will be posted here and in KB2659

13.06.2018 UPDATE: The fix is available via support team.
03.07.2018 UPDATE: New 2.0.1 version that addresses the issue is available via repositories.

cbuchleiter
Lurker
Posts: 1
Liked: never
Joined: Apr 14, 2018 3:31 pm
Full Name: Chris Buchleiter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cbuchleiter » Apr 14, 2018 3:39 pm

RHEL 7.4 has been updated with the 3.10.0-862 kernel and is exhibiting the same issue.

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » Apr 15, 2018 9:33 pm

Hi Chris,

Thank you for reporting that!

tsightler
VP, Product Management
Posts: 5376
Liked: 2209 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by tsightler » Apr 15, 2018 11:11 pm

Does this happen even if the module is recompiled from source? I know our module uses some ABI calls that are not marked as stable and thus could break between kernel trains in RHEL7.

stevelin
Novice
Posts: 5
Liked: 1 time
Joined: Apr 20, 2018 5:06 pm
Full Name: Steve Linthicum
Contact:

RHEL 7.5

Post by stevelin » Apr 20, 2018 5:09 pm

I've been testing veeam linux agent on a RHEL 7.4 physical oracle server in preparation of the Oracle plugin becoming available later this year. Last week we did our normal monthly patch cycle on this dev host and it patched to RHEL 7.5. Now when I try to run a backup through veeam the server reboots. I tried uninstalling and reinstalling and setting the backup up from scratch but get the same behavior. Run a backup, reboot and backup fails.

Anyone else seeing this issue?

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » Apr 20, 2018 6:43 pm

Hi,

This is a known issue, we are working on the fix. Please note that even if you recompile the module from source it won't work on 3.10.0-862

Thanks

GaryLynch
Lurker
Posts: 1
Liked: never
Joined: Apr 05, 2018 2:18 pm
Full Name: Gary Lynch
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by GaryLynch » Apr 26, 2018 8:41 am

I too am experiencing this problem. Have been evaluating the Agent on a server that is due to go into production soon and yesterday I performed some updates and now the server crashes. Do we know how long it will take to fix? This could dictate when our server can go into production or we use alternative backup methods.

Thanks
Gary

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » Apr 27, 2018 8:00 am

Support for 7.5 will be introduced in the next VAL update which will take at least a month to be released.

Thanks

cameltom
Novice
Posts: 3
Liked: never
Joined: Apr 30, 2018 7:39 am
Full Name: Thomas Gächter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cameltom » Apr 30, 2018 7:41 am

hi all,
any news about update for veeamagent (Linux)???
btw. not just Redhat 7.5 are affectet ..... 7.4 with last kernel-update (specter fix!)too.

regards

hajek
Influencer
Posts: 11
Liked: 1 time
Joined: Aug 01, 2016 4:11 pm
Full Name: Tomas Hajek
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by hajek » May 02, 2018 7:45 pm

I have 2 systems that are running RHEL 7.5 with Kernel 3.10.0-862.el7.x86_64 and VAL 2.0.0.400, one (Dell PowerEdge R720xd) has been successfully backing up for about 3 weeks and another (Dell PowerEdge R630) for 7 days. I have two other systems, however, (Dell PowerEdge R330's) that exhibit the same crashing behavior which lead me to this post today. Is this expected that some systems work and others don't? Should I submit a support request to include information on working vs. non-working systems? Is it know what the root cause of the issue is? Should I downgrade kernel versions to be on the safe side for my systems that are working (if so what version is safe)?
thanks

stevelin
Novice
Posts: 5
Liked: 1 time
Joined: Apr 20, 2018 5:06 pm
Full Name: Steve Linthicum
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by stevelin » May 03, 2018 2:37 pm

Support will tell you like they did me, RHEL 7.5 is not supported.

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » May 04, 2018 12:10 pm

Hi,
Is this expected that some systems work and others don't?
We've observed such behaviour during tests. For now it can be described as follows:
If the module has been built via dkms on 7.4 system with kernel version < 3.10.0-862, then dkms will fail to rebuild the module after kernel update, BUT weak-updates will kick in and the module might keep working. This is not supported as we don't know potential pitfalls of such approach.

Thanks

kge8889
Novice
Posts: 5
Liked: never
Joined: Jun 12, 2017 1:34 pm
Full Name: Ken Everett
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by kge8889 » May 10, 2018 4:03 pm

I have also seen this issue on a test server that was recently updated to 7.5. For now we will have to delay updating the production servers until this is resolved which becomes a serious issue in another month. We will have to ask for an exception since the servers can't be patched as it will break backups.

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » May 11, 2018 11:13 am

Hi,

The issue will be addressed in the next release.

Thanks

cameltom
Novice
Posts: 3
Liked: never
Joined: Apr 30, 2018 7:39 am
Full Name: Thomas Gächter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cameltom » May 14, 2018 5:36 am

Hi,
and when is the delivery date of this new release?!
We get really in trouble with the auditor ..... because of "old" kernels .... or not existing backups!

regards
tom

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » May 16, 2018 10:36 am

Unfortunately I cannot provide you with any estimates except the one that I've mentioned before. The update will be released as soon as our QA team ensures that everything is in place and work stable.

Thank you

Riverog
Lurker
Posts: 1
Liked: never
Joined: May 17, 2018 2:03 pm
Full Name: Gustavo Rivero
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Riverog » May 17, 2018 2:07 pm

Its been a month since the error was identified. You stated a month for the fix. A month of no backups is significant in a a business. So there is no date on the fix?

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » May 17, 2018 2:31 pm

Hi Gustavo,

In fact neither me nor anyone else has ever stated that the fix will be released within a month. What was stated 3 weeks ago is that "Support for 7.5 will be introduced in the next VAL update which will take at least a month to be released."

Thank you for your understanding

Gostev
SVP, Product Management
Posts: 24421
Liked: 3403 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev » May 20, 2018 9:47 pm

Just to remind here as well, our commitment is to release support for new platform/OS/kernel versions within 90 days of general availability, with the actual historical average time being 2 months, depending primarily on complexity of required fixes. To avoid ending up in situations such as above (having no backup for weeks due to incompatibility), we recommend not to upgrade to new platform/OS/kernel versions before they are officially supported by the corresponding Veeam product. Thank you!

stevelin
Novice
Posts: 5
Liked: 1 time
Joined: Apr 20, 2018 5:06 pm
Full Name: Steve Linthicum
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by stevelin » May 21, 2018 1:31 pm

So what does that say about it also not working in the latest updated 7.4 which is fully supported and there is no 90 day wait?!?!

nielsengelen
Veeam Software
Posts: 2609
Liked: 537 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by nielsengelen » May 21, 2018 2:35 pm 1 person likes this post

Steve, RHEL 7.4 has received the kernel update which causes this issue in the beginning of April. If you don't install this update, RHEL 7.4 systems will continue to work fine. Thanks!
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

ironwolve
Lurker
Posts: 1
Liked: never
Joined: May 21, 2018 8:14 pm
Full Name: B H
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by ironwolve » May 21, 2018 8:19 pm

Is there maybe a possible workaround? I just migrated to a new nfs server, and experiencing the backup veeam bug now.

Not sure why the client is tied to a kernel or a specific version of redhat/centos.

P.Tide
Product Manager
Posts: 5179
Liked: 447 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by P.Tide » May 22, 2018 8:08 pm

Not sure why the client is tied to a kernel or a specific version of redhat/centos.
VAL uses its own kernel module to operate, which depends on kernel's internal structures, one of which got modified.

Thanks

cameltom
Novice
Posts: 3
Liked: never
Joined: Apr 30, 2018 7:39 am
Full Name: Thomas Gächter
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by cameltom » May 24, 2018 10:16 am

Hi,

this i can read in the docu:
Prebuilt binaries require kernel >= 2.6.32-131.0.15 for CentOS / RHEL 6 and kernel >= 3.10.0-123 for CentOS / RHEL 7 to function.
For other kernels please install the veeamsnap module using the source rpm package available at the repository.

i downloaded the veeamsnap from this repository ...... with the same error on dkms build!

It's now 6 weeks since this error is well known ...... please let your customers know, when they can get a solution. In this situation i have to think about a backup change to a other solution.
Just because when "Murphy" visit us and we need a restore to a date after 10. April 2018 (!!!) ..... we just are not able to do it ..... it starts really to be a businesses critical issue.

regards

Gostev
SVP, Product Management
Posts: 24421
Liked: 3403 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev » May 24, 2018 12:33 pm

Current target is mid-June, we will keep you updated if these timelines change - we're trying to accelerate the release by all means possible, but we cannot sacrifice stability testing in particular.

danswartz
Expert
Posts: 159
Liked: 19 times
Joined: Apr 26, 2013 4:53 pm
Full Name: Dan Swartzendruber
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by danswartz » May 25, 2018 2:25 pm

Thanks, appreciate the heads-up. I was painfully bitten when I tried installing and running VAL on 3.10.0-862.3.2.el7.x86_64. Everything (apparently) installed okay, yet when I tried to do a backup, the system panicked and rebooted. Ugh...

Gostev
SVP, Product Management
Posts: 24421
Liked: 3403 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev » May 26, 2018 8:34 pm

Keep in mind there's a workaround for those who don't really NEED to use the problematic kernel update for whatever reason, and that is to simply go back to the previously used kernel until the new VAL version is released.

You can list all installed kernels using the following command:

Code: Select all

awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg
Then, you should modify "/boot/grub2/grubenv" by changing the saved_entry to match the kernel you want to load by default.

ManOrs
Enthusiast
Posts: 30
Liked: 3 times
Joined: Dec 20, 2016 6:39 am
Full Name: Manuel Orsatti
Location: Italy
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by ManOrs » May 27, 2018 9:53 am

if you don't like modifying "/boot/grub2/grubenv" by hand, you can instead use

Code: Select all

grub2-set-default <number>
where <number> is the position on the list obtained with awk (starting the count from 0)

Gostev
SVP, Product Management
Posts: 24421
Liked: 3403 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by Gostev » May 27, 2018 11:12 pm

Thanks Manuel, this certainly seems much less error-prone than what I suggested :D

paul777
Novice
Posts: 9
Liked: never
Joined: Feb 21, 2016 9:07 pm
Full Name: Paul
Contact:

Re: Veeam Agent for Linux and RHEL 7.5 crash

Post by paul777 » Jun 07, 2018 4:34 pm

Hi everyone,

We find ourselves in the same boat. Luckily we can wait about 3 more weeks, maybe until after July 4. After that it's back to TSM if not fixed. From reading this, last news was maybe a month from April 27 so hopefully the update/fix is close. If any new news is available please let us know. Thanks!

Locked

Who is online

Users browsing this forum: No registered users and 3 guests