Veeam Agent for Linux and RHEL 7.5 crash

[PTide]

Issue
Using Veeam Agent for Linux 2.0 or earlier on the latest release of Red Hat Enterprise Linux (RHEL) 7.5 and its derivatives (CentOS 7.5, Oracle RHCK 7.5) causes the operating system to crash during backup snapshot creation.

Cause
veeamsnap kernel module used in Veeam Agent for Linux 2.0 and earlier is not compatible with newly released RHEL 7.5 kernel 3.10.0-862

More info
We are researching the issue in collaboration with RedHat. Status updates will be posted here and in KB2659

13.06.2018 UPDATE: The fix is available via support team.
03.07.2018 UPDATE: New 2.0.1 version that addresses the issue is available via repositories.

[cbuchleiter]

RHEL 7.4 has been updated with the 3.10.0-862 kernel and is exhibiting the same issue.

[PTide]

Hi Chris,

Thank you for reporting that!

[tsightler]

Does this happen even if the module is recompiled from source? I know our module uses some ABI calls that are not marked as stable and thus could break between kernel trains in RHEL7.

[stevelin]

I've been testing veeam linux agent on a RHEL 7.4 physical oracle server in preparation of the Oracle plugin becoming available later this year. Last week we did our normal monthly patch cycle on this dev host and it patched to RHEL 7.5. Now when I try to run a backup through veeam the server reboots. I tried uninstalling and reinstalling and setting the backup up from scratch but get the same behavior. Run a backup, reboot and backup fails.

Anyone else seeing this issue?

[PTide]

Hi,

This is a known issue, we are working on the fix. Please note that even if you recompile the module from source it won't work on 3.10.0-862

Thanks

[GaryLynch]

I too am experiencing this problem. Have been evaluating the Agent on a server that is due to go into production soon and yesterday I performed some updates and now the server crashes. Do we know how long it will take to fix? This could dictate when our server can go into production or we use alternative backup methods.

Thanks
Gary

[PTide]

Support for 7.5 will be introduced in the next VAL update which will take at least a month to be released.

Thanks

[cameltom]

hi all,
any news about update for veeamagent (Linux)???
btw. not just Redhat 7.5 are affectet ..... 7.4 with last kernel-update (specter fix!)too.

regards

[hajek]

I have 2 systems that are running RHEL 7.5 with Kernel 3.10.0-862.el7.x86_64 and VAL 2.0.0.400, one (Dell PowerEdge R720xd) has been successfully backing up for about 3 weeks and another (Dell PowerEdge R630) for 7 days. I have two other systems, however, (Dell PowerEdge R330's) that exhibit the same crashing behavior which lead me to this post today. Is this expected that some systems work and others don't? Should I submit a support request to include information on working vs. non-working systems? Is it know what the root cause of the issue is? Should I downgrade kernel versions to be on the safe side for my systems that are working (if so what version is safe)?
thanks

[stevelin]

Support will tell you like they did me, RHEL 7.5 is not supported.

[PTide]

Hi,

Is this expected that some systems work and others don't?

We've observed such behaviour during tests. For now it can be described as follows:
If the module has been built via dkms on 7.4 system with kernel version < 3.10.0-862, then dkms will fail to rebuild the module after kernel update, BUT weak-updates will kick in and the module might keep working. This is not supported as we don't know potential pitfalls of such approach.

Thanks

[kge8889]

I have also seen this issue on a test server that was recently updated to 7.5. For now we will have to delay updating the production servers until this is resolved which becomes a serious issue in another month. We will have to ask for an exception since the servers can't be patched as it will break backups.

[PTide]

Hi,

The issue will be addressed in the next release.

Thanks

[cameltom]

Hi,
and when is the delivery date of this new release?!
We get really in trouble with the auditor ..... because of "old" kernels .... or not existing backups!

regards
tom

[PTide]

Unfortunately I cannot provide you with any estimates except the one that I've mentioned before. The update will be released as soon as our QA team ensures that everything is in place and work stable.

Thank you

[Riverog]

Its been a month since the error was identified. You stated a month for the fix. A month of no backups is significant in a a business. So there is no date on the fix?

[PTide]

Hi Gustavo,

In fact neither me nor anyone else has ever stated that the fix will be released within a month. What was stated 3 weeks ago is that "Support for 7.5 will be introduced in the next VAL update which will take at least a month to be released."

Thank you for your understanding

[Gostev]

Just to remind here as well, our commitment is to release support for new platform/OS/kernel versions within 90 days of general availability, with the actual historical average time being 2 months, depending primarily on complexity of required fixes. To avoid ending up in situations such as above (having no backup for weeks due to incompatibility), we recommend not to upgrade to new platform/OS/kernel versions before they are officially supported by the corresponding Veeam product. Thank you!

[stevelin]

So what does that say about it also not working in the latest updated 7.4 which is fully supported and there is no 90 day wait?!?!

[nielsengelen]

Steve, RHEL 7.4 has received the kernel update which causes this issue in the beginning of April. If you don't install this update, RHEL 7.4 systems will continue to work fine. Thanks!

[ironwolve]

Is there maybe a possible workaround? I just migrated to a new nfs server, and experiencing the backup veeam bug now.

Not sure why the client is tied to a kernel or a specific version of redhat/centos.

[PTide]

Not sure why the client is tied to a kernel or a specific version of redhat/centos.

VAL uses its own kernel module to operate, which depends on kernel's internal structures, one of which got modified.

Thanks

[cameltom]

Hi,

this i can read in the docu:
Prebuilt binaries require kernel >= 2.6.32-131.0.15 for CentOS / RHEL 6 and kernel >= 3.10.0-123 for CentOS / RHEL 7 to function.
For other kernels please install the veeamsnap module using the source rpm package available at the repository.

i downloaded the veeamsnap from this repository ...... with the same error on dkms build!

It's now 6 weeks since this error is well known ...... please let your customers know, when they can get a solution. In this situation i have to think about a backup change to a other solution.
Just because when "Murphy" visit us and we need a restore to a date after 10. April 2018 (!!!) ..... we just are not able to do it ..... it starts really to be a businesses critical issue.

regards

[Gostev]

Current target is mid-June, we will keep you updated if these timelines change - we're trying to accelerate the release by all means possible, but we cannot sacrifice stability testing in particular.

[danswartz]

Thanks, appreciate the heads-up. I was painfully bitten when I tried installing and running VAL on 3.10.0-862.3.2.el7.x86_64. Everything (apparently) installed okay, yet when I tried to do a backup, the system panicked and rebooted. Ugh...

[Gostev]

Keep in mind there's a workaround for those who don't really NEED to use the problematic kernel update for whatever reason, and that is to simply go back to the previously used kernel until the new VAL version is released.

You can list all installed kernels using the following command:

Code: Select all

awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg

Then, you should modify "/boot/grub2/grubenv" by changing the saved_entry to match the kernel you want to load by default.

[ManOrs]

if you don't like modifying "/boot/grub2/grubenv" by hand, you can instead use

Code: Select all

grub2-set-default <number>

where <number> is the position on the list obtained with awk (starting the count from 0)

[Gostev]

Thanks Manuel, this certainly seems much less error-prone than what I suggested

[paul777]

Hi everyone,

We find ourselves in the same boat. Luckily we can wait about 3 more weeks, maybe until after July 4. After that it's back to TSM if not fixed. From reading this, last news was maybe a month from April 27 so hopefully the update/fix is close. If any new news is available please let us know. Thanks!