Veeam Agent for Linux 6 volume backups fail for RHEL 8.7

[aj_potc]

After updating Veeam Linux agents to the latest version, I'm unable to make volume backups under RHEL 8.7.

Every attempt throws the error:

Code: Select all

Failed to load module [veeamsnap]

To test further, I've spun up a fresh VM, installed RHEL 8.7 from ISO, and then started VAL 6 in workstation mode. When I start a volume backup, I get the same error. This version of RHEL and its kernel should be supported, and worked fine with VAL 5.

I just wanted to see if the dev team is aware of the issue, and if a fix or workaround is known.

I opened a support ticket (case #05879296) with my findings and with logs attached.

Thanks!

[HannesK]

Hello,
let's see what support finds out. One of the servers had secure boot enabled.

Which of the servers had exactly the same kernel worked before with VAL5? Because 4.18.0-425.10.1 is not listed for VAL 5 in https://www.veeam.com/kb2804 (update of that article for V6 is ongoing)

Best regards,
Hannes

[aj_potc]

Hi Hannes,

One system was updated to kernel 4.18.0-425.10.1 recently, and I believe it was running on that version successfully with VAL 5. I'll see what support says about it.

But I have another test system (in this case Rocky Linux) which is running an older kernel, 4.18.0-372.9.1. That was also working fine with VAL 5, but now fails with VAL 6. The error log for that system is very similar to the one for RHEL, which has the following lines:

Code: Select all

vsnap    |   Checking the compatibility of the kernel module with others
vsnap    |   Loading kernel module [veeamsnap] with parameters [zerosnapdata=1 debuglogging=0 snapstore_block_size_pow=14 change_tracking_block_size_pow=18 logdir=/var/log/veeam fixflags=0 logmaxsize=15728640] and loader [/usr/sbin/veeamsnap-loader].

But with one very odd addition that makes no sense to me:

Code: Select all

vsnap    | WARN|Your Linux kernel version is higher than the version this product build has been tested against, which may cause issues with some functionality.

Any idea why that line would show up for a kernel that is compatible?

After that line, I see the following:

Code: Select all

vsnap    |   Module [veeamsnap] has been loaded.
vsnap    | ERR |The module [veeamsnap] was not loaded.

I realize Rocky Linux is not yet supported, but I find it odd that an older kernel that worked fine with VAL 5 wouldn't work properly under VAL 6. Any ideas about that?

Thanks very much for your help.

[VAXman65]

I opened Case #05880975 — Linux system backup job fails with Failed to create volume snapshot

Key part of the job log appears to be:

Code: Select all

[16.02.2023 05:30:46.939] <140423288297216> lpbcore  |   Taking snapshots.
[16.02.2023 05:30:46.939] <140423288297216> vmb      |     [SessionLog][processing] Creating volume snapshot.
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | WARN|Method invocation was not finalized. Method id [1]. Class: [lpbcorelib::interaction::ISnapshotOperation]
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | ERR |Failed to load module [veeamsnap].
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:Failed to create Veeam snapshots.
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:in void lpbcorelib::service::CSnapshotService::CreateVeeamSnapshots(const SVeeamSnapDeviceRequests&, const CSnapshotResourcePtr&, const Ptr&) at /mnt/Sources/LPBCoreLib/lpbcorelib/service/SnapshotService.cpp:146
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:Failed to finish snapshot creation process.
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:in virtual lpbcorelib::interaction::ISnapshotResourceLockPtr lpbcorelib::service::CSnapshotOperation::EndCreateSnapshot(lpbcorelib::service::CSnapshotTask::Ptr&) at /mnt/Sources/LPBCoreLib/lpbcorelib/service/SnapshotOperation.cpp:33
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:Failed to execute method [1] for class [lpbcorelib::interaction::ISnapshotOperation].
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:in void remoting::CStubBase<TStub, TObject>::ExecuteMethod(remoting::method_id_t, remoting::IStub::SDispatchContext&) const [with TStub = lpbcorelib::interaction::proxystub::CSnapshotOperationStub; TObject = lpbcorelib::interaction::ISnapshotOperation; remoting::method_id_t = unsigned int] at /mnt/Sources/LPBCoreLib/lpbcorelib/interaction/proxystub/./../../../ProtocolLib/remoting/StubBase.h:77
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:Failed to invoke method [1] in class [lpbcorelib::interaction::ISnapshotOperation].
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |--tr:in void remoting::CProxyBase::CMethodInvocationStream<TObject>::FinalizeExc() [with TObject = lpbcorelib::interaction::ISnapshotOperation] at /mnt/Sources/LPBCoreLib/lpbcorelib/interaction/proxystub/../../../../ProtocolLib/remoting/ProxyBase.h:92
[16.02.2023 05:30:53.242] <140423288297216> prtcl    | >>  |An exception was thrown from thread [140423288297216].
[16.02.2023 05:30:53.242] <140423288297216> vmb      |     [SessionLog][error] Failed to create volume snapshot.
[16.02.2023 05:30:53.243] <140423288297216> vmb      |     [SessionLog][error] Failed to take volume snapshot.
[16.02.2023 05:30:53.244] <140423288297216> lpbcore  |   Taking snapshots. Failed.
[16.02.2023 05:30:53.244] <140423288297216> vmb      |   [SessionLog][info] Executing post-job script.
[16.02.2023 05:30:53.245] <140423288297216> lpbcommon|   Executing custom script: [/usr/local/sbin/veeam_backup_status-email.sh]. Arguments: []
[16.02.2023 05:30:53.245] <140423279904512>          | Thread started.  Role: 'script output redirector to buffer', thread id: 140423279904512, parent id: 140423288297216.
[16.02.2023 05:30:53.245] <140423288297216> lpbcommon|     Wait child process for finish.
[16.02.2023 05:30:53.245] <140423271511808>          | Thread started.  Role: 'script output redirector to log', thread id: 140423271511808, parent id: 140423288297216.
[16.02.2023 05:30:53.245] <140423263119104>          | Thread started.  Role: 'Pid waiter. timeout: 600', thread id: 140423263119104, parent id: 140423288297216.
[16.02.2023 05:30:53.249] <140423263119104>          | Thread finished. Role: 'Pid waiter. timeout: 600'.
[16.02.2023 05:30:54.246] <140423271511808>          | Thread finished. Role: 'script output redirector to log'.
[16.02.2023 05:30:54.246] <140423279904512>          | Thread finished. Role: 'script output redirector to buffer'.
[16.02.2023 05:30:54.247] <140423288297216> lpbcore  | BackupJobPerformer: Creating backup. Failed.
[16.02.2023 05:30:54.247] <140423288297216> vmb      | [SessionLog][error] Failed to perform backup.
[16.02.2023 05:30:54.247] <140423288297216> vmb      | [SessionLog][error] Failed to load module [veeamsnap].
[16.02.2023 05:30:54.248] <140423288297216> vmb      | [SessionLog][error] Processing finished with errors at 2023-02-16 10:30:54 UTC.
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | ERR |Failed to load module [veeamsnap].
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:Failed to create Veeam snapshots.
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:in void lpbcorelib::service::CSnapshotService::CreateVeeamSnapshots(const SVeeamSnapDeviceRequests&, const CSnapshotResourcePtr&, const Ptr&) at /mnt/Sources/LPBCoreLib/lpbcorelib/service/SnapshotService.cpp:146
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:Failed to finish snapshot creation process.
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:in virtual lpbcorelib::interaction::ISnapshotResourceLockPtr lpbcorelib::service::CSnapshotOperation::EndCreateSnapshot(lpbcorelib::service::CSnapshotTask::Ptr&) at /mnt/Sources/LPBCoreLib/lpbcorelib/service/SnapshotOperation.cpp:33
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:Failed to execute method [1] for class [lpbcorelib::interaction::ISnapshotOperation].
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:in void remoting::CStubBase<TStub, TObject>::ExecuteMethod(remoting::method_id_t, remoting::IStub::SDispatchContext&) const [with TStub = lpbcorelib::interaction::proxystub::CSnapshotOperationStub; TObject = lpbcorelib::interaction::ISnapshotOperation; remoting::method_id_t = unsigned int] at /mnt/Sources/LPBCoreLib/lpbcorelib/interaction/proxystub/./../../../ProtocolLib/remoting/StubBase.h:77
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:Failed to take volume snapshot.
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:in static lpbcorelib::interaction::ISnapshotResourceLockPtr lpbcorelib::backup::CSnapshotHelper::TakeSnapshot(vmbplatform::model::CBackupJobPtr, const std::__cxx11::list<boost::shared_ptr<lpbcorelib::backup::IBackupObject> >&, const object_id_t&, vmbplatform::model::CSessionLoggerPtr, lpbcorelib::backup::CCbtDataContainer&, bool) at /mnt/Sources/LPBCreLib/lpbcorelib/backup/snapshot/SnapshotHelper.cpp:135
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |Backup job has failed.
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |--tr:in virtual void lpbcorelib::backup::CBackupJobPerformer::execute() at /mnt/Sources/LPBCoreLib/lpbcorelib/backup/BackupJobPerformer.cpp:210
[16.02.2023 05:30:54.248] <140423288297216> lpbcore  | >>  |An exception was thrown from thread [140423288297216].
[16.02.2023 05:30:54.250] <140423662724032>          |   Closing socket device.
[16.02.2023 05:30:54.250] <140423662724032> lpbcore  | LpbManSession: Processing commands. ok.
[16.02.2023 05:30:54.250] <140423662724032>          | Closing socket device.
[16.02.2023 05:30:54.273] <140423288297216> lpbcore  | JOB STATUS: FAILED.

[VAXman65]

My fail is actually on Rocky 8.7, but it appears to be all the same.

[root@hendrick ~]# uname -a
Linux hendrick.lan.kitsnet.us 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Thu Jan 12 16:32:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@hendrick ~]# cat /etc/rocky-release
Rocky Linux release 8.7 (Green Obsidian)
[root@hendrick ~]# veeamconfig --version
v6.0.0.1060

[PTide]

Hello everyone,

Kindly check whether you have installed the updated UEFI certificate for SecureBoot:

http://repository.veeam.com/backup/linu ... noarch.rpm

Please let us know if it helped.

Thank you!

[Gostev]

https://helpcenter.veeam.com/docs/agent ... tml?ver=60

[aj_potc]

In my case, the solution to this problem was different for each of three systems.

On one system, the presence of Secure Boot prevented Veeam from working. I was able to fix that by disabling Secure Boot.

On another system, the veeamsnap module was not properly installed. A reinstallation with yum resolved that.

And in the third case, a Rocky Linux system, I discovered that the veeamsnap module checks for the OS ID value in /etc/os-release, and will not load if it finds "rocky" instead of "rhel".

[PTide]

nd in the third case, a Rocky Linux system, I discovered that the veeamsnap module checks for the OS ID value in /etc/os-release, and will not load if it finds "rocky" instead of "rhel".

Unfortunately Rocky (and other CentOS substitutes) is not supported yet. If you are confident that it will work fine, and really want to try, then use dkms module please.

Thank you!