CentOS 8.3.2011 Installed Veeam Agent 5.0, but veeamsnap issues

[rleon]

Hi all,

On a CentOS 8.3.2011 host, done the Veeam UEFI cert mokutil import, rebooted, installed the Veeam online Repo rpm, then installed Veeam Agent 5.0 successfully, as shown below.

Code: Select all

# rpm -i ./veeamsnap-ueficert-5.0.0.4318-1.noarch.rpm
  Certificate /etc/uefi/certs/veeamsnap-ueficert.crt has been imported successfully
# mokutil --import /etc/uefi/certs/veeamsnap-ueficert.crt
#reboot
*system rebooted*

Code: Select all

# cat /etc/centos-release
CentOS Linux release 8.3.2011
#
# yum install -y veeam
    ...
    Total download size: 69 M
    Installed size: 181 M
    Downloading Packages:
    (1/2): kmod-veeamsnap-5.0.0.4318-1.el8.x86_64.rpm                                                                                              81 kB/s |  87 kB     00:01
    (2/2): veeam-5.0.0.4318-1.el8.x86_64.rpm                                                                                                      6.0 MB/s |  69 MB     00:11
    Total                                                                                                                                         6.0 MB/s |  69 MB     00:11
    warning: /var/cache/dnf/veeam-0528727e29088715/packages/kmod-veeamsnap-5.0.0.4318-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID efdcea77: NOKEY
    Veeam Backup for GNU/Linux - x86_64                                                                                                            39 kB/s | 3.1 kB     00:00
    Importing GPG key 0xEFDCEA77:
     Userid     : "Veeam Software RPM Repository <support@veeam.com>"
     Fingerprint: 1DD6 EAD9 AF6E 8F34 465D 790C EA4F C237 EFDC EA77
     From       : http://repository.veeam.com/keys/RPM-EFDCEA77
    Key imported successfully
    Running transaction check
    Transaction check succeeded.
    Running transaction test
    Transaction test succeeded.
    Running transaction
      Preparing        :                                                                                                                                                      1/1
      Installing       : kmod-veeamsnap-5.0.0.4318-1.el8.x86_64                                                                                                               1/2
      Running scriptlet: veeam-5.0.0.4318-1.el8.x86_64                                                                                                                        2/2
      Installing       : veeam-5.0.0.4318-1.el8.x86_64                                                                                                                        2/2
      Running scriptlet: veeam-5.0.0.4318-1.el8.x86_64                                                                                                                        2/2
    Created symlink /etc/systemd/system/multi-user.target.wants/veeamservice.service > /usr/lib/systemd/system/veeamservice.service.
      Verifying        : kmod-veeamsnap-5.0.0.4318-1.el8.x86_64                                                                                                               1/2
      Verifying        : veeam-5.0.0.4318-1.el8.x86_64                                                                                                                        2/2
    Installed products updated.
    Installed:
      kmod-veeamsnap-5.0.0.4318-1.el8.x86_64   veeam-5.0.0.4318-1.el8.x86_64
#

How ever, Linux Agent backup job failed with:

Code: Select all

5:30:02 PM	Required backup infrastructure resources have been assigned 	
5:30:18 PM	Preparing to backup 	
5:30:18 PM	Failed to create volume snapshot 	
5:30:19 PM	Failed to take volume snapshot 	
5:30:19 PM	Failed to perform managed backup 	
5:30:19 PM	Module loading operation has returned code [255] 	
5:30:19 PM	Failed to load module [veeamsnap] with parameters [zerosnapdata=1 debuglogging=0 snapstore_block_size_pow=14 change_tracking_block_size_pow=18 logdir=/var/log/veeam fixflags=0 logmaxsize=15728640]. 	
5:32:21 PM	Processing finished with errors at 4/19/2021 5:32:21 PM 	

Turns out dkms status returns nothing, so, apparently veeamsnap is not installed even though the installation completed succesesfully...?

Code: Select all

# dkms status
#

Any ideas?
Thanks for your help.

[Natalia Lupacheva]

Hello,

Have you already tried the steps described in this KB?

Thank you!

[rleon]

Hi, yes I check out the KB, but I don't think it looks like the same issue.
I just found that:

Code: Select all

# modprobe veeamsnap
modprobe: ERROR: could not insert 'veeamsnap': Required key not available

... So apparently it is because Veeam's veeamsnap is an "unsigned kernel module" and that it is not allowed on UEFI Secure Boot enabled systems.

Is there a way to load this unsigned kernel module without disabling Secure Boot?

[Natalia Lupacheva]

Yes, message "Required key not available" appears if the kernel can't check if the module is valid, because Secure Boot is enabled - the module should be signed with a valid key for the kernel.
If you use UEFI system, there is a perfect explanation of your current situation here.

Did you enroll the key as it's described here?

To make UEFI systems with Secure Boot work with the pre-built veeamsnap kernel module, you need to enroll the Veeam public key to the MOK list using the mokutil utility. The key is available in the

Code: Select all

veeamsnap-ueficert-5.0.0.4318-1.noarch

package residing on the Veeam repository. To enroll the key:

1. Request the enrollment of the public key with the following command:

Code: Select all

mokutil --import veeamsnap-ueficert.crt.

2. Reboot the Veeam Agent computer and complete the enrollment in the UEFI console.

Thanks!

[rleon]

Hi Natalia,

Thanks for reminding me to check the mokutil again. Everything is working now.
I thought I did the mokutil procedure properly but turned out I didn't.

After you run mukutil --import and reboot, there is a 10-second window for you to login to the MOK enrollment menu using the server console (Console screen for VMs, or actual keyboard and monitor for physical hosts).
If you miss it, it will just boot normally back into Linux without actually having imported the Veeam cert into the UEFI MOK.

[Natalia Lupacheva]

Hi Rui,

Thank you for sharing your results, glad you've solved the issue.

Thanks!

[PTide]

Hi,

@rleon,

Just a small correction:

dkms status returns nothing, so, apparently veeamsnap is not installed even though the installation completed succesesfully...?

Since you installed kmod package, it is perfectly fine that dkms wouldn't return you anything related to Veeam.

DKMS overwatches and manages kernel modules that were built from source right on your machine. That would have been the case if you had installed the noarch veeamsnap package.

Kmods are packages that carry prebuilt and pre-signed kernel modules, hence dkms doesn't care about those.

Thanks!

[rleon]

Hi @PTide,

Yes, I later found out your documentation actually mentions this, that Veeam Agent on RHEL/CentOS does not use the dkms method, except for the following distros.

Veeam Agent for Linux 5.0 User Guide snippet:

Code: Select all

Use the dkms packages with the following distributions instead of the pre-built binary veeamsnap kernel
module packages:
Debian 9.0 – 10.4
Ubuntu 14.04, 16.04, 18.04, 19.10, 20.04
Oracle Linux 7 – 8.2 (RHCK)
Oracle Linux 7 – Oracle Linux 8.0 (up to UEK R6)
Fedora 32
openSUSE Tumbleweed