Veam mixes up multiple encrypted disks

[ibschreiber]

Hi,
I have 3 disks in my machine and on each disk are encrypted volumes.

Linux (Ubuntu 16.04 LTS) is mounting these disks in an unpredictable order (ls of /dev/mapper):

Code: Select all

lrwxrwxrwx  1 root root       7 Nov 20 14:54 BACKUP -> ../dm-4
crw-------  1 root root 10, 236 Nov 20 14:54 control
lrwxrwxrwx  1 root root       7 Nov 20 14:54 daten-linux -> ../dm-5
lrwxrwxrwx  1 root root       7 Nov 20 14:54 lukslvm -> ../dm-0
lrwxrwxrwx  1 root root       7 Nov 20 14:54 vgubuntu-Daten_LinuxSSD -> ../dm-3
lrwxrwxrwx  1 root root       7 Nov 20 14:54 vgubuntu-root -> ../dm-2
lrwxrwxrwx  1 root root       7 Nov 20 14:54 vgubuntu-swap -> ../dm-1

sometimes, BACKUP is mapped to dm-4, sometimes it is dm-5.
With daten-linux it's the other way around.
Veeam Agent doesn't use the entries in /dev/mapper but the /dev/dm-* to describe what to backup and to map this.
The result is that when the assigned dm-? changes and veeam takes a backup, it backs up the whole disks as everything changed.
Any idea how to prevent this?

[PTide]

Hi,

It's not quite clear what schema do you have on your disks. Do you have LUKs over LVM? Does your VG occupy all three disks? Are BACKUP, daten-linux, lukslvm the encrypted partitions? Please clarify.

Thanks

[ibschreiber]

Hi, I'll try to clarify things:
There are three disks:
sda sdb sdc
on each disk, there is a crypt-luks partition

The luks on sda shows up as a device /dev/dm-0
on this device is a lvm group called vgubuntu with three volumes
vgubuntu-Daten_LinuxSSD -> ../dm-3
vgubuntu-root -> ../dm-2
vgubuntu-swap -> ../dm-1

the luks partition on sdb and sdc gets dm-4 and dm-5 assigned. But, there is no way to tell which one gets which after booting the machine.

Here another view from lsblk:

Code: Select all

NAME                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdb                             8:16   0   1,8T  0 disk  
└─sdb1                          8:17   0   1,8T  0 part  
  └─BACKUP                    253:4    0   1,8T  0 crypt /mnt/BACKUP
sdc                             8:32   0 931,5G  0 disk  
├─sdc2                          8:34   0 247,8G  0 part  /mnt/Daten_Win
└─sdc1                          8:33   0 683,7G  0 part  
  └─daten-linux               253:5    0 683,7G  0 crypt /mnt/Daten_Linux
sda                             8:0    0 256,2G  0 disk  
├─sda2                          8:2    0  97,8G  0 part  
├─sda5                          8:5    0 156,3G  0 part  
│ └─lukslvm                   253:0    0 156,3G  0 crypt 
│   ├─vgubuntu-swap           253:1    0    24G  0 lvm   [SWAP]
│   ├─vgubuntu-root           253:2    0    64G  0 lvm   /
│   └─vgubuntu-Daten_LinuxSSD 253:3    0  68,3G  0 lvm   /mnt/Daten_LinuxSSD
├─sda3                          8:3    0     2G  0 part  /boot
└─sda1                          8:1    0   100M  0 part  

The problem is this: one time after boot, /dev/maper/ looks like this:

Code: Select all

lrwxrwxrwx  1 root root       7 Nov 22 11:31 BACKUP -> ../dm-4
lrwxrwxrwx  1 root root       7 Nov 22 11:31 daten-linux -> ../dm-5

the next time it looks like this:

Code: Select all

lrwxrwxrwx  1 root root       7 Nov 22 11:31 BACKUP -> ../dm-5
lrwxrwxrwx  1 root root       7 Nov 22 11:31 daten-linux -> ../dm-4

Because Veeam uses dm-4 and dm-5 to identify what to backup, the wrong disk gets backuped.
If veeam would use /dev/mapper/daten-linux instead of /dev/dm5 (which would be fine in the first case but not the second), everything would work like expected. But as db-xy can change after each reboot it just doesnt work.

Here a view of my veeam setup:

Code: Select all

   
Choose volumes to backup                                                 
 ────────────────────────────────────────────────────────────────────────── 
   Name         │                                                           
   Backup mode  │  NAME                       MOUNT POINT                   
 > Volumes      │                                                           
   Destination  │  [] dm-4                   /mnt/BACKUP                  
   Network      │  [+ ] dm-5                   /mnt/Daten_Linux            
   Advanced     │  [\] sda                                              │   
   Schedule     │    [+] sda1                                           │   
   Summary      │    [+] sda2                                           │   
                │    [+] sda3                 /boot                     │   
                │    [ ] sda4                                           │   
                │  [ ] sdb                                              │   
                │  [+] sdc                                              │   
                │    [+] sdc2                 /mnt/Daten_Win              
 ────────────────────────────────────────────────────────────────────────── 
                                                                            
                                               [Prev]   [Next]   [Cancel]   

Any ideas?

[PTide]

Hi,

The described behaviour has been confirmed. Scheduled for improvement in the next release. Please explain how do you open the encrypted devices to device mapper, do you use a keyfile?

Thanks

[ibschreiber]

Hi,
I'm glad you could reproduce it. Any Idea when the next release will be available?

For unlocking I use a two step approach:
To unlock thy system disk, I have to enter a password at boot time.
The other two disks are then unlocked with a file that is on the system disk.
So, you can't get the file to decrypt the other two disks without decrypting the system disk.

If any one reads this and thinks: good idea, I'l do that, too. Save that file to an external media and lock it away in a save place. Otherwise there will be no way to access the data on the second disk without having the file!
I only do this because I don't care. Why? Because I have a backup of all that data and the encryption is only there in case the hardware gets stolen or lost. Also think about saving other luks relevant data as described in different manuals for luks data safety.

[ibschreiber]

as a short workaround I wrote this script.
It's run on each start and checks if the assignment is OK. Depending on the result it enables or disables the backup job. The result is displayed so I know if backup is active or not.
I added veeamconfig to the sudoers config so it can execute without a password...

Code: Select all

#!/bin/bash
ls -la /dev/mapper |grep -q 'BACKUP -> ../dm-4'
if [ $? -eq 0 ]; then
	echo "OK - job enabled"
	sudo /usr/bin/veeamconfig schedule enable --jobId {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}
else
	echo "ERROR - disabling veeam job!!!!!"
	sudo /usr/bin/veeamconfig schedule disable --jobId {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}
fi
sudo /usr/bin/veeamconfig schedule show --jobId {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}

read -p "Press enter to continue"

[ibschreiber]

Hi,
my machine just installed veeam_2.0.0.400_amd64.deb. Is this issue fixed with this version?
Kind regards,
Michael

[PTide]

Hi,

No, unfortunately it's not. By "next release" I meant the one after 2.0, sorry for not being clear enough.

Thanks

[ibschreiber]

Hi,
any news on this? Will it be fixed soon? I still run my workaround script....
Thanks,
Michael

[PTide]

Hi,

We've already fixed the number of issues with the agent trying to pass dm-<N> names instead of normal mapper names to the job manager. Can you confirm that you still observe the issue on 2.0.1.665?

Thanks!

[ibschreiber]

Hi,
I'm sorry for my late response, I missed the mailnotification about your post.
I'll check if the fix is working the next time the machine boots with the wrong order of disks.
Kind Regards,
Michael

[ibschreiber]

Hi,
Just installed V3, broke the backup completely.
Found this in the release notes:
Volume-level backup job relies on a device name under the /dev directory and requires volumes to have
their corresponding device names under the /dev directory staying persistent (e.g. /dev/md-127, /dev/dm-
1). Otherwise the job will back up the wrong volume.

So this bug is still there?

[PTide]

Unfortunately, yes. However, there were some changes - v3.0 allows to select mountpoints for backup. That is, if you select a mountpoint (in UI), then it does not matter what device is mounted there. Please give it a try and let me know if it worked for you.

Thanks!

[ibschreiber]

So it's been over a year and the bug is still there. V3 breaks luks support completely. What is the timeline for this bug to get fixed?
Regards,
Michael

[PTide]

Since the way VAL works with devices goes across the whole product, fixing that bug (technically, it is rather design limitation than a bug) might cause other bugs to appear.

Instead, in order to address the issue of non-persistent names, an option to use a mountpoint as a primary identifier for an object has been introduced in 3.0

As to 3.0 breaking support for LUKS - I will post update in the corresponding thread.

Thanks!

[PTide]

UPDATE:

Fix is ready and can be obtained via support team