Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi,
I have 3 disks in my machine and on each disk are encrypted volumes.

Linux (Ubuntu 16.04 LTS) is mounting these disks in an unpredictable order (ls of /dev/mapper):

Code: Select all

lrwxrwxrwx  1 root root       7 Nov 20 14:54 BACKUP -> ../dm-4
crw-------  1 root root 10, 236 Nov 20 14:54 control
lrwxrwxrwx  1 root root       7 Nov 20 14:54 daten-linux -> ../dm-5
lrwxrwxrwx  1 root root       7 Nov 20 14:54 lukslvm -> ../dm-0
lrwxrwxrwx  1 root root       7 Nov 20 14:54 vgubuntu-Daten_LinuxSSD -> ../dm-3
lrwxrwxrwx  1 root root       7 Nov 20 14:54 vgubuntu-root -> ../dm-2
lrwxrwxrwx  1 root root       7 Nov 20 14:54 vgubuntu-swap -> ../dm-1
sometimes, BACKUP is mapped to dm-4, sometimes it is dm-5.
With daten-linux it's the other way around.
Veeam Agent doesn't use the entries in /dev/mapper but the /dev/dm-* to describe what to backup and to map this.
The result is that when the assigned dm-? changes and veeam takes a backup, it backs up the whole disks as everything changed.
Any idea how to prevent this?
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

Hi,

It's not quite clear what schema do you have on your disks. Do you have LUKs over LVM? Does your VG occupy all three disks? Are BACKUP, daten-linux, lukslvm the encrypted partitions? Please clarify.

Thanks
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi, I'll try to clarify things:
There are three disks:
sda sdb sdc
on each disk, there is a crypt-luks partition

The luks on sda shows up as a device /dev/dm-0
on this device is a lvm group called vgubuntu with three volumes
vgubuntu-Daten_LinuxSSD -> ../dm-3
vgubuntu-root -> ../dm-2
vgubuntu-swap -> ../dm-1

the luks partition on sdb and sdc gets dm-4 and dm-5 assigned. But, there is no way to tell which one gets which after booting the machine.

Here another view from lsblk:

Code: Select all

NAME                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdb                             8:16   0   1,8T  0 disk  
└─sdb1                          8:17   0   1,8T  0 part  
  └─BACKUP                    253:4    0   1,8T  0 crypt /mnt/BACKUP
sdc                             8:32   0 931,5G  0 disk  
├─sdc2                          8:34   0 247,8G  0 part  /mnt/Daten_Win
└─sdc1                          8:33   0 683,7G  0 part  
  └─daten-linux               253:5    0 683,7G  0 crypt /mnt/Daten_Linux
sda                             8:0    0 256,2G  0 disk  
├─sda2                          8:2    0  97,8G  0 part  
├─sda5                          8:5    0 156,3G  0 part  
│ └─lukslvm                   253:0    0 156,3G  0 crypt 
│   ├─vgubuntu-swap           253:1    0    24G  0 lvm   [SWAP]
│   ├─vgubuntu-root           253:2    0    64G  0 lvm   /
│   └─vgubuntu-Daten_LinuxSSD 253:3    0  68,3G  0 lvm   /mnt/Daten_LinuxSSD
├─sda3                          8:3    0     2G  0 part  /boot
└─sda1                          8:1    0   100M  0 part  
The problem is this: one time after boot, /dev/maper/ looks like this:

Code: Select all

lrwxrwxrwx  1 root root       7 Nov 22 11:31 BACKUP -> ../dm-4
lrwxrwxrwx  1 root root       7 Nov 22 11:31 daten-linux -> ../dm-5
the next time it looks like this:

Code: Select all

lrwxrwxrwx  1 root root       7 Nov 22 11:31 BACKUP -> ../dm-5
lrwxrwxrwx  1 root root       7 Nov 22 11:31 daten-linux -> ../dm-4
Because Veeam uses dm-4 and dm-5 to identify what to backup, the wrong disk gets backuped.
If veeam would use /dev/mapper/daten-linux instead of /dev/dm5 (which would be fine in the first case but not the second), everything would work like expected. But as db-xy can change after each reboot it just doesnt work.

Here a view of my veeam setup:

Code: Select all

   
Choose volumes to backup                                                 
 ────────────────────────────────────────────────────────────────────────── 
   Name         │                                                           
   Backup mode  │  NAME                       MOUNT POINT                   
 > Volumes      │                                                           
   Destination  │  [] dm-4                   /mnt/BACKUP                  
   Network      │  [+ ] dm-5                   /mnt/Daten_Linux            
   Advanced     │  [\] sda                                              │   
   Schedule     │    [+] sda1                                           │   
   Summary      │    [+] sda2                                           │   
                │    [+] sda3                 /boot                     │   
                │    [ ] sda4                                           │   
                │  [ ] sdb                                              │   
                │  [+] sdc                                              │   
                │    [+] sdc2                 /mnt/Daten_Win              
 ────────────────────────────────────────────────────────────────────────── 
                                                                            
                                               [Prev]   [Next]   [Cancel]   
Any ideas?
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

Hi,

The described behaviour has been confirmed. Scheduled for improvement in the next release. Please explain how do you open the encrypted devices to device mapper, do you use a keyfile?

Thanks
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi,
I'm glad you could reproduce it. Any Idea when the next release will be available?

For unlocking I use a two step approach:
To unlock thy system disk, I have to enter a password at boot time.
The other two disks are then unlocked with a file that is on the system disk.
So, you can't get the file to decrypt the other two disks without decrypting the system disk.

If any one reads this and thinks: good idea, I'l do that, too. Save that file to an external media and lock it away in a save place. Otherwise there will be no way to access the data on the second disk without having the file!
I only do this because I don't care. Why? Because I have a backup of all that data and the encryption is only there in case the hardware gets stolen or lost. Also think about saving other luks relevant data as described in different manuals for luks data safety.
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

as a short workaround I wrote this script.
It's run on each start and checks if the assignment is OK. Depending on the result it enables or disables the backup job. The result is displayed so I know if backup is active or not.
I added veeamconfig to the sudoers config so it can execute without a password...

Code: Select all

#!/bin/bash
ls -la /dev/mapper |grep -q 'BACKUP -> ../dm-4'
if [ $? -eq 0 ]; then
	echo "OK - job enabled"
	sudo /usr/bin/veeamconfig schedule enable --jobId {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}
else
	echo "ERROR - disabling veeam job!!!!!"
	sudo /usr/bin/veeamconfig schedule disable --jobId {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}
fi
sudo /usr/bin/veeamconfig schedule show --jobId {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}

read -p "Press enter to continue"
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi,
my machine just installed veeam_2.0.0.400_amd64.deb. Is this issue fixed with this version?
Kind regards,
Michael
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

Hi,

No, unfortunately it's not. By "next release" I meant the one after 2.0, sorry for not being clear enough.

Thanks
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi,
any news on this? Will it be fixed soon? I still run my workaround script....
Thanks,
Michael
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

Hi,

We've already fixed the number of issues with the agent trying to pass dm-<N> names instead of normal mapper names to the job manager. Can you confirm that you still observe the issue on 2.0.1.665?

Thanks!
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi,
I'm sorry for my late response, I missed the mailnotification about your post.
I'll check if the fix is working the next time the machine boots with the wrong order of disks.
Kind Regards,
Michael
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

Hi,
Just installed V3, broke the backup completely.
Found this in the release notes:
Volume-level backup job relies on a device name under the /dev directory and requires volumes to have
their corresponding device names under the /dev directory staying persistent (e.g. /dev/md-127, /dev/dm-
1). Otherwise the job will back up the wrong volume.

So this bug is still there?
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

Unfortunately, yes. However, there were some changes - v3.0 allows to select mountpoints for backup. That is, if you select a mountpoint (in UI), then it does not matter what device is mounted there. Please give it a try and let me know if it worked for you.

Thanks!
ibschreiber
Influencer
Posts: 22
Liked: 4 times
Joined: Nov 21, 2017 11:51 am
Full Name: Michael Schreiber
Contact:

Re: Veam mixes up multiple encrypted disks

Post by ibschreiber »

So it's been over a year and the bug is still there. V3 breaks luks support completely. What is the timeline for this bug to get fixed?
Regards,
Michael
Michael Schreiber
CEO Creatronics GmbH - Greding - Germany
Veeam user and VCSP
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

Since the way VAL works with devices goes across the whole product, fixing that bug (technically, it is rather design limitation than a bug) might cause other bugs to appear.

Instead, in order to address the issue of non-persistent names, an option to use a mountpoint as a primary identifier for an object has been introduced in 3.0

As to 3.0 breaking support for LUKS - I will post update in the corresponding thread.

Thanks!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veam mixes up multiple encrypted disks

Post by PTide »

UPDATE:

Fix is ready and can be obtained via support team
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests