Need advise on Veeam setup for LUKS+BTRFS

[crond]

Hi All,

I’m looking for a way to switch to Veeam for BTRFS volume backup and can use some help to figure out correct setup.

Starting point:
* Debian10 ESXi VM as a source
* Data disk is encrypted with LUKS with BTRFS on top and with multiple subvolumes mounted
* BTRBK based scripts runs daily to create RO snapshots of each volume with different (per-subvolume) backup schedule and retention policy.
* Same script replicates snapshots to backup server (Debian10) and has its own per-subvolume retention policy to keep X daily / Y weekly / Z monthly backups. In addition, it replicates some sub-volumes to external(rotating) drivers for storage off-site.
* Everything works as expected.

With Veeam I’d like to keep functionality I have and
1) Add ability to backup to cloud (B2/Wasabi)
2) I don't want to have unencrypted data outside of main VM.

Due to encryption on VM side the vSphere based backup is not practical due to large initial and incremental backups (for example when btrfs volume is rebalanced). Veeam Agent for Linux seems to be the only option, but I'm running into issues.

My initial idea was to create a job for each subvolume. With such approach backup itself seems to work fine, but I’m having trouble with backup-copy and export jobs. Objects in the backup-copy jobs are named based on hostname of the source machine, so I end up with a dozen of objects with exactly the same name.
Is there any way to customize the name on per job basis?

My next thought was to do a full disk backup based on DM disk (after it’s unlocked my LUKS).
I’m still running the first cycle but I see that VAL found btrfs pool and identified all snapshots/subvolumes to be backed up, but apparently it transfers each subvolume and snapshot individually and with COW is disabled. With several hundred snapshots, this is a problem. With ~10% of snapshots processed, the size of initial backup is already way over the size of source physical drive. Is there a way to enable COW ?

[HannesK]

Hello,

Due to encryption on VM side the vSphere based backup is not practical due to large initial and incremental backups (for example when btrfs volume is rebalanced). Veeam Agent for Linux seems to be the only option, but I'm running into issues.

Volume based = block based backup. That means, you will see the same large increments in both cases when you re-balance.

If you want small incremental backups for your kind of setup, then last century backup style is needed: file based backup. In Veeam terms that means NAS backup (faster) or file level based agent backup (slower). If you enable encryption in the jobs, then you don't have unencrypted data outside the VM.

For me it sounds, that you transport physical ideas to a virtual world where most things are not needed anymore. I recommend to simplify if possible.

Best regards,
Hannes

[PTide]

* Data disk is encrypted with LUKS with BTRFS on top and with multiple subvolumes mounted

This requirement makes it tricky - VAL supports BTRFS only if it resides directly on a physical device with no additional abstraction layers (such as LVM, software RAID, dm-crypt and so on) below or above it.

VAL also does not maintain any deduplication - each BTRFS subvolume or a snapshot being backed up results in a separate disk in the backup file (as you have noticed, backup might get large).

As Hannes's suggested, you can try using a file-level backup.

Thanks!

[crond]

Hi Hannes,

Please correct me if I wrong, but VAL Volume based backup can have different objects specifically:
1) Device (block based)
2) mount point (depends on OS in case of BTRFS it's snapshot based)
3) LVM
4) BTRFS (snapshot based)

From what I see (2) does uses BTRFS features and host tools only
I.E VAL creates BTRFS snapshot on the source volume and than copy data over either with "cp -a" or with btrfs send | receive depending on source location. Correct ?

Also looking at what VAL is actually doing, I can see that after initial backup, it in fact does send only diff between previous snapshot and new snapshot
<139688718886656> | Creating child process: btrfs with arguments: send, -p, /tmp/{17d734b0-6e24-44cc-a861-ab448a651dde}/source_root/.veeam_snapshots/{2dd4af3a-3eea-429c-b66d-58aa606f89c6}/2792_Backup, /tmp/{17d734b0-6e24-44cc-a861-ab448a651dde}/source_root/.veeam_snapshots/{2bbc3eb9-0527-4493-a02b-2552725bf566}/2792_Backup

It means that VAL/VBR has capacity/infrastructure to handle parent tracking properly (I.E handling removing of old snapshots and merge of incremental backups to full) .
So VAL can do the same for "foreign" snapshots (not created by VAL) when backing up entire BTRFS pool instead of single mount point. Maybe you can consider it as a "feature" request. I'm also willing to take any experimental patches if such options exists.

If you can also help me with the second part of my questions I would really appreciate that
* Is it possible to customize the name of target object in backup job I.E instead of hostname have "hostname+<custom string>" ?

P.S Finally I don't see how NAS backup can be (even theoretically) faster in case of BTRFS compared to VAL/Volume/mount point based backup, can you please explain ?

Thanks!

[HannesK]

Hello,
as for feature request... PTide is the responsible product manager for that product. He already answered that it is complicated. Well, as I said, I believe your whole setup is too complicated and should be simplified.

P.S Finally I don't see how NAS backup can be (even theoretically) faster in case of BTRFS compared to VAL/Volume/mount point based backup, can you please explain ?

I said that NAS backup is faster than file based VAL backup. Per default VAL is doing block based backup (this was the first google hit that explains the difference https://www.linkedin.com/pulse/file-lev ... mike-wang/). We have our own change block tracking driver that makes incremental backups fast.

Best regards,
Hannes

[PTide]

Also looking at what VAL is actually doing, I can see that after initial backup, it in fact does send only diff between previous snapshot and new snapshot
<139688718886656> | Creating child process: btrfs with arguments: send, -p, /tmp/{17d734b0-6e24-44cc-a861-ab448a651dde}/source_root/.veeam_snapshots/{2dd4af3a-3eea-429c-b66d-58aa606f89c6}/2792_Backup, /tmp/{17d734b0-6e24-44cc-a861-ab448a651dde}/source_root/.veeam_snapshots/{2bbc3eb9-0527-4493-a02b-2552725bf566}/2792_Backup

Right, VAL uses send /receive for BTRFS processing.

So VAL can do the same for "foreign" snapshots (not created by VAL) when backing up entire BTRFS pool instead of single mount point. Maybe you can consider it as a "feature" request. I'm also willing to take any experimental patches if such options exists.

Would elaborate a little bit on the request, please? Do you want to specify a "foreign" snapshot as an object to back up nad keep COW benefits (incremental backups) in place?

* Is it possible to customize the name of target object in backup job I.E instead of hostname have "hostname+<custom string>" ?

Currently it's not possible. You can try creating a separate backup copy job for each source job instead.

Thanks!

[crond]

Would elaborate a little bit on the request, please? Do you want to specify a "foreign" snapshot as an object to back up nad keep COW benefits (incremental backups) in place?

When VAL is backing up complete BTRFS pool, it should be possible to configure it to take "parent" information of subvolumes/snapshots into account, instead of doing a full backup of each subvolume/snapshot. Otherwise the backup in my case is 2+ orders of magnitude bigger than the source block device. This should be straightforward taking into account that Veeam has already implemented snapshot handling/maintenance on the backup-repository side (I.E for /mount_point ).

Currently it's not possible. You can try creating a separate backup copy job for each source job instead.

Right ... That's what I'm currently doing. But it's pain in the neck.
* If I have to export backups they end up with the same names (I.E <hostname>Date ).
* if I have to restore from backup-copy it means that I need to restore each subvolume group individually(based on backup-copy jobs), and then merge them together manually.