Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
LickABrick
Enthusiast
Posts: 58
Liked: 30 times
Joined: Dec 23, 2019 7:26 pm
Full Name: Lick A Brick
Contact:

[GUIDE] Proxmox incremental backups with Veeam

Post by LickABrick » 16 people like this post

Backup up Proxmox using Veeam
Tested on PVE 6.1-8, Kernel 5.3.18-3-pve with Veeam Agent for Linux 1.0.7

In this guide I will go through the process of setting up Veeam to do incremental backups of your Proxmox VM's and CT's.
If in any stage you see me making a mistake, or if you have feedback/suggestions please let me know so I can modify this post :)

Table of contents
• Pros and cons
• Installing Veeam Agent
• Using Veeam Backup and Replication (Community Edition) to backup
• Restoring individual files
• [COMING SOON] Backup with only Veeam Agent (no VBR)
• [COMING SOON] Disaster recovery / Restoring entire VM/CT's

First let's start with some pro's and con's (some are my own opinion):

Pro's
• Easy to setup
• Incremental backup's which saves disk space and decreases backup time
• Great performance
• Veeam is widely used in enterprise environments so it gives you some experience with the product

Con’s
• Restores can take a little time to complete (personally I don’t mind this)
• Restores cannot be done from VBR, but need to be done from the Proxmox host
• Not all file systems are supported (only tested it on LVM-THIN)
• The Linux Agent cannot be ran inside a LXC container
• Not sure how multi node clusters will work (might test this)

Installing Veeam Agent

1. Install the PVE headers (do not forget this step!)

Code: Select all

apt-get install pve-headers
2. Install sudo (not installed by default)

Code: Select all

apt-get install sudo
3. Download the Veeam Agent for Linux and upload it to your Proxmox host (you need to sign up before you can download it)
4. Install the .deb file after which you update your package info.

Code: Select all

dpkg -i veeam-release-deb_1.x.x_amd64.deb && apt-get update
5. Now we can install the Veeam Agent

Code: Select all

apt-get install veeam
6. Check if Veeamsnap is installed correctly with:

Code: Select all

dkms status
Output shoud look like this:
Image

The agent is now installed succesfully.

Using Veeam Backup and Replication (Community Edition) to backup

This requires a Windows machine which preferably runs 24/7
Install VBR (Veeam Backup and Replication) using the documentation, it is a very straightforward next, next, finish type installation.

You can add all kinds of backup repositories, but for most home users it will probably be a NFS/SMB share. In this guide we will use a NFS share.

Adding a backup repository

The backup repository will determine where your backups are being stored.

1. Open VBR and add a new backup repository:
Image
2. Pick 'Network attached storage'
3. Pick 'NFS Share'
4. Give your backup repository a name, in my case I use a old QNAP so I called it ‘QNAP-1’ and click Next
5. Enter the path to your NFS share, in my case I used the following settings:
Image
6. I left further settings as default. Press 'Next' until you can create the repository. After you created the repository you should see the following screen:
Image
7. Press 'Next' --> 'Finish' and you should see the repository in the list.
Image

Add the Proxmox server

We will need to give Veeam access to our Proxmox host, this can be done by adding the server to Veeam.
To do this I recommend creating a new user on the proxmox host.

1. On your Proxmox host create a new user (my user is named sa_veeam) with

Code: Select all

adduser <username>
Enter the required fields:
Image
2. Back in VBR add your server:
Image
3. Enter the IP/hostname of your proxmox host and press Next
Image
4. In the following screen press ‘Manage accounts’, press ‘Add’ and add your newly created credentials. Because we are using a non-root user check all three boxes and enter your proxmox root password, this will be used (once) to give your user the right permissions.
Image
5. Press ‘Apply’ (press ‘Yes’ if it warns about a new ssh fingerprint), if everything went well you should see the following screen:
Image
6. After pressing ‘Finish’ you have successfully added your Proxmox host to VBR.

Creating backup job(s)

Okay, we have now added a backup repository and a Proxmox node to VBR. It is now time to back this baby up!
Before creating your backup job we need know where on our Proxmox node our VM’s / CT’s are stored. You can find this in the resource tab of a CT:
Image

In this case the CT’s / VM’s are stored on ‘LVM-THIN01’. By default this is stored in /dev/LVM-THIN01:
Image
Okay now we know where everything is stored we can create the backup job.

1. In VBR create a new backup job (pick Linux Computer)
Image
2. Give your backup job a name (e.g. ‘Backup Proxmox01 LVM-THIN01’)
4. On the computers screen press ‘Add’ --> 'Individual Computer' and enter your Proxmox host, and the credentials we created earlier:
Image
5. Select ‘Volume level backup’ and press Next
6. Click Add --> LVM
7. Now enter the path to where your VM’s / CT’s are stored.

If you only want to backup a certain VM or CT you can give the full path to the disk (e.g. /dev/LVM-THIN01/vm-100-disk-0) but in most cases I assume you want to backup all your VM’s and CT’s so we can enter the path to where these are stored (/dev/LVM-THIN01 in my case).
Image

8. On the storage page you select the backup repository you it to backup to. The retention policy determine how long the backups will be kept for (e.g. 7 restore points will keep 7 days of backups if you backup once a day)
If you want to have a secondary backup destination (e.g. a second NAS) you can configure it here too.
After picking your desired settings press ‘Advanced’
Image
By default Veeam will create a full backup every first Monday of the month, I recommend keeping it this way

9. I recommend enabling backup file encryption for a more secure backup. Press ‘Add’ and enter a secure password (MAKE SURE TO SAVE THIS PASSWORD SOMEWHERE SAVE!!) after this press OK
10. Press Next until you are on the Schedule page. Here you can determine your backup schedule. I set mine as following (backing up every night at 01:00). After this press ‘Apply’:
Image
11. Select ‘Run the job when I click Finish’ and click Finish

Your backup job will now start it’s first run. You can check the status by clicking on the backup job:
Image

It can take a little while before the job starts transferring data to your backup repository so you might need to be a little patient.
After the job is finished you will see the following output:
Image

Nice, our first backup has been run successfully!

Backing up PVE Container and Virtual machine configs

The first backup job is fine if we want to restore files located inside our VM / CT, but if we want to restore a complete VM / CT we will also need to have the Proxmox config file. As these are stored in separate files on the Proxmox host we will need to create a new backup job which will back these files up.
(these config files are probably only needed if you have manually deleted them, or if your complete Proxmox host has been corrupted/crashed)

1. Create another Linux Computer backup job, we name it ‘Proxmox01 CT and VM config backup’.
2. Enter the Proxmox host details again.
3. Pick ‘File level backup’ this time and check the ‘Backup directly from…..)
Image
4. Add the folders containing the PVE configs:
Image
5. This time we let this job run after our CT + VM backup has been completed:
Image
6. Press ‘Finish’ and let the job run


You are now successfully backing up your VM’s, Containers and their PVE config file.

Restoring individual files

Now onto the part we hope we never (or almost never) have to use… restoring files.
Sadly we can not restore files from within VBR (as supposed to when you backup vSphere / Hyper-V).
Restoring has to be done from the Proxmox host. Here is how you can do it:

1. On your Proxmox host execute the following command:

Code: Select all

veeam
2. Press 'R'
(Loading might take some time)
3. Select the job you want to restore from and press TAB to select the restore point (in this example we only have one). After selecting the restore point press ENTER
Image
4. It will now (after loading) tell you the backup has been mounted to ‘/mnt/backup’:
Image
5. You can now (via CLI or my preferred method is WinSCP) restore your files:
Image
6. When your done, open up veeam again by entering

Code: Select all

veeam 
and press ‘U’ to unmount the backup.
Image
LuCloudo
Lurker
Posts: 2
Liked: 2 times
Joined: Mar 11, 2019 10:57 am
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by LuCloudo »

Hats off to LickABrick, nice work!

I tested the same logic with KVM (qemu-kvm-ev) on generic CentOS 7 with local LVM (highly likely thick) a while back and it worked as expected, but unfortunately I haven't got a chance to do this in our production environment as the VMs are running off thick LVM shared via iSCSI without a proper clustering mechanism (Proxmox and other KVM implementations), which seems to be quite common among Linux admins but I personally wouldn't do this.

Given you use thin LVM as per your screenshot, so I guess it's local LVM.

I'm also quite curious if anyone's ever tried this implementation against shared LVM via iSCSI before on Proxmox or any other KVM platforms.

As you've pointed out, this approach is only suitable for VMs running off LVM, but, it also seems to be quite prevalent.

Another thing to try would be making Veeam call a script to do quiecsing via qemu guest agent.

Incremental backup at the host level is essential to our environment, but unfortunately not many KVM platforms I've come across have been able to do this yet.
O1982
Influencer
Posts: 15
Liked: 2 times
Joined: Dec 14, 2017 12:56 pm
Full Name: Owen Thompson
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by O1982 »

Thank you very much for taking the time to post this. It's excellent info and I will try it out soon :D

THANKS!
LickABrick
Enthusiast
Posts: 58
Liked: 30 times
Joined: Dec 23, 2019 7:26 pm
Full Name: Lick A Brick
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by LickABrick »

LuCloudo wrote: May 11, 2020 1:56 am Another thing to try would be making Veeam call a script to do quiecsing via qemu guest agent.
Thanks! Does veeam not already make sure the backups are in a consistent state? https://helpcenter.veeam.com/docs/agent ... tml?ver=40

What would the advantage be?
StefanZ
Veteran
Posts: 369
Liked: 98 times
Joined: Mar 04, 2019 10:31 am
Full Name: Stefan Zimmermann
Location: Germany
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by StefanZ »

Great work LickABrick, thanks for sharing this with the community.

About the consistency discussion: veeamsnap can create consistency by taking a snapshot on the host's file systems. This will work for containers because "guest" and host are sharing the same kernel I/O scheduler. For a VM however, you have higher isolation and you need to work with the VM to achieve 100% consistency - and that's it what guest agent quiescence could achieve here.
But for a VM you could also consider installing the Veeam agent within the VM of course.

I am not 100% how Proxmox handles shared LVMs. If this is a active/passive thing, then this should be possible with the Veeam agent, however, veeamsnap does lock the devices and that can lead into situations where failover is not possible anymore - so you might consider not using veeamsnap in that case and achieve consistency via scripts only. Happy to see if someone tests this.
Kind regards, Stefan
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by PTide »

Great job!

Just keep in mind that Proxmox distro is not listed on VAL user guide and therefore is not officially supported.
Does veeam not already make sure the backups are in a consistent state?
In addition to what Stefan has said - VAL also flushes dirty pages from RAM before a snapshot is taken so that filesystem of the host is consistent. However, it hasn't been tested yet if that operation also affects guests' disks. That is, it would be safer to instruct guests to prepare themselves before a snapshot is taken.

Thanks!
Ejdesgaard
Enthusiast
Posts: 43
Liked: 8 times
Joined: Aug 24, 2012 11:59 am
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by Ejdesgaard »

@PTide
Just keep in mind that Proxmox distro is not listed on VAL user guide and therefore is not officially supported.
According to https://helpcenter.veeam.com/docs/agent ... tml?ver=40 Debian-8.0 -> 10.3 is supported.

There are 2 ways of installing proxmox:
#1 - the bundled, almost-one-click-install of Proxmox VE, that you, on it's surface could call a distribution of its own, but underneath it's a stable debian release + proxmox repositores.
#2 - Clean Debian stable install, followed by adding the Proxmox VE repositories and installing the proxmox-ve related packages.

I would argue that _at least_ #2 is officially supported since it's just a Debian stable + "some software"
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by PTide »

Hi Ejdesgaard,
#2 - Clean Debian stable install, followed by adding the Proxmox VE repositories and installing the proxmox-ve related packages.
I would argue that _at least_ #2 is officially supported since it's just a Debian stable + "some software"
As far as I know (please correct me if I am wrong), proxmox-ve package (which basically constitutes the whole Proxmox thing) depends on pve-kernel. We haven't tested veeamsnap with PVE kernel yet, hence we cannot blindly claim that it is supported.

Thanks!
sbou
Novice
Posts: 3
Liked: never
Joined: Aug 07, 2019 6:40 pm
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by sbou »

@jdesgaard @PTide

You are right, Proxmox userland is debian stable, however it's kernel is based on Ubuntu + patches
https://git.proxmox.com/?p=pve-kernel.git;a=summary
optyfr
Lurker
Posts: 2
Liked: never
Joined: Sep 21, 2021 11:48 am
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by optyfr »

It worked flawlessly until I switched to Promox 7 which is using Kernel 5.11 ...
LickABrick
Enthusiast
Posts: 58
Liked: 30 times
Joined: Dec 23, 2019 7:26 pm
Full Name: Lick A Brick
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by LickABrick »

@optyfr not sure what errors you get, but this: post367276.html might help:
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by PTide » 1 person likes this post

It won't help. If you want it to work on 5.11 you need to either try this experimental branch, or wait until we release the updated version (soon).

Thanks!
optyfr
Lurker
Posts: 2
Liked: never
Joined: Sep 21, 2021 11:48 am
Contact:

Re: [GUIDE] Proxmox incremental backups with Veeam

Post by optyfr »

Now switched on veeamsnap 5.0.2 and veeam 11
Unfortunately it hangs indefinitely the proxmox server when starting volume snapshot
Kernel is 5.13, the official latest proxmox one, and system.map is present
No message on tty, all io are blocked
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests