Backup Linux Cloud VPS

[jeffshead]

Can Veeam be used to pull backups from Linux VPS's, in the cloud, and push restores to the same cloud servers?

If so, how can this be done without opening and forwarding ports on the local network where Veeam is installed? Can Veeam B&R initiate (pull) the backup via SSH from the cloud VPS's?

Can the Bootable recovery media be installed and booted from a partition on the cloud VPS's if the VPS provider does not allow custom ISO's to be mounted?

[aj_potc]

Yes, you can use Veeam to pull backups from cloud servers. The VBR server orchestrates the backup, but the actual backup data flows between the cloud server and the target repository, which can be on the same network as the system to be backed up.

The VBR connects to the client and starts the backup process via SSH, so you don't need to open any additional incoming ports on the client side. The SSH credentials/key are stored by the VBR server.

As for bootable media, no, you don't have to use the bootable ISO provided by Veeam. You can boot into a different recovery environment offered by your provider, install Veeam agent for Linux, and start the Veeam restoration from there. I've tested this procedure successfully. Below are my notes for doing this using an RHEL-based system. For Debian or Ubuntu, it's a similar process.

If not possible to boot from Veeam boot ISO:
Boot a different LiveCD, then install Veeam agent:
Download and activate Veeam repo information.

Code: Select all

yum install http://repository.veeam.com/backup/linux/agent/rpm/el/8/x86_64/veeam-release-el8-1.0.7-1.x86_64.rpm

Install the Veeam package

Code: Select all

yum install veeam

Edit /etc/veeam/veeam.ini to add the following so that it can perform bare metal recovery:

Code: Select all

[recoveryui]
enableOnLiveSystem = 1

Restart Veeam service

Code: Select all

systemctl restart veeamservice

Start Veeam agent in recovery mode:

Code: Select all

veeamconfig recoveryui

[jeffshead]

Thanks for the info!

Unfortunately, I can't get it to work. Are there any ports, other than port 22, that need to be opened on the client?

Most of my VPS's are AlmaLinux 8.7. Not sure if that's the reason but I get the following error: Skipping example.com: operating system is not supported.

I've tried both IP address and domain name of the server when creating the backup job. I also tried editing the /etc/os-release file as some have suggested.

hostnamectl:

Code: Select all

Static hostname: example.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 2fe7e0fsdg8c451aa32109b4ffca83c3
           Boot ID: ac39db85d1d359d92261f99b057f38b
    Virtualization: kvm
  Operating System: AlmaLinux 8.7 (Stone Smilodon)
       CPE OS Name: cpe:/o:almalinux:almalinux:8::baseos
            Kernel: Linux 4.18.0-425.13.1.el8_7.x86_64
      Architecture: x86-64

[jeffshead]

After opening port 6160 on the VPS, Veeam seems to get a little further.



3/26/2023 10:06:49 PM Failed Connecting to Veeam Installer service on server Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond x.x.x.x:6160 0:01:25

I also opened port 6162 on the VPS but that did not seem to help either.

[jeffshead]

The VBR connects to the client and starts the backup process via SSH, so you don't need to open any additional incoming ports on the client side...

Not sure what's different between my setup and yours but I did more testing and it seems port 22 is not the only port which needs to be opened. Any ideas?

Since opening ports 6160 and 6162, I see that the transport was installed on the remote VPS. So far, I've detected attempted traffic on port 6160, 6162 and 100006. I'll wager some dynamic ports are also needed.

It's looking like a VPN connection is needed unless I open up a bunch of ports.

[Mildur]

Hi Jeff

Correct.
NAT is not supported for our Veeam Agents.
Either use VPN or Veeam Cloud Connect which works without NAT. Cloud Connect is for Service Provider to provide backup and replication services to their customers. Enterprise organization may also get access to VCC after approval by Veeam.

https://helpcenter.veeam.com/docs/agent ... 60#network

If you back up to a repository managed by a Veeam backup server, Veeam Agent for Linux must be able to establish a direct IP connection to the Veeam Backup & Replication server. Veeam Agent for Linux cannot work with Veeam Backup & Replication that is located behind the NAT gateway.

May I know, have you complete root access to those VPS VMs?
If not, Installing our full Agent with the veeamsnap module will be an issue. I remember those VPS where you had root permission to install all sort of software, but non which require write access to kernel modules.
In that case, you may try to use our veeam-nosnap agent:
https://helpcenter.veeam.com/docs/agent ... tml?ver=60


Best,
Fabian

[jeffshead]

May I know, have you complete root access to those VPS VMs?

Yes, as far as I know, I do.

I will have to rethink this. I have half a dozen Linux servers for which I want to create monthly, bare metal restore images. I'm not so sure that I can use a VPN during the restore process because of the limited feature set made available by the VPS provider.

[aj_potc]

Not sure what's different between my setup and yours but I did more testing and it seems port 22 is not the only port which needs to be opened. Any ideas?

On the client to be backed up (that is, where you have Veeam Agent for Linux installed), you don't need any other incoming ports opened. I'm sure about that, because I'm backing up firewalled systems that have only port 22 open.

However, the client to be backed up does need to be able to talk to the Windows server where you have the VBR installed. Like Fabian mentioned, it doesn't work if the VBR is behind a network using NAT. It's a very unfortunate limitation.

It's looking like a VPN connection is needed unless I open up a bunch of ports.

Opening ports alone won't fix this. You can connect the VBR to your cloud network via a VPN, as you suggested. I've found this to be rather complex, and have never been successful in getting it to work. I suppose if your hosting provider offered it as a feature (e.g., they have their own VPN endpoint set up to allow you to connect), then it may be easier.

Another solution is to install the VBR on a Windows VM inside the cloud provider's network. This is probably what Veeam would recommend.

Alternatively, you can install the VBR on an external network with a publicly routable IP and set up a firewall that allows access only from your clients. Veeam recommends not putting VBR servers on the public Internet, however.

Regardless of the location of the VBR server, your Veeam repositories can be external. So if your goal is to make bare metal backups of the VMs and ship those elsewhere, then this should work. You just have to make sure the clients to be backed up can contact the VBR server.

[jeffshead]

So I set up a VPN connection and ran a test backup which failed.

I got these errors with the veeam agent installed:

Code: Select all

Failed to create volume snapshot
Failed to load module [veeamsnap].
Failed to perform managed backup
Failed to take volume snapshot

I get the following errors with the veeam-nosnap agent installed:

Code: Select all

Insufficient free space for a snapshot in a volume group: almalinux.
Failed to perform managed backup
Failed to take volume snapshot
Failed to create volume snapshot

[Mildur]

Almalinux is not supported by veeam. There are workarounds to make it work. But no official support:
post480832.html#p480832

Best,
Fabian

[aj_potc]

There may be a couple of issues here.

In my experience, the veeamsnap module's loader is expecting to see "rhel" as the ID field in /etc/os-release. So that's one thing to check.

The workaround that Fabian linked involves a situation in which the kernel version you're running doesn't match the kernel version supported by veeamsnap. In that case, you can force the module to load anyway, as explained here:

https://docs.serversaustralia.com.au/do ... lmalinux-8

Alternatively, you can try to boot into an earlier kernel version that is supported by Veeam, e.g.:
4.18.0-425.10.1.el8_7.x86_64

[jeffshead]

Thanks guys!

In my experience, the veeamsnap module's loader is expecting to see "rhel" as the ID field in /etc/os-release. So that's one thing to check.

So I changed the ID to "rhel", uninstalled the veeam-nosnap agent and reinstalled the regular veeam agent and the backup completed in 7 minutes without error!

I will setup the VPN connections on the other remote VPS's, perform backups and restore one and report back.

Next, I will try to find or create pre/post backup scripts that can connect/disconnect the VPN instead of doing it all manually.

[aj_potc]

Glad I could help!

Out of curiosity, what software are you using to set up the VPN? I never found an elegant way to get this to work, but have a feeling it might be possible using Wireguard or something similar.

[jeffshead]

what software are you using to set up the VPN

I installed the basic OpenVPN v2 client on the VPS's. Definitely not an elegant solution but it seems to work for the backups. I still have to test a restore.

[jeffshead]

This is really disappointing... Can't perform a restore using the VPS provider's Rescue option (LiveCD). Runs out of free space when installing Veeam.

Is there a way to perform a bare metal restore if you do not have the option of using a custom LiveCD? Most of these VPS's have only 1GB RAM. A couple have 3GB.

[jeffshead]

The VPS provider is willing to upload the Veeam ISO. If I boot from it, will I be able to install the OpenVPN client as I can with their Rescue LiveCD?

[PTide]

Hi,

If you upload any custom liveISO of your choice (e.g. Ubuntu or Fedora), you will be able to install pretty much anything you want in it (including Veeam without the snapshot driver, which will allow you to start recovery environment).

If you would like to upload Veeam recovery ISO, then it might get tricky as the ISO is stripped to the bare-minimum required and is missing lots of stuff that is neede for functions other than Veeam-related restore tasks.

I am currently looking into ways to modify grub in such way that it will boot from veeam ISO located on a local drive automatically, not sure about ETA though (the ISO is missing casper). Anyway, I noted it as a feature request so that in one of the next releases we will add a "button" to patch the system so that it boots into Veeam recovery automatically.

Thanks!

[aj_potc]

The VPS provider is willing to upload the Veeam ISO. If I boot from it, will I be able to install the OpenVPN client as I can with their Rescue LiveCD?

I experimented with the Veeam ISO back in version 5, and I recall it's based on Debian. I was able to install some extra packages I needed like rsync with a simple apt-get install command, so it's worth a try.

I don't know if the OpenVPN client you need is in the repos set up in the Veeam ISO, but this may be something you could configure. (In the RHEL world, it would be pretty easy to add an extra repo; I'm not so sure about Debian, but I would imagine it's similar.)

[ITP-Stan]

I'm doing the same thing for a home-lab.
Back-up a Cloud VPS running Linux (Rocky 9) to on-premise VBR.
I'm using OpenVPN server on the VBR and have installed OpenVPN client on the VPS.
I've configured a start-up script to automatically start the VPN client connection.

After lots of problems when running updates and having an incompatible kernel version for the snap module I chose another path.
I changed my VPS to use LVM for disk layout and keep a certain amount of free space on the VG so I can use the nosnap agent.
Now I can run updates without worrying it will break my back-ups.

I have not really tested bare metal restore options.

[jeffshead]

@ITP-Stan - Have you tried a bare metal restore, yet?

I'm still trying to come up with a workable routine that can be used on all of my VPS's.

[aj_potc]

You mentioned previously that you were unable to install Veeam using the provider's recovery ISO due to running out of space (I assume RAM). Is this still the case? I still think that would be your best bet, as you can install Veeam using my instructions and launch a recovery. No need to use Veeam's own ISO for that.

Alternatively, did you have any luck installing packages while booting from the Veeam-provided recovery ISO?

[jeffshead]

You mentioned previously that you were unable to install Veeam using the provider's recovery ISO due to running out of space (I assume RAM). Is this still the case?

Yes. Some VPS's have only 1GB of RAM.

[ITP-Stan]

@ITP-Stan - Have you tried a bare metal restore, yet?

I have not performed a bare metal restore on the VPS provider yet.
I have done a baremetal restore to on on-premise vmware virtual machine, that worked with some minor issues like networking.

In my case it's homelab use, not production use.

[aj_potc]

Yes. Some VPS's have only 1GB of RAM.

I think you'll have trouble getting around this. The RHEL 8/9 minimum requirements are 1.5 GB, and I suppose the Veeam LiveISO needs something similar.

Which Veeam recovery media did you try booting from:

https://repository.veeam.com/backup/lin ... media/x64/

A bare metal recovery with Veeam is absolutely possible. I've done it several times in testing and as a method to migrate a VPS to another provider, but never through an OpenVPN tunnel like you're trying.

[PTide]

Just a reminder that if something does not work as expected with our default recovery media, you can always use any liveCD of your choice and install Veeam Agent on it, and enable Recovery Environment (see bullet #2).

Thanks!

[aj_potc]

I suggested those steps in my initial reply above, but I think his trouble has more to do with very limited RAM, which makes it a challenge to load everything he needs in addition to the Veeam recovery console.