Build an immutable backup repository - article series

[Regnor]

You're absolutely right Anton. While I was thinking about in-guest iSCSI, it doesn't change much about the fact that you can attack it via the hypervisor.
Physical is the way to go if you want to do it correct.

[albertwt]

Yes, It makes perfect sense to create the immutable backup repository on a physical machine or to disconnect the management network.

Thanks, @Gostev and @Regnor for the input.

[Ciso_2021]

A question
We are trying to archief this, I have a question can we share the repo between two offices ( multiple jobs )?
Most of the jobs gonna need to be copy job or it gonna be a wast of the hardware ?

[Gustav]

@Ciso_2021: Yes - and, apart from the (probably) increased storage volume requirement, that would probably save hardware.

[Ciso_2021]

Unfortunately just discovered today it’s not possible to have 3 offices veeam copy theirs jobs to the remote Ubuntu server.

[Gustav]

Instead of copying, try direct backup jobs.

[cgsm]

Hello,

I just followed this guide completely and came across and error. Veeam recommends NOT installing the Agent, as noted in Part 6, on the repo server. Doing so results in errors in VBR where you cannot edit the repo in any manner (change name, description, etc.).

Per Veeam support:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

Veeam hardened repositories do not use root, while some VAL components do.

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened, and there is a possibility that it may not function at all.

Errors in VBR when the Agent is installed on the repo:
- Enabling restricted mode for Installer Error: VAL components are installed on the target machine VAL components are installed on the target machine.
- Failed to save Backup Repository: VAL components are installed on the target machine.
- Infrastructure item save failed Error: VAL components are installed on the target machine

[einhirn]

Hi,

Per Veeam support:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

I understand how this could be a problem with a managed backup agent, which seems to be the case described here. Does this also apply to a standalone/unmanaged install backing up to a different storage via SMB?

[Gustav]

I just followed this guide completely and came across and error. Veeam recommends NOT installing the Agent, as noted in Part 6, on the repo server. Doing so results in errors in VBR where you cannot edit the repo in any manner (change name, description, etc.).

I don't know about that. To me, the natural choice was and is Veeam, indeed as it has ran with zero issues since the articles were written and can be monitored from your Veeam console including notifications if installed as to my guide - also offering boot media and bare metal recovery.

You are, of course, free to backup the Linux server using any other method as to your preferences, though this will for sure not be supported by Veeam.

[@IT_Guru]

Hello,

I just followed this guide completely and came across and error. Veeam recommends NOT installing the Agent, as noted in Part 6, on the repo server. Doing so results in errors in VBR where you cannot edit the repo in any manner (change name, description, etc.).

Per Veeam support:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

Veeam hardened repositories do not use root, while some VAL components do.

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened, and there is a possibility that it may not function at all.

Errors in VBR when the Agent is installed on the repo:
- Enabling restricted mode for Installer Error: VAL components are installed on the target machine VAL components are installed on the target machine.
- Failed to save Backup Repository: VAL components are installed on the target machine.
- Infrastructure item save failed Error: VAL components are installed on the target machine

We are experiencing the same issue "Failed to save Backup Repository: VAL components are installed on the target machine". Can you provide the official support statement that Veeam Support is making about "Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.".

If this statement is true then using Veeam to back up the operating system using the linux agent isn't really possible.

[cgsm]

This is the exact message I received from Veeam in regards to my case 05919113:

After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

Veeam hardened repositories do not use root, while some VAL components do.

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened, and there is a possibility that it may not function at all.

[Gustav]

@cgsm:
Could it be, that you have attempted to install the Linux agent remotely - from the Veeam B&R console? That won't work.
I think that is what the response you received from Veeam is about.

Veeam must be installed "physically" on the Linux machine (requires sudo), and - apart from the repository - it must be untouched by Veeam.

My first attempt was, of course, to install the Linux agent from the console.
I recall, that I never made it work - I even had to reinstall the Linux machine to remove all traces of Veeam, then install the repository, and later (on the Linux machine) install Veeam for Linux as described.
This way, Veeam on the Linux machine runs isolated and, thus, uncontrolled from your normal Veeam backup environment. Any change of the backup plan must be done on the Linux machine.

[Ciso_2021]

Instead of copying, try direct backup jobs.

Unfortunately this is not possible as some offices are using slow internet like 100/20MB.
Is there any other way of doing this ? Cannot have a physical server for each office.

[mkaec]

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened...

I'm puzzled by this. My experience with the agent (in free mode) is that the backup server can't ask it to do things (so it wouldn't need to be listening on ports). The Agent does backups and then sends them over to the repository. If I open the restore wizard, I get an error that I cannot proceed with single use credentials.

I was never thrilled with the plan of just crossing my fingers and hoping that nothing bad happened to the hardened repository's system volume. It would be really nice if a supported way of using VAL with a hardened repository could emerge.

[doktornotor]

Not really sure how's the hardened repo's system volume related here? What kind of maintenance are you planning to do there via Agent/VBR? You either use physical console or some dedicated KVM VLAN (iLO, iDRAC, IRMC) to manage the machine where the hardened repo is located. Not Veeam.

[mkaec]

If the OS volume dies, that takes the repo offline. Being able to restore using a VAL backup would be a lot easier than rebuilding from scratch.

[doktornotor]

Really? Reinstalling very much vanilla machine with nothing but the OS running from ISO takes about, hmmm... 30 minutes (or even less with something like kickstart).

[Gustav]

Unfortunately this is not possible as some offices are using slow internet like 100/20MB.
Is there any other way of doing this ? Cannot have a physical server for each office.

You may check out the WAN accelerator that is enabled with a paid license (and the trial, of course). It can optimise things, though not do magic.

We have a similar line between two locations, but we didn't even consider running Veeam across that line, indeed not as Veeam runs on very modest hardware.
So, we picked an old trusted HP machine and dedicated that to Veeam at the remote location and another, even older Lenovo (now 15 years old!), for the Linux repository - this is an area where you can reuse machines out of warranty, as you easily can replace them should they fail.

[mkaec]

Really? Reinstalling very much vanilla machine with nothing but the OS running from ISO takes about, hmmm... 30 minutes (or even less with something like kickstart).

Yes. Restoring from backup would be much easier and save a ton of time. There's a reason the first post of this thread has a 9 part guide with 180 screen shots showing how to properly do the build. (Thanks, Gustav! Awesome work!)

[mkaec]

So, I ran into a complication from having VAL on the repository. If I try to edit the repository in B&R, I get errors:

- Enabling restricted mode for Installer Error: VAL components are installed on the target machine
- Failed to save Backup Repository: VAL components are installed on the target machine
- Infrastructure item save failed Error: VAL components are installed on the target machine

To me, this is actually a good thing. One possible line of attack is for an attacker to edit the repository and change the immutable period to 0, then wait for all the immutable backups to age out. With VAL on the repository, this isn't possible. In a weird way, it's an extra line of protection. Once a repository is set up, there's not really a need to edit it. I only tried editing it today to show something to a colleague. If I do need to really edit the repository, I'm fine uninstalling VAL first.

I did a port scan of the repository before and after VAL was installed, and the same ports were listening (6160 & 6162). So, it doesn't appear that VAL has opened up any dedicated ports. And backups have been running ok. So for me, at this point VAL appears to be a net gain.

[ratkinsonuk]

Having spent a number of weeks working with support to get our VHR server and infrastructure set up, I've just come across this 'VAL components are installed on the target machine' error. It looks like I'm going to have to back-track and rebuild the VAL backup into a self-contained mode again.

I want to say out how disappointing this is as a long-term Veeam customer. We rely on the different tiers within the Veeam infrastructure to allow us to manage and monitor the backups. Veeam spends a vast amount of it's resources giving us tools like SureBackup to make sure we're kept safe and everything is reliable.

Veeam Hardened Repository has then come along, which is a great tool to incorporate into software what we'd normally have to do with expensive storage devices, namely security and immutability. Unfortunately we're then told, 'Don't worry about backing up your VHR server, just rebuild it from scratch'.

In our case, that would take a long time given that it's not a simple Linux configuration. And all the time we're rebuilding the server, we can't get on with the task of restoring the rest of our infrastructure that's potentially been wiped, encrypted or broken. What if I went out to Veeam users and said, don't worry about backing up your servers using Veeam, just rebuild them from scratch if there's a problem. I think I'd soon get booted out.

It doesn't take much thought to come up with plenty of reasons why it's difficult to have VHR and VAL on the same server. However....what I would like to hear is that Veeam have agreed it's necessary, will be developed in the future, and for it to be fully incorporated into B&R so we can properly manage it rather than having to jump off and on servers that are meant to be completely locked down and shouldn't have human day-to-day interaction. I know you guys have a MASSIVE R&D list already, but surely this really is a basic requirement of the product?

Cheers, Rob.

[Gostev]

I'm afraid this will not be developed in future because VAL presence will increase the attack surface significantly, which goes against the very concept of hardened repository. Thanks

[jazzoberoi]

Has anyone tried this?

https://www.starwindsoftware.com/blog/s ... eplication

[ratkinsonuk]

I've not read it in detail, but the only obvious benefit is they're setting up and hardening the Linux server for you. Pretty much everything else seems to be the same as any other VHR install. I think I'd rather just follow the Veeam guidelines and set up my own server.

[mkaec]

I'm afraid this will not be developed in future because VAL presence will increase the attack surface significantly, which goes against the very concept of hardened repository. Thanks

How does it increase the attack surface? Ideally, we'd like a way to send OS backups out of the repository, but not have additional open ports. Recovery could be done by local media.

[Gostev]

The attack surface would be increased by at least a few orders of magnitude. Hardened repository is implemented with relatively little code, among which only the tiniest portion runs in a privileged process. Veeam Agent for Linux on the other hand is a beast... too many features and lots of code => many more potential vulnerabilities.

[jazzoberoi]

Hi gostev,
Is it a possibility that veeam can provide us with a Linux hardened repo appliance iso with the capability of backing up the config of the VHR?

This way in a scenario where VHR needs to be rebuilt, it’ll be a simple and straight forward process..

[Gostev]

Could you detail what configuration settings are you thinking about?

[johannesk]

I'm interested in the prebuild ISO @Gostev
Rick Vanover did this nice demo
https://community.veeam.com/blogs-and-p ... itory-4808
But now the ISO download has been removed with this comment

'Installable .ISO: )18-December 2023 - We’ve removed this download and a new one is coming “soon”!)'

Will there be another installable ISO made available?

regards,
Jóhannes

[Gostev]

This particular ISO did not come from Veeam R&D. Quote from the link you posted: "It's a community project". So you should ask your question about its futures there.