Comprehensive data protection for all workloads
Regnor
VeeaMVP
Posts: 1007
Liked: 314 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Build an immutable backup repository - article series

Post by Regnor » 1 person likes this post

You're absolutely right Anton. While I was thinking about in-guest iSCSI, it doesn't change much about the fact that you can attack it via the hypervisor.
Physical is the way to go if you want to do it correct.
albertwt
Veteran
Posts: 941
Liked: 53 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Build an immutable backup repository - article series

Post by albertwt »

Yes, It makes perfect sense to create the immutable backup repository on a physical machine or to disconnect the management network.

Thanks, @Gostev and @Regnor for the input.
--
/* Veeam software enthusiast user & supporter ! */
Ciso_2021
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 13, 2021 7:19 pm
Full Name: Julien Ange
Contact:

Re: Build an immutable backup repository - article series

Post by Ciso_2021 »

A question
We are trying to archief this, I have a question can we share the repo between two offices ( multiple jobs )?
Most of the jobs gonna need to be copy job or it gonna be a wast of the hardware ?
Gustav
Enthusiast
Posts: 50
Liked: 66 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

@Ciso_2021: Yes - and, apart from the (probably) increased storage volume requirement, that would probably save hardware.
Ciso_2021
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 13, 2021 7:19 pm
Full Name: Julien Ange
Contact:

Re: Build an immutable backup repository - article series

Post by Ciso_2021 »

Unfortunately just discovered today it’s not possible to have 3 offices veeam copy theirs jobs to the remote Ubuntu server.
Gustav
Enthusiast
Posts: 50
Liked: 66 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav » 1 person likes this post

Instead of copying, try direct backup jobs.
cgsm
Enthusiast
Posts: 97
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: Build an immutable backup repository - article series

Post by cgsm » 1 person likes this post

Hello,

I just followed this guide completely and came across and error. Veeam recommends NOT installing the Agent, as noted in Part 6, on the repo server. Doing so results in errors in VBR where you cannot edit the repo in any manner (change name, description, etc.).

Per Veeam support:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

Veeam hardened repositories do not use root, while some VAL components do.

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened, and there is a possibility that it may not function at all.

Errors in VBR when the Agent is installed on the repo:
- Enabling restricted mode for Installer Error: VAL components are installed on the target machine VAL components are installed on the target machine.
- Failed to save Backup Repository: VAL components are installed on the target machine.
- Infrastructure item save failed Error: VAL components are installed on the target machine
einhirn
Enthusiast
Posts: 54
Liked: 18 times
Joined: Feb 02, 2015 1:51 pm
Contact:

Re: Build an immutable backup repository - article series

Post by einhirn »

Hi,
Per Veeam support:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.
I understand how this could be a problem with a managed backup agent, which seems to be the case described here. Does this also apply to a standalone/unmanaged install backing up to a different storage via SMB?
Gustav
Enthusiast
Posts: 50
Liked: 66 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

cgsm wrote: Mar 08, 2023 5:22 pm I just followed this guide completely and came across and error. Veeam recommends NOT installing the Agent, as noted in Part 6, on the repo server. Doing so results in errors in VBR where you cannot edit the repo in any manner (change name, description, etc.).
I don't know about that. To me, the natural choice was and is Veeam, indeed as it has ran with zero issues since the articles were written and can be monitored from your Veeam console including notifications if installed as to my guide - also offering boot media and bare metal recovery.

You are, of course, free to backup the Linux server using any other method as to your preferences, though this will for sure not be supported by Veeam.
@IT_Guru
Novice
Posts: 4
Liked: never
Joined: Jan 22, 2019 3:50 pm
Full Name: Bill Bednarzyk
Contact:

Re: Build an immutable backup repository - article series

Post by @IT_Guru »

cgsm wrote: Mar 08, 2023 5:22 pm Hello,

I just followed this guide completely and came across and error. Veeam recommends NOT installing the Agent, as noted in Part 6, on the repo server. Doing so results in errors in VBR where you cannot edit the repo in any manner (change name, description, etc.).

Per Veeam support:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

Veeam hardened repositories do not use root, while some VAL components do.

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened, and there is a possibility that it may not function at all.

Errors in VBR when the Agent is installed on the repo:
- Enabling restricted mode for Installer Error: VAL components are installed on the target machine VAL components are installed on the target machine.
- Failed to save Backup Repository: VAL components are installed on the target machine.
- Infrastructure item save failed Error: VAL components are installed on the target machine
We are experiencing the same issue "Failed to save Backup Repository: VAL components are installed on the target machine". Can you provide the official support statement that Veeam Support is making about "Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.".

If this statement is true then using Veeam to back up the operating system using the linux agent isn't really possible.
cgsm
Enthusiast
Posts: 97
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: Build an immutable backup repository - article series

Post by cgsm » 1 person likes this post

This is the exact message I received from Veeam in regards to my case 05919113:
After further research on the error you are seeing and discussing with my team: Veeam agent for Linux is not supported on Linux servers that hold the hardened repository role.

Veeam hardened repositories do not use root, while some VAL components do.

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened, and there is a possibility that it may not function at all.
Gustav
Enthusiast
Posts: 50
Liked: 66 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

@cgsm:
Could it be, that you have attempted to install the Linux agent remotely - from the Veeam B&R console? That won't work.
I think that is what the response you received from Veeam is about.

Veeam must be installed "physically" on the Linux machine (requires sudo), and - apart from the repository - it must be untouched by Veeam.

My first attempt was, of course, to install the Linux agent from the console.
I recall, that I never made it work - I even had to reinstall the Linux machine to remove all traces of Veeam, then install the repository, and later (on the Linux machine) install Veeam for Linux as described.
This way, Veeam on the Linux machine runs isolated and, thus, uncontrolled from your normal Veeam backup environment. Any change of the backup plan must be done on the Linux machine.
Ciso_2021
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 13, 2021 7:19 pm
Full Name: Julien Ange
Contact:

Re: Build an immutable backup repository - article series

Post by Ciso_2021 »

Gustav wrote: Mar 06, 2023 10:30 am Instead of copying, try direct backup jobs.
Unfortunately this is not possible as some offices are using slow internet like 100/20MB.
Is there any other way of doing this ? Cannot have a physical server for each office.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Build an immutable backup repository - article series

Post by mkaec »

Having the agent and utilizing the same machine as the hardened repository, may result in a repository that is not sufficiently hardened...
I'm puzzled by this. My experience with the agent (in free mode) is that the backup server can't ask it to do things (so it wouldn't need to be listening on ports). The Agent does backups and then sends them over to the repository. If I open the restore wizard, I get an error that I cannot proceed with single use credentials.

I was never thrilled with the plan of just crossing my fingers and hoping that nothing bad happened to the hardened repository's system volume. It would be really nice if a supported way of using VAL with a hardened repository could emerge.
doktornotor
Enthusiast
Posts: 95
Liked: 31 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: Build an immutable backup repository - article series

Post by doktornotor »

Not really sure how's the hardened repo's system volume related here? What kind of maintenance are you planning to do there via Agent/VBR? You either use physical console or some dedicated KVM VLAN (iLO, iDRAC, IRMC) to manage the machine where the hardened repo is located. Not Veeam.

:?
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Build an immutable backup repository - article series

Post by mkaec »

If the OS volume dies, that takes the repo offline. Being able to restore using a VAL backup would be a lot easier than rebuilding from scratch.
doktornotor
Enthusiast
Posts: 95
Liked: 31 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: Build an immutable backup repository - article series

Post by doktornotor »

Really? Reinstalling very much vanilla machine with nothing but the OS running from ISO takes about, hmmm... 30 minutes (or even less with something like kickstart).
Gustav
Enthusiast
Posts: 50
Liked: 66 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

Ciso_2021 wrote: Mar 29, 2023 7:14 pm Unfortunately this is not possible as some offices are using slow internet like 100/20MB.
Is there any other way of doing this ? Cannot have a physical server for each office.
You may check out the WAN accelerator that is enabled with a paid license (and the trial, of course). It can optimise things, though not do magic.

We have a similar line between two locations, but we didn't even consider running Veeam across that line, indeed not as Veeam runs on very modest hardware.
So, we picked an old trusted HP machine and dedicated that to Veeam at the remote location and another, even older Lenovo (now 15 years old!), for the Linux repository - this is an area where you can reuse machines out of warranty, as you easily can replace them should they fail.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Build an immutable backup repository - article series

Post by mkaec » 1 person likes this post

doktornotor wrote: Apr 02, 2023 2:21 am Really? Reinstalling very much vanilla machine with nothing but the OS running from ISO takes about, hmmm... 30 minutes (or even less with something like kickstart).
Yes. Restoring from backup would be much easier and save a ton of time. There's a reason the first post of this thread has a 9 part guide with 180 screen shots showing how to properly do the build. (Thanks, Gustav! Awesome work!)
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Build an immutable backup repository - article series

Post by mkaec » 1 person likes this post

So, I ran into a complication from having VAL on the repository. If I try to edit the repository in B&R, I get errors:

- Enabling restricted mode for Installer Error: VAL components are installed on the target machine
- Failed to save Backup Repository: VAL components are installed on the target machine
- Infrastructure item save failed Error: VAL components are installed on the target machine

To me, this is actually a good thing. One possible line of attack is for an attacker to edit the repository and change the immutable period to 0, then wait for all the immutable backups to age out. With VAL on the repository, this isn't possible. In a weird way, it's an extra line of protection. Once a repository is set up, there's not really a need to edit it. I only tried editing it today to show something to a colleague. If I do need to really edit the repository, I'm fine uninstalling VAL first.

I did a port scan of the repository before and after VAL was installed, and the same ports were listening (6160 & 6162). So, it doesn't appear that VAL has opened up any dedicated ports. And backups have been running ok. So for me, at this point VAL appears to be a net gain.
ratkinsonuk
Expert
Posts: 111
Liked: 16 times
Joined: Dec 10, 2018 10:59 am
Full Name: Robert Atkinson
Contact:

Re: Build an immutable backup repository - article series

Post by ratkinsonuk »

Having spent a number of weeks working with support to get our VHR server and infrastructure set up, I've just come across this 'VAL components are installed on the target machine' error. It looks like I'm going to have to back-track and rebuild the VAL backup into a self-contained mode again.

I want to say out how disappointing this is as a long-term Veeam customer. We rely on the different tiers within the Veeam infrastructure to allow us to manage and monitor the backups. Veeam spends a vast amount of it's resources giving us tools like SureBackup to make sure we're kept safe and everything is reliable.

Veeam Hardened Repository has then come along, which is a great tool to incorporate into software what we'd normally have to do with expensive storage devices, namely security and immutability. Unfortunately we're then told, 'Don't worry about backing up your VHR server, just rebuild it from scratch'.

In our case, that would take a long time given that it's not a simple Linux configuration. And all the time we're rebuilding the server, we can't get on with the task of restoring the rest of our infrastructure that's potentially been wiped, encrypted or broken. What if I went out to Veeam users and said, don't worry about backing up your servers using Veeam, just rebuild them from scratch if there's a problem. I think I'd soon get booted out.

It doesn't take much thought to come up with plenty of reasons why it's difficult to have VHR and VAL on the same server. However....what I would like to hear is that Veeam have agreed it's necessary, will be developed in the future, and for it to be fully incorporated into B&R so we can properly manage it rather than having to jump off and on servers that are meant to be completely locked down and shouldn't have human day-to-day interaction. I know you guys have a MASSIVE R&D list already, but surely this really is a basic requirement of the product?

Cheers, Rob.
Gostev
Chief Product Officer
Posts: 31806
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Gostev »

I'm afraid this will not be developed in future because VAL presence will increase the attack surface significantly, which goes against the very concept of hardened repository. Thanks
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: Build an immutable backup repository - article series

Post by jazzoberoi »

ratkinsonuk
Expert
Posts: 111
Liked: 16 times
Joined: Dec 10, 2018 10:59 am
Full Name: Robert Atkinson
Contact:

Re: Build an immutable backup repository - article series

Post by ratkinsonuk »

I've not read it in detail, but the only obvious benefit is they're setting up and hardening the Linux server for you. Pretty much everything else seems to be the same as any other VHR install. I think I'd rather just follow the Veeam guidelines and set up my own server.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Build an immutable backup repository - article series

Post by mkaec »

Gostev wrote: Feb 27, 2024 4:57 pm I'm afraid this will not be developed in future because VAL presence will increase the attack surface significantly, which goes against the very concept of hardened repository. Thanks
How does it increase the attack surface? Ideally, we'd like a way to send OS backups out of the repository, but not have additional open ports. Recovery could be done by local media.
Gostev
Chief Product Officer
Posts: 31806
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Gostev »

The attack surface would be increased by at least a few orders of magnitude. Hardened repository is implemented with relatively little code, among which only the tiniest portion runs in a privileged process. Veeam Agent for Linux on the other hand is a beast... too many features and lots of code => many more potential vulnerabilities.
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: Build an immutable backup repository - article series

Post by jazzoberoi » 1 person likes this post

Hi gostev,
Is it a possibility that veeam can provide us with a Linux hardened repo appliance iso with the capability of backing up the config of the VHR?

This way in a scenario where VHR needs to be rebuilt, it’ll be a simple and straight forward process..
Gostev
Chief Product Officer
Posts: 31806
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Gostev »

Could you detail what configuration settings are you thinking about?
johannesk
Expert
Posts: 159
Liked: 37 times
Joined: Jan 19, 2016 1:28 pm
Full Name: Jóhannes Karl Karlsson
Contact:

Re: Build an immutable backup repository - article series

Post by johannesk » 1 person likes this post

I'm interested in the prebuild ISO @Gostev
Rick Vanover did this nice demo
https://community.veeam.com/blogs-and-p ... itory-4808
But now the ISO download has been removed with this comment

'Installable .ISO: )18-December 2023 - We’ve removed this download and a new one is coming “soon”!)'

Will there be another installable ISO made available?

regards,
Jóhannes
Gostev
Chief Product Officer
Posts: 31806
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Gostev »

This particular ISO did not come from Veeam R&D. Quote from the link you posted: "It's a community project". So you should ask your question about its futures there.
Post Reply

Who is online

Users browsing this forum: No registered users and 310 guests