Build an immutable backup repository - article series

[johannesk]

ok, but will there be a linux installation ISO made available by Veeam for VHR setup similar to the one Rick did for the mentioned demo?

[Gostev]

Time will tell but this is not something Veeam R&D is actively working on at this time.

[mkaec]

I think the ask is if Veeam could provide something similar. People who are using VAL on their hardened repositories don't want all those features that make it a beast. They just don't want to be SOL if something goes sideways with the repository.

[robg]

I agree, it should be an ISO, boot it on something and answer a few prompts.

[Regnor]

In general the setup of a Veeam Hardened Repository isn't that complicated. Especially if you've done it once and documented all necessary commands, you can easily replicate it.

During a disaster there shouldn't be a need to rebuild the VHR, except you've lost the hardware but this would also mean there's no repository/backup left.
But anyways you could still only do the basic steps and reconnect the VHR back to Veeam in that case. Just make sure you don't open any holes during the setup which an attacker could exploit.

[robg]

"isn't that complicated" is a matter of perspective.. Compared to the rest of Veeam, which is clicking through wizards, it's a lot more complicated. Why not look at it from that standpoint. People want turnkey simplicity.

That would definitely be a competitive edge. And please don't do silly things like the vCenter linux appliance, not make it rely on self-signed certificates that expire in 5 years, if a restore is needed on year 6

[jazzoberoi]

+1 for ISO appliance which can be installed on a physical machine.
A few user led projects are already floating around, along with one from Starwinds.. so it is certainly possible to do by Veeam

[mkaec]

It's complicated enough that people are writing detailed guides about how to do it. That's a good point that if the repository OS is damaged the event probably didn't also take out production VMs. But that's not guaranteed to be the case.

[Gustav]

In general the setup of a Veeam Hardened Repository isn't that complicated. Especially if you've done it once and documented all necessary commands, you can easily replicate it.

Perhaps for a skilled Linux admin. But for a Windows admin like me with little knowledge about Linux, the statement will read:

In general, the setup of a Veeam Hardened Repository is quite complicated. However, if you've done it once and documented all necessary commands, you can easily replicate it.

That's exactly what my article series addresses.

[Nick-SAC]

+1 For a Veeam supplied ISO...

[TheWaterbug]

I am following this guided tour right now, on a decommissioned Western Digital Sentinel DX4000 box, and it's very well-written. Thank you!

It could use a few minor updates, as the current version of Ubuntu is 24.04.1, vs. the Ubuntu 20.04 that the guide was written against, and there are a few additional questions to be answered during setup.

I am running into a few issues. Is the correct thread to discuss? Or is there another?

[TheWaterbug]

^^ Never mind. I just downloaded the Immutable Repository ISO, and I'm going to go that route instead, since it's supported.

[Gustav]

Errors in VBR when the Agent is installed on the repo:
- Enabling restricted mode for Installer Error: VAL components are installed on the target machine VAL components are installed on the target machine.
- Failed to save Backup Repository: VAL components are installed on the target machine.
- Infrastructure item save failed Error: VAL components are installed on the target machine

I got received this error today having updated VBR to version 12.3.0.310_20241211, leaving the Linux server and the hardened repository inaccessible (marked with red crosses).
I had to realise, that to proceed you must first uninstall VAL on the Linux machine with this command:

Code: Select all

sudo apt-get remove veeam veeam-libs veeamsnap

Then, as described in my articles, elevate the veeamuser account and, from the Veeam console, create a single-use access, update the Linux server agent, rescan the repository and, finally, demote the veeamuser account.

That leaves the question about how to backup the Linux server itself. Veeam gave up on this, even advices not to do it, and the Veeam Hardened Repository ISO can't or won't as stated as a "Limitation" here: https://helpcenter.veeam.com/docs/backu ... ml?ver=120

The ISO does not currently have a “repair mode”. If the operating system volume is lost due to a RAID failure or similar issue, you will need to install another supported Linux-based operating system and mount the storage volume or volumes to access your data.

You probably won't have any success trying to reinstall Veeam Agent for Linux. I tried, but it won't even download:

Code: Select all

Connecting to download2.veeam.com (download2.veeam.com)|3.164.240.4|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-12-31 09:47:01 ERROR 403: Forbidden.

Your only option, I guess, is a local backup like https://relax-and-recover.org/ not related to Veeam at all.

[Gostev]

"Can't ot won't" is not true! The Repair option is coming in the next immediate ISO update.

[ratkinsonuk]

As someone that spent a long time with Veeam Support trying to find a workaround to back up the VHR server using Veeam, I can confirm that it's impossible to mix VHR and Veeam B&R. They're blocked from working with each other (by design). In the end, I installed and configured the Veeam Linux agent, hooked it up to B&R and then hardened the repository server as best I could. To me, a backup of the server is more important than having on-prem file immutability, which is mostly what's now missing from my instance.

I think it would be nice if the Limitation section did have an entry to say a VHR cannot be backed up using Veeam B&R.

[Gustav]

"Can't ot won't" is not true! The Repair option is coming in the next immediate ISO update.

Great! Any timeframe for this update? And will it be only a "repair" option or a full bare-metal-recovery feature?

[Gostev]

What is the difference between the two in this situation? The goal is to recover OS when losing the OS disk. Are you looking to do anything more than that?

[Gustav]

What is the difference between the two in this situation?

A repair assumes an OS disk operating at some level and that you have the saved configuration. Often the repair will then be able to repair a missing boot or broken configuration; if not, the OS must first be manually installed using the ISO and then the repair should be able to reestablish the configuration.

A bare-metal-recovery assumes nothing but a blank disk. Having the latest backup and created a recovery media, you can recover completely in minuttes in one go - as currently offered by VAL.

[Gostev]

Got it. As it comes to the managed hardened repository repair, we're talking specifically about "blank OS disk" scenario.

[TheWaterbug]

"Can't ot won't" is not true! The Repair option is coming in the next immediate ISO update.

Is this it?

You can repair a hardened repository installed from the Veeam Hardened Repository ISO. Any data stored on the data volumes will be preserved. To do this, a remote console or bootable USB stick is required. For more information, see Installing from Veeam Hardened Repository ISO.

[Gostev]

Yes.