Immutable Backups on Linux Repository

[brock]

I have a few questions on configuring Linux Repo's for Immutable backups. Here is an overview of our current configuration:

Primary Site:
B&R Server 1: Server 2019
Linux Repo 1: Ubuntu 20.04 (Underlying storage: MSA 2050 - 1) (XFS and added as a SOBR)

Secondary Site:
B&R Server 2: Server 2019
Linux Repo 2: Ubuntu 20.04 (Underlying storage: MSA 2050 - 2) (XFS and added as a SOBR)

(Backups are Incremental with Synthetic Fulls) (GFS is not currently used for Backup Copies)

The Secondary site backs up the primary sites data to MSA 2050 - 2 on Linux Repo 2. It then sends a backup copy to Linux Repo 1 on the MSA 2050 - 1.

The Primary site backs up the Secondary sites data to MSA 2050 - 1 on Linux Repo 1. It then sends a backup copy to Linux Repo 2 on the MSA 2050 - 2.

So there is always a backup from one Site/B&R Server and a backup copy from the other Site/B&R Server on each Linux Repo. One of the listed limitations is that: "The hardened repository cannot be shared between different Veeam Backup & Replication servers.". Would this limitation be an issue in the above scenario where the same Linux Repo has backups from B&R Server 1 and backup copies from B&R Server 2?

Also, I'm not 100% sure how you differentiate between whether you used a "single-use credential" or a "persistent credential". What is the best way to determine that? To continue on that, is there a preferred credential to use? I assume it would somewhat depend on whether or not the Repo will have immutable AND non-immutable backups on it? In regards to backup and backup copies, is it best to have both sources be immutable?

Lot's of questions, but I'm just wanting to make sure we properly implement these Immutable Backups into production.

Thanks!

[HannesK]

Hello,
as far as I understood, you added Linux Repo1 to B&R Server 1 and 2 (same the other way around). That would be "sharing" between different backup servers. That means you are running an unsupported setup, which technically usually works fine as long as the servers are not overloaded.

There is no difference whether it's single-use credentials or not. I recently updated a thread about sharing components between backup servers.

To continue on that, is there a preferred credential to use?

I prefer "more security". So "single-use credentials" is the only valid option from my point of view.

In regards to backup and backup copies, is it best to have both sources be immutable?

I'm not sure I understand. Can you maybe explain more? The source is irrelevant for the destination. BCJ must have GFS enabled to allow immutability.

I put together a lot of information around Hardened Repository in this whitepaper. Maybe that helps.

Best regards,
Hannes

[brock]

To start off, I read through your whitepaper, and it was EXTREMELY informational and relevant to the topic. Thank you for that!

as far as I understood, you added Linux Repo1 to B&R Server 1 and 2 (same the other way around). That would be "sharing" between different backup servers. That means you are running an unsupported setup, which technically usually works fine as long as the servers are not overloaded.

So yes, Linux Repo 1 is added to both servers, BUT they are added on each server to a different mapped location, similar to the below:

B&R Server 1:
Backup location - /repo2/DRBackups
Backup copy location - repo1/DRReplication

B&R Server 2:
Backup location - /repo1/ProductionBackups
Backup copy location - /repo2/ProductionReplication

In this scenario, does it still fall under the requirements of sharing between 2 B&R Servers?

These servers are definitely not overloaded, but I don't necessarily want to run an unsupported setup.

I prefer "more security". So "single-use credentials" is the only valid option from my point of view.

Single Use credentials it is!

I'm not sure I understand. Can you maybe explain more? The source is irrelevant for the destination. BCJ must have GFS enabled to allow immutability.

Apologies, I could have worded that better. I'm wondering if it is recommended to have backups, as well as backup copies be immutable. I assume that would be the more secure method, but I didn't know if there were any reasons to NOT have your backup copy jobs be immutable?

To add to that, why do BCJ's need to be GFS-enabled? I'm sure the answer is valid, but I'd like to know, technically, why that is a requirement.

Thanks!

[Mildur]

To add to that, why do BCJ's need to be GFS-enabled? I'm sure the answer is valid, but I'd like to know, technically, why that is a requirement.

Without GFS, a backup copy chain is forever incremental.
Veeam cannot perform merge processes on a immutable chain. This is a process which needs to be done each time or day you run the job. Merging the vbk with the oldest vib file.

I'm wondering if it is recommended to have backups, as well as backup copies be immutable. I assume that would be the more secure method, but I didn't know if there were any reasons to NOT have your backup copy jobs be immutable?

I would configure for both backup repos the immutability flag. But I think, that depends on how much security you want to have.

[HannesK]

Hello,
as Fabian already answered two of the questions, it only leaves this one open

In this scenario, does it still fall under the requirements of sharing between 2 B&R Servers?

yes. different folders on the same server is still the same server. If both would point to the same directory, that would definitely cause technical issues. If you can live with the dependencies during upgrade, then I would probably still go for that scenario. I mean, you could also go with containers or any other virtualization as mentioned in the other thread.

Best regards,
Hannes

[brock]

What sort of dependencies might I be looking at?

If I was to ONLY make the backups immutable, not the backup copies, would it fall under a supported scenario? Then, only 1 B&R server would be connected to each repo with immutable files. Or does it only care that the repo is mounted to multiple B&R servers, and it has nothing to do with the immutability coming from two separate servers?

Trying to find a way to do this in a supported method with my current configuration. I can't believe I'm the only person that has a Primary and Secondary location with 1 shared repo for each location that want's to have a supported solution for Immutable Linux Repo's....but maybe I am.

Thanks again!

[HannesK]

the point is two backup servers accessing the same machine. the "shared hardware" is independent from immutability.

you are not the only one. the whole thread is about customers doing that. multiply that with factor 1000x or more and then we have the the number of customers doing globally (there are only very few customers on the forums).

most customers have one VBR server managing all locations and components. that makes it much easier during upgrade.

[brock]

Ok, to further this discussion, I am really wanting to take advantage of Immutable Backups on our Linux Repo's, but I really like the redundancy of how our VBR Servers are configured.

- What is the requirement for having a separate repo? Does that mean an entirely separate Linux Server? Or could it simply be a separate mount point?

- If I was to consolidate down to 1 VBR server, would it be better suited at our Primary location, or the secondary?

- What would the process look like to consolidate from 2 to 1? Could I just build the jobs identical to how they were built on the secondary VBR Server and then add the repo's and import the backups and have access to all the backups that were previously indexed on the secondary VBR Server? I'm concerned with possible loss of backups/access to backups when making this cut-over.

- Will I run into any issues when adding the repo's as hardened repo's? Can I simply just switch over to the hardened credential on the currently added repo without removing it?

- If I stick with the 2 VBR Servers, and do use immutability on both repo's, what sort of issues might I run into if this is not a supported configuration? Do you think that there may be an added support for this type of scenario at some point, or is this more a caveat with the immutable filesystem and less to do with Veeam itself?

- In the case of having 1 VBR Server, is it safe/advisable to make a VM Replica of the VBR Server VM at the opposing site? I currently do config backups of each VBR Server to the opposing site, but in the case of having only 1 server, it'd be nice to have that built into a replica job that could easily be spun up from vCenter in the event of a major issue.

Thanks,

[HannesK]

1) separated processes are needed. I mentioned containers in the link earlier and there was some feedback on things needed with LXC / LXD

2) yes, same job configuration and "map backup" can do the job. I suggest testing with a test job.

3) hmm, not sure what you mean, but you can add the hardened repository to the other server, yes. just make sure, that not both servers are writing to the same repository

4) sharing components is independent from immutability. the main topic is overloading components and upgrade dependencies. the file system is irrelevant. it's only about sharing Veeam components.

5) yes, replicating the VBR server is one of the most common ways for higher availability. The currently second VBR server should do that replication if the database is running directly on the VBR server (VSS snapshots, VM stun etc.).

[brock]

1. And Containers would be fully supported?

2. I'm currently intending to keep the backup locations the way they currently are, even if i go to 1 VBR server. So Site 1's servers are backed up at Site 1 (Repo1), and then the backup copies are sent to Site 2 (Repo2). Site 2's servers are backed up at Site 2 (Repo2), and the backup copies are sent to Site 1 (Repo1). Does that seem like an appropriate way to handle them? If I move to 1 VBR server I'm intending to keep that method unless recommended otherwise, so I was hoping I could retain those backups at those locations and just import the metadata for the backups that currently reside on whichever VBR server I retire.

3. I'm wondering if I can just "edit" the "Managed" Linux Server's SSH Connection and switch from a standard credential to a "hardened" single-use credential, or do I need to remove the Linux server and repo's and re-add them?

4. Understood, so the best option is to avoid 2 VBR servers mounting to the same repo.

5. That is great news. So when moving down to 1 VBR server, I need to decide whether my primary site or secondary site should house that VM and then I can just create a VM Replica of it at the opposing site so if I lose the VBR server or the entire site, I can just jump into vCenter and power up the replica and have access to all the backups in a matter of a few minutes. That is almost as good as having a VBR server at each location. The database for each VBR server is resident on the VBR server itself.

Thanks!

[HannesK]

1. Quote from the other thread

containers or any kind of virtualization is transparent for the software. We don't declare support for every single virtualization technology in general. Running the repository role in a virtualized environment is supported, yes.

2. That's the recommended design in general, yes.

3. yes, you can change to single-use credentials for existing Linux hosts

5. I think it's even better than having two VBR servers, because that removes all update dependencies and simplifies the overall usability. One console for everything

[brock]

HannesK, I've come across another issue with this setup. Per the below referenced KB, "Hotadd" mode is no longer available if you have a B&R server "VM replica" in the same vCenter server environment as the "Active" B&R server. I noticed backups were running extremely slow from one of our B&R servers. Upon inspection, I discovered that it was logging a message that it was unable to hot add source disk, and would fail over to network mode. This took a :30 minute backup and turned it into an 8-10 hour backup. Per the KB, it looks like the options are to either stop replicating the B&R Server or to replicate it to a standalone ESXi host outside of the vCenter environment.

https://www.veeam.com/kb2159?ad=in-text-link

Currently, this is how it would look:

Production Site:
BKUP1 (Uses BKUP2 Proxy for DR backups)<<<This is where the issue lies. It sees BKUP2 AND BKUP2 VM Replica and won't allow HotAdd Mode.
BKUP1 Proxy
BKUP2 VM Replica

DR Site:
BKUP2 (Uses BKUP1 Proxy for Production Backups)
BKUP2 Proxy

Once consolidated, this is likely how it would look:

Production Site:
BKUP1 (Uses DR Proxy for DR Backups) (Uses BKUP1 Proxy for Production Backups)<<<Wouldn't the issue now be that when BKUP1 does Production Backups, it sees BKUP1 and BKUP1 Proxy and fails to use Hotadd Mode?
BKUP1 Proxy

DR Site:
DR Proxy
BKUP1 VM Replica

Do you know of a viable solution for this, aside from those two recommendations? I don't think replicating to a standalone host that is not part of the vCenter environment is a very practical option. I had considered the idea that if you had the proxy server separate from the B&R server, that would probably be a viable solution. At that point, you'd be doing a VM Replica of the B&R server, NOT the proxy server, which is the server that appears to be causing the issue.

Thanks,

[HannesK]

Hello,
I'm locking this thread now. Please use the other thread to stay "on-topic". This thread is about hardened repository - not about how to back up the backup server

Thanks for your understanding,
Hannes