how to integrate vo365 into a vbr11 hardened linux repo?

[pesos]

Hi all,

We are trying to design a ransomware-resistant backup infra.

We currently have a primary datacenter site that is running b&r11 writing local hyper-v workloads to a basic local storage REFS repo. The hyper-v host with the local storage is also a vbo proxy which writes vo365 backups to a repo on the same local REFS volume. Generally this is fine except for the risk of ransomware rendering all backups useless.

We are just about done implementing a separate management domain (with extremely limited staff access) with separate firewalls/networks/AD domain for the hyper-v servers and vbr servers to live in. While not a true air gap, this at least provides us a solid amount of separation between the main production AD network/servers and the mgmt/hyperv/veeam level which is only accessible by two people and requires multiple hoops/mfa/locked down wan ips to access.

We are looking to take this one step further and implement a 3rd separate network with a single server functioning as a linux hardened repository (this may be a virtualized linux box atop a standalone hyper-v server vs. a physical linux box, not sure yet - the windows box would be behind a separate dedicated appliance and only be accessible via mfa/locked down wan ip list and the virtualized linux box would then be controlled via the virtual console - no need for direct remote access to it). This hardened repository will live at a different physical site than the datacenter with the large primary repo storage (but they have a robust 500 mbit link).

I'm assuming it will be straightforward enough to configure backup copy jobs from the datacenter vbr server over to the offsite hardened repository. What I'm unclear on is the best way to get the VBO5 repo data into the hardened repo. Is it as simple as loading the VB for Windows agent onto the windows proxy that has the direct attached storage holding the primary vbo repo and backing that up to the offsite hardened repo? I have seen posts regarding backing up the VBO server itself with VBR, however I don't think this would work for us since the VBO server doesn't hold the actual VBO data... the proxy server does (locally attached storage).

Thanks!

[Mike Resseler]

Pesos,

Backing up the VBO service (so the controller and the proxies) can be done with VBR and with Veeam agent for windows. We indeed sometimes only say VBR but the agent works also. In your case, you need to be sure that all the servers of VBO are protected so that you have all the data

[pesos]

Thanks Mike! So to be clear, I would create a single vbr job that needs to include both the virtualized vbo server - and the VBWagent pointed to the vbo repo? Or separate jobs are ok?

If it comes to a restore, is the explorer for exchange sufficient to dig into the hardened repo and recognize the vbo layer and restore from it, or is there more to the restore story? Sorry I’m having trouble getting fully wrapped around all the layers

[Mike Resseler]

You should create a single job to make sure that they are being processed more or less the same time. Preferred outside of the hours that the VBO jobs are running.

For your second question, I am not 100 percent aware about this hardened repo but as far as I know it should not matter for the Veeam explorer and it should recognize that it is a VBO server and show the data. Maybe @HannesK knows this?

[pesos]

With how long vbo jobs take to process these days, getting a vbr job to thread the needle and run outside of that time is going to be a tall order indeed!

[Mike Resseler]

@pesos I saw your post in the other forum thread. Please open a support case for this one

[HannesK]

Hello,
Hardened Repository handles data like any other repository, with the exception that supported workloads become immutable https://helpcenter.veeam.com/docs/backu ... ml?ver=110

If something works with a normal repository, then it also works with a Hardened Repository. Immutability is an add-on feature. For a backed up VBO server with application aware processing, it shows it like this



To restore Microsoft 365 content, I would use VBO as it has the credentials.

Best regards,
Hannes

PS: I would simplify the setup and put everything in VMs. Running applications in the management partition of Hyper-V is a bad idea in general.

[pesos]

Apologies, somehow I never saw this last post.

Do you mean that the entire VBO server and dataset should be virtualized (instead of right now where the repo it writes to lives on a physical server)?

[HannesK]

Hello,

The hyper-v host with the local storage is also a vbo proxy which writes vo365 backups to a repo on the same local REFS volume

as far as I understood the initial post, the VBO server server is already virtualized. it's running in the root partition of Hyper-V

yes, a normal VM should be used instead of the root partition VM. Applications should always run in VMs and never in the root partition.

Best regards,
Hannes

[pesos]

I’ve never heard of the root partition being called a vm. That’s the hyperv host right? Maybe we are saying the same thing. Anyways, the vbo server is already a “normal” vm. The hyperv host is a vbo *proxy* so it downloads and writes the data to a local drive…. So are you saying the proxy and vbo storage should also be virtualized instead of us writing directly to physical storage (so that the virtualized vbo storage can more easily be backed up to the hardened repo)?

[HannesK]

yes, the Hyper-V user interface is a VM (see the link I posted above). Similar like old ESX (4.0?) and before. The root (or parent) partition is the only partition that has direct access to physical memory and devices. That's why no application should run in the root or parent partition.

yes, everything should be a normal VM.

[JRRW]

It requires rearchitecting things, but another option is to go with a HLR that can present both Block and Object - then you can just have VBO write to the Object Repository which itself will often have immutability and the ability to self-replicate/backup to another repository like itself.

[pesos]

yes, the Hyper-V user interface is a VM (see the link I posted above). Similar like old ESX (4.0?) and before. The root (or parent) partition is the only partition that has direct access to physical memory and devices. That's why no application should run in the root or parent partition.

yes, everything should be a normal VM.

Am I the only person who gets "The forum you selected does not exist" every time I try to reply to a post and log in?

So if we virtualize the actual storage of the vbo backups so that b&r can in turn back that up to the hardened repo, is that more or less efficient than using agent for windows to push the data to the repo? This will be a WAN target. I guess for purposes of restore, we'd want to go the virtualization route it sounds like...

[Mike Resseler]

RE: "The forum you selected does not exist" is not coming up for me, but check next time you read a post whether you are already logged in or not. And you can select the box to "remember me" (That is what I do at least )

RE: Virtualize the actual storage: That is indeed a possibility, but if you use a windows agent that can work also. Both methods should work fine, not sure why you think it is less efficient virtual then using Agent for windows though.

[pesos]

Hi Mike, I didn't say I thought it was less efficient, I asked if it was less or more Wasn't sure if having it virtualized made things more complicated as far as incremental adds etc - but we will go that route so it can all be captured without need for the agent at all.

I always click "remember me" however it never sticks, so every time I click Post Reply I get prompted to log in, and when i do I get the forum does not exist message. Then I have to click back a couple times and click Post Reply again and go about pestering you some more

[pesos]

Hi Mike, we virtualized the vo365 server and pushed all its many terabytes to the hardened repo offsite (took a long time on 100mbit link, but it finished). We have 185 days set on this job as the immutability window. What happens when the initial full falls outside that window? Are we able to run a synthetic full at that point since the full file will once again be mutable? Would hate to have to do another active full as it takes two weeks and is prone to interruptions.

Thanks!

[Mildur]

Hi

You need periodically synthetic or active fulls to have immutable backups. If you have configured a Forever Incremental Backup Job for your VBO365 VM, the backup files will not be immutable.

Assuming your hardened repo has xfs with reflink as a filesystem, use weekly synthetic fulls. They don‘t need the entire space.
My Linux hardened repos are only 30 days immutable, because gfs restore points are always immutable their entire retention time.
A retention time of 30 days for a backup from a vbo365 server should be enough, because each one of this 30 days contains the entire retention time of all vbo365 backup data.

[pesos]

Thank you. So a synthetic full is possible with the base full being immutable?

[Mildur]

Yes, I have updated my post with additional text

[pesos]

Thank you!

Excuse my ignorance because I have never worked with xfs/linux much before setting up this new hardened repo... But is the synthetic full dependent on the active full being intact? If it is, what happens when that initial active full becomes mutable (and/or is deleted)?

[Mildur]

When a synthetic full happens, then it backups the changed blocks from the production environment and takes already existing blocks on the backup repository from unchanged blocks.

Without refs/xfs filesystem, the blocks will be copied to a new FullBackup file (synthesizing backup files from existing blocks).

When you have refs/xfs as a filesystem, the changed blocks will also be taken from the production environment, but instead of copying the unchanged blocks on the backup repo to a new file, veeam will reuse the existing block without doing a copy. It will only be a reference. Veeam calls this feature FastClone.

For your question. Veeam will be able to delete the active full after retention and immutability period is over. But the blocks which are also referenced in the synthetic full will not be deleted. The filesystem will handle that for veeam.

[pesos]

Got it, so just like refs. Thank you!

[Mildur]

Exactly
Just make sure, that you have created the xfs filesystem with this command, or it will not use FastClone. Even if you have enabled it in the backup repository properties.

Code: Select all

 mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sda1

[Mike Resseler]

Thanks @Mildur for the answers here. I start to think you have a cheating system, I can't open my notification mails that fast and you already answered

[Mildur]

Your welcome
My Cheating System is my Mail client
I don‘t like the unread count

[pesos]

Unfortunately every time we try to create a synthetic full, it gets to about 80ish percent and then fails with

Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.

[pesos]

I noticed on the console of our linux box "out of memory killed process veeamagent" a number of times, so I tripled its ram from 16gb to 48gb and reattempted - successful synthetic full!

[Mike Resseler]

@pesos

If you want to get this troubleshooted, feel free to create a support call. Unfortunately I cannot help you with this as this is a VBR feature and VB365. You might want to post your experience on the VBR forums and see if others have faced the same issue

[pesos]

Thanks Mike. I assume it was resource exhaustion as none of the other jobs had the same issue... with this job being extremely large I assume 16gb ram just wasn't quite cutting it. Seems to be good now with triple that (maybe overkill but that's ok).