Restore from hardened repository

[SAFA_IT]

Worst case scenario ...

You have immutable backups on your hardened Linux repository, but you have been hit by ransomware, so you don't
trust any of your Windows servers - including the server Veeam is installed on.

The task ...

Import VMs to ESXi using .vbm & .vbk files from the hardened repository. It seems a bit time consuming to set up a Veeam server from scratch, so I was thinking about either keeping a Windows device with the extract utility off the network, or a bootable Linux iso with the Linux version of the extract utility.

No doubt I have missed something obvious, but I have read many good ideas on this forum over the years, so I thought I would put it out there for comments.

[Mildur]

It seems a bit time consuming to set up a Veeam server from scratch

If you have many vm’s to extract, it would be faster to setup a new windows server, import the backup files from the linux repo and start a instant vm recovery session and the users are working again.

Keeping a secondary backup server ready to start up is another option. In case of a disaster recovery, You could start up the windows server, install veeam on this server, import the config backup file and all restore points of your VM‘s are accessible todo the restores without extracting the vmdks first.

[SAFA_IT]

Thanks for the prompt reply.

I was looking at it from the perspective where you couldn't trust any of your Windows servers, including the secondary backup. Like a 'last chance' to get data back when everything else has failed - at that point I would not be too concerned with how long it took.

In reality, I'd probably use offsite DR or tape before I got to the scenario I presented.

I recall a recent story where the company restored their domain to the previous day using their backup solution before checking on when their systems were originally compromised. If someone has been intruding on your system for a while you might have to go back a long time before you are safe.

[Mildur]

If you don‘t trust the content of the backups, you can use secure Restore to check the vms before putting them back to operation:

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[SAFA_IT]

I remember trying to set that up for SureBackup, but it was not compatible with my AV solution.

The way I see it, being clean of malware shouldn't be the deciding factor on when to restore from. If someone has had access to your system for (say) the last two weeks and has gone undetected, they could be changing the values on spreadsheets, phone numbers, contacts and so on.

I'm getting a bit off topic though - I think I will keep a spare copy of the Veeam server offline and keep it updated with the config backup as advised

Thanks!

[SAFA_IT]

In case it is of use to anyone else, here is what I did.

I used Windows Server Backup to backup the Veeam server and performed a bare metal restore to a physical server. I then restored the configuration backup and plugged the hardened repository directly into the physical server.
I re-scanned the hardened repository for good measure then attempted a guest file restore. The result was "Required backup files are missing or temporarily unavailable".
Never mind, I thought, what I really want to do here is do a Disk Restore > VM files restore (VMX,NVRAM). So I tried that and got the same message.

However, if you click through the message the restore works as expected.

Clicking other restore options also brings up the backup files missing message, so perhaps this is a bug.

So, the conclusion I take from this is that on the downside, it is a lot of work just to extract some files, but on the upside you don't have to worry about connecting to the hardened repository - it is immediately available when you connect it to the restored Veeam server.

I would like to see a recovery environment for this, but I know that suggestion won't be popular.

[soncscy]

What do you mean by recovery environment exactly? I'm just not sure I'm envisioning exactly what you're talking about.

>Import VMs to ESXi using .vbm & .vbk

If that's your goal, Veeam has had a standalone backup extractor for some time: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

I think I recall seeing a forum post that it's even a standalone download now. Pop that on a known good machine, extract the desired vm files (vmx, vmdk, etc), and enjoy. I've not personally run it on a hardened repository, but I don't see why it would matter.

I do think that your solution, while working, overcomplicates it. Fabian's idea is best in my opinion for a true disaster.

For me, in a real disaster, first I just want to restore VMs, and the time to get a Windows ISO up and running in a secure area and doing an install is far less than the potential for post-restore issues. I'd rather go at it with a clean slate and get a restore environment ready to go instead of trying to get the original backup environment running right then and there. Backups are important, but restoring the lost environment is more-so in my opinion.

For our relatively modest production environment for my team (just a few ESXi hosts), deploying a new Windows VM is about 10 minutes or so from OVF to Windows Desktop. Installing Veeam is usually a factor of about 20 minutes. At that point, you're just a few added servers away from starting restores.

I would strongly suggest plot out such a DR strategy in your environment and test it a few times -- how long would it take, can you store the OVF safely somewhere, if you needed to download fresh Windows isos from somewhere, how long would it take?

[SAFA_IT]

What I had in mind was an ISO that booted to Win RE or Linux and gave you access to the repository, extract utility, an SCP client and a file explorer.
I have been saving extract to multiple locations for many years, but in a ransomware situation I would be looking for a copy that couldn't be tampered with and a bootable iso like Backup Exec had would suit me fine. I only have a few hosts, but I don't have a team looking after them - just me, so I need to keep the amount of work required in a full disaster scenario to a minimum. I don't even have someone to say 'He's working on it'.

Extract works fine if you have the files stored locally or on a share, but it is looking for a file as the source not a repository.

Now, you can take the opinion that I whatever method you decide for yourself works fine during testing, but I have been in a situation where a power surge took a SAN and half the guest servers including the Veeam server, so I have experienced this sort of situation in production. This left me with the opinion that the recovery process has to be as simple as possible, as you are presented with other issues to deal with at the same time. I appreciate that if you have a team, you have more flexibility over the procedure.
Doubtless I could make something myself to suit my needs, but I don't think I should have to - just like I don't think I should have to perform a file copy if I want multiple configuration backups.

[soncscy]

Er, isn't this exactly what I proposed? I suggest OVF instead of an ISO, but even on slower gigabit networks, you'd have about the same speed for deployment, and I'd argue that a deployed VM from OVF would be better. It would hit every point on your checklist:

- Would require extra steps to be tampered with*
- Easily deployed by a single person

* Keep in mind that something being an OVF/ISO __does not mean it cannot be tampered with by a malicious actor!

I believe you misunderstood my post; while I discussed my team, the process is designed that it can be done from anywhere by anyone without any dependencies. In fact, from the description of "deploy from an OVF", I'm not really sure how you concluded that the process is team dependent. Our process assumes:

- Everything is gone and we're starting from scratch
- Backups only exist in secondary locations (S3, Tape, off-site repositories)*

* To be honest, I find your situation kind of...idealized I guess? Immutable repositories are nice, but what about scenarios where attacker just formatted the immutable volume? Or if the power surge hit all the servers? I don't get the idea of secondary backups from your description

For us, any one person with appropriate privilege levels can perform the entire procedure start to finish, which means you could also in such a scenario, and we can start performing restores at full speed as soon as veeam installs. Extractor is a nice utility, but I think it adds a lot of "latency" in that you have to extract and copy first, then register each VM manually.

And as boring as it is, we have the entire process documented both digitally and even in special DR binders on boring old paper that all of the core team have a copy of.

I'm not telling you what to do of course, but I strongly advise you test both scenarios out -- your described set up sounds more complex to me than just installing a new vbr server somewhere.

But, if you really do want a Linux LiveCD, it's actually really easy: https://help.ubuntu.com/community/LiveCDCustomization

It's a great weekend project (I've done it before multiple times), and if you're set on doing it this way, do it right I suppose and share your result with us! Surely someone else will like it too

[SAFA_IT]

As I said earlier in the thread, I would go to the secondary backups before I got to the hardened repository. I'd like to think the replicas would be clean, secondary copies likewise and tape, offsite backups and offsite DR replicas. If all these were unavailable, I would be more likely to be looking for my passport for a quick getaway.
Immutable backup was the big selling feature for v11 - I was late to adopt as I had to wait for my VCC supplier to move from v10. All I was looking for was a quick way to make use of this feature when it would really be required . My idea of a weekend is not knocking up ISOs. Maybe after 10 years or so Veeam is just getting too complicated for my needs.

As it stands, I think I might try a clean install on the test server using the windows ISO and Veeam stored on external media as the source. Thanks for the suggestion.

[wishr]

Hi David,

The easiest way to recover would be to spin up VBR on literally any machine. It could be even your personal laptop or a fresh windows VM/physical machine. Simply restore the configuration from configuration backup, add your Linux box to the newly created VBR server and start recovering your mission-critical infrastructure using any of VBR capabilities. This is the most proven and simple way to deal with such situations if having/using a prepared in advance DR VBR server is not acceptable, just make sure your configuration backup is actual and stored in a safe spot.

Another option is to have a physical DR VBR server out of the network in a locked room. In the case of a DR, you will just need to power it on, restore configuration backup, then start restoring your data.

Thanks

[SAFA_IT]

Thanks Fedor - I might treat myself to a new laptop.

Your physical solution is what I am working with at the moment. I was wondering how often the configuration backup should be taken up for this scenario as the recommendation is 'periodically'.

[wishr]

Good question. The configuration backup schedule should depend on the schedule of your jobs. Ideally, you'd like to run configuration backup at the end of a day, when all other data protection activities have already run and all configuration changes (if there are any) have been applied. Also, there should be no running jobs at the time when configuration backup is taken to avoid job sessions with the status "failed" on the new VBR server after configuration restore.

[SAFA_IT]

Thanks again - I just checked the time I use and there are no running jobs then.

[SAFA_IT]

"just make sure your configuration backup is actual and stored in a safe spot"

While testing my DR physical server the importance of a current rather than recent configuration backup became clear. I had already been copying the most recent backup to multiple locations, but in the scenario where only the on site immutable backup was left, it was probably fair to assume the configuration backup(s) would be deleted. I did think about storing the file on the immutable backup device, but to speed up the process I went for a solution of writing the most recent configuration backup to DVD on a daily schedule. Using tao mode, only the latest backup is visible. Using DVR -R, it is still possible in Windows to format the disk, but the data is still recorded and can be retrieved easily.

[wishr]

That's an interesting solution, David. Thanks for sharing!

I would also add the procedure of verifying the hash of the config backup file on the DVD drive to the whole workflow to make sure the file is not corrupted and written correctly.

Another reliable solution is to copy/write the configuration backup to a USB stick/drive and disconnect the drive right after, then store it somewhere in a locked case/room .

[SAFA_IT]

USB is indeed an easier option, but I can't be on site every day to remove it. WORM USB media seems an interesting and neater solution than DVD. If I can get hold of one I might give that a try.