How to restore ENTIRE Windows environment if only Linux repos survive?

[StoopidMonkey]

We are considering blowing away our existing "standard" repositories and converting them to Hardened Linux Repositories. What I want to plan for is a doomsday event where ransomware wipes out our ENTIRE Windows environment but leaves only the HL Repos standing. Everything else is presumed wiped to bare metal with no cloud servers. In addition to those repos and functioning (but empty) server hardware), what else do I need at a minimum to restore our entire domain-joined Hyper-V environment and what is the order of restore? For example, would we need to rebuild a Veeam server first even before restoring Hyper-V hosts and domain controller VMs? Would we be able to create USB recovery media with the backup server on it? I've been trying to find documentation on this but almost everything presumes some servers have survived to run the Windows Veeam Console on.

[foggy]

Hi Chris, you can fire up Veeam B&R instance anywhere - on your laptop, for example - import Veeam B&R configuration backup (or add the hardware there and rescan the repository, if configuration backup is not available) and proceed with the restoration. Of course, Hyper-V hosts should be in place by that moment. There's also a possibility of using the extract.exe standalone utility. Regarding domain controllers recovery, please review this article.

[soncscy]

> ENTIRE Windows environment but leaves only the HL Repos standing.

Sorry to rain on the parade, but why such a weird scenario? Remember, if there is some attack on the environment, they will sit and wait until they get root/admin for __all__ elements, not just the Windows ones. With root, the immutable attribute can be removed; in that case, what good is such a scenario?

Hardened repositories are great and fine for most "basic" issues, but they're not a panacea! It's just a weird scenario to prepare for, and as I read it, I personally have worries on how you envision your attack surface, but maybe I'm misreading your post and misinterpreting your ideas.

But, in the off-chance that you're envisioning that the hardened repo will survive an attack, divorce yourself from this vision! Ransomware isn't an 'if', it's a "when", and it will take everything.

[mkretzer]

@soncscy You can in fact make a hardened repo very safe!

If you disable all remore management (SSH off, no hardware management controller, not timesync from non-secure sources, not timesync more than once a week with limited allowed time change, seperate network for the copy target) and if you regularly monitor your environment for config changes (changes to encryption passwords, immutability settings and so on) there are only few ways left to destroy these backups - most of them require some kind of security leak in Veeam itself. The whole concept is really good from my point of view.

Problem is that you have to go all the way, not using any half-measures. We for example were not happy that we lost all hardware monitoring if we want to make the system really secure. Thats why we have dedicated cameras monitoring the Veeam Hardware LEDs and using multiple spares for everything. Also, a classic VGA/USB Console attached directly to the server(s).

[StoopidMonkey]

> ENTIRE Windows environment but leaves only the HL Repos standing.

Sorry to rain on the parade, but why such a weird scenario? Remember, if there is some attack on the environment, they will sit and wait until they get root/admin for __all__ elements, not just the Windows ones. With root, the immutable attribute can be removed; in that case, what good is such a scenario?

I don't think it's such a weird scenario. The situation you described, the absolute nightmare scenario, involves a skilled and dedicated attacker that chooses my organization specifically as a target. That is very bad indeed, and I don't think there's any way of preventing that with 99% certainty without spending government agency levels of money on over the top security measures. What would you even suggest in such a scenario, where an attacker obtaining even cloud repository credentials should be treated as a foregone conclusion? Would even tape pass this test if the attacker is in physical proximity?

That aside, the situation I imagine as more likely is a less specific malware or ransomware attack that gets past our firewall and anti-malware defenses and propagates through my entire network at 2AM, destroying data on every Windows server it can find, including our current Veeam repositories. That is a somewhat common scenario, is it not? Investing in tape archives or green-lighting a sizable cloud storage monthly expense will take some time, and I figure I can take advantage of v11's hardening capabilities now but I want to do it right AND make sure that I know in advance what else I need to set aside on USB/whatever so that I can perform a complete restoration of our network on wiped-clean hardware assuming only these hardened repositories survive.

[soncscy]

>The situation you described, the absolute nightmare scenario, involves a skilled and dedicated attacker that chooses my organization specifically as a target. That is very bad indeed, and I don't think there's any way of preventing that with 99% certainty without spending government agency levels of money on over the top security measures

Not at all.

All it takes is just one mistake from any privileged person in your environment to click on the wrong thing and not really think about it.

Remember, just a few years ago, ransomware as a service was literally a thing; just access some .onion site, click on a few options, point at a domain, and a system would start just spamming out innocent emails to every address it could find on that domain until it got a hit.

My model is very simple -- your servers __will__ get compromised if they're connected, so everything either lands on air gapped storages which are removed/off-sited, or on software immutable options.

While Veeam's hardened repo is a great step in the right direction, keep in mind, all it takes is just a mistake from someone in your infra; a single notepad document with some passwords, someone reusing passwords for a password manager, etc. The attackers can sit and wait innocently as long as they want and it costs them nothing.

I strongly advise drop the idea that you need Mossad level tech to pull off a modern ransomware attack as it's just not the case and it hasn't been for years.

[StoopidMonkey]

@soncscy, sorry for the confusion as that's not what I was suggesting. I was suggesting it would take a large amount of security to prevent an attack by a human within physical proximity to our premises, which is less secure than a bank or government building. I know launching ransomware attacks is relatively easy which is what concerns me. Still though, sicking a bug like that onto our domain via email is a bit different than the presumed ability to obtain ALL admin credentials, be it for Active Directory, the Linux repos, a cloud storage account, etc. Our security posture has much room to improve so I figure Step One is to prepare for a Scorched Earth scenario as it relates to our Windows servers.

So let me ask: assuming your stance is that a total compromise WILL happen, what measures do you have in place to account for that?

[soncscy]

Sure, thank you for clarifying -- I might sound too direct, but it's never anything but brevity, I promise.

My answer to "you will be compromised" is a system that copies backups to air-gappable media as fast as possible. Tape is my preferred (it's just very convenient), but rotated drives and servers that get disconnected physically are my ideal.

The failure point for such repositories includes:

- lack of discipline: your #1 risk -- if you cannot trust your team to properly rotate tapes/drives, then in my opinion, you have no possible way too survive a ransomware attack. Tapes/Drives __must__ be removed as soon as the copy finishes. Tapes in slot are a risk! 8 TB drives for your most critical drives are __not__ that expensive, even a business running on non-existent margin must be able to go online and find a deal for usable drives
- Physical failures of the media; especially for spinning drives, you risk drives sitting in a safe/worse and not spinning up when you plug it back in, USB busses dying, etc. Remember that your risk is spread across every medium you place backups on and everyone who has access. I can name at least one instance where I convinced a client to spread their rotated drive copies across two sets of rotated drives, and it paid off! One employee who was scheduled to be let go erased the drives and claimed ignorance; having the secondary set with another employee allowed recovery during a ransomware attack.

I'm very dour on my opinion when it comes to ransomware, but it's not without reason as I've had dozens of very difficult conversations with clients after they got hit. I know it's maybe a fallacy, but it's very much so true -- ransomware __will__ find a way. I like Veeam, and trust their code, but I also trust that code will be violated in ways they didn't expect. For my mind, it's much more simple to assume the primary backups "will be compromised" and to plan secondary backups that are immediately airgapped as soon as written. For the high-priority clients I have, we simply add it to the rotation duties that any errors as a chance of the rotation are investigated.

It's a lot of work, but ransomware operators are far more willing to do more! You have to be more resilient than such operators. At worst, dedicating yourself/team to such a regime simply wastes < 30 minutes a day (surely, no more than 30 minutes). Isn't 30 minutes x 365 days far cheaper than restarting your entire business?

I'm happy to take the paranoid title, but I don't think there's a single day when I go to reddit/hacker news/slashdot and don't read a few posts on some huge company with the $$$ to stage a proper countermeasure to ransomware that gets hit by a ransomware attacker.

My logic is simple -- if the companies with hundreds of thousands+++$ can't protect against ransomware, what hope does a smaller outfit have? It's far easier and more reasonable in my opinion to treat it as an inevitable -- ransomware __will__ happen; the question is what will the attacker be able to hit? If it's disconnected, what can an attacker do?

Expect your primary backups will be hit, write policies that accommodate for this, and be wary of anyone who offers a software solution without independent research on vulnerabilities in their solution. And I mean independent!

[StoopidMonkey]

Yikes. I understand the reasoning, but yikes.

Anyways, so let's say you get whacked and have nothing but the air gapped tape that survives. What else do you NEED in addition to the VBK files on tape to restore the environment? A copy of VBR and the config file on USB to boot onto a server? Would the server have to be rebuilt first to install VBR on it first before the VBK restores are possible, or can you restore them while running VBR Recovery Media from a boot menu?

[soncscy]

To be honest, I'm not sure what you mean by "yikes" here. Maybe I'm lost across languages, but I'll assume the best for conversation's sake.

Yes, I think seriously because I deal with serious issues. I take no pleasure when I'm right about my clients who ignore the warnings I and my team give, and it's heart breaking when we get the call that starts with "So...we've been hit by ransomware". We do our best.

To answer your question, we do rsync's of the backup configs first to ensure that there is a secondary location for such things (Veeam, PLEASE find a good way to make non-file to tape configuration backup copies).

What we offer clients who elect to use us for storage or S3 targets is a "Recovery Service" option (there's a more expensive one-off emergency option, and what I feel is a reasonably priced package that basically assures we will have a Veeam server ready in 2 hours or less with recovery from public s3 storage providers).

If they don't have S3 storages, we at least provide credentials for the Veeam Server; the clients are guided through how to add a local server (literally any Windows machine) as a Windows managed server and restores begin. One of our offerings is storage on our Vmware environment, but most clients just nuke their VMware infrastructure and start restores.

On occasion, I've approved test restores to an isolated environment to ensure that the backups are valid.

But yes, you get the idea -- for the smoothest restore, you need a Veeam server, a target hypervisor or compute resource (EC2 or Azure), and a decent connection.

We never do Instant Recovery in such situations except for very urgent circumstances (my team personally doesn't have the infrastructure to absorb IO heavy instant recoveries for more than a machine or two), and we set the expectation that they need to accommodate for a full restore.

A 'naked' Veeam server is enough from the experiences I've had -- you just need to get the backups accessible and the target environment, and maybe a proxy or two. For physical machines, having up-to-date recovery media is great, and strongly recommended (this is __NOT a service we offer due to driver challenges!__)

All I can say is I've seen too many "perfect plans" be foiled by ransomware attackers; prepare for this scenario! Assume you're in the __worst__ scenario and you only have backups from a week ago; how do you recover!

This is the only logical position at this point in history, because you will lose your recent backups. You will lose your primary backups. You will lose your fastest restore options. You will lose your production environment. Accept your loss and build against it.

[StoopidMonkey]

Hi Chris, you can fire up Veeam B&R instance anywhere - on your laptop, for example - import Veeam B&R configuration backup (or add the hardware there and rescan the repository, if configuration backup is not available) and proceed with the restoration. Of course, Hyper-V hosts should be in place by that moment. There's also a possibility of using the extract.exe standalone utility. Regarding domain controllers recovery, please review this article.

One more follow-up to this: what if the backups are encrypted and both the exisiting VBR servers and the Enterprise Manager are lost? Do I need to have copies of the encryption keys or would just having the password used to create them be enough to access imported backup files?

[foggy]

Password would be enough.