I know you have gotten more answers than maybe asked for already, but... If you have backup server local account information, you technically can connect any computer over network to access files on the backup server it that account has access rights on shares. Unless you have removed all the system shares for all volumes as well, then it will be tricky.
Ok, let's break this down. Target on myself? First they have to find me, and they have to know what companies I provide IT services for. Good luck.molan wrote: ↑Jun 28, 2021 3:30 pm I hate to say this as I am sure you will react badly, but have seem to be extremely arrogant in your thinking there is no way someone could ever get in to your server without your password. Its like you want to call out all the trolls and hackers and put a big target on yourself.
The only way they are reaching a replica is by exploiting ESXi (already patched). Good luck.Also your replica's are 100% vulnerable. because of how a replica needs to function to be useful. The bad actor doesn't have to get access to the replica storage location to corrupt them. All they need to do is subtlety corrupt the source before you know there is a problem. The replica process itself will do the rest.
We could stop at "the second you are attacked" -- Given everything I have explained, how is that possible?This means the second you are attacked and a server is encrypted then your replica's are also encrypted and useless. Now I know you will say, but I can go back X number of days with my replica's, but it isn't good enough. Ransomware often lays dormant for this exact reason and will very probably be in those older images and will re-activate once restored encrypting your data gain. This problem exists for backups too.
Sound suggestions, but unneeded complexity. You only have to REALLY do the simple things I've outlined. Control your access well, don't make dumb mistakes.But to answer your question here is what I would suggest.
1. Immutable Backups. Both for on premise and your cloud copies ( you are storing copies off site correct?)
2. MFA on your backup server. Use a service like DUO mobile to add an MFA layer onto the RDP and console access of your backup server gives you an extra layer of protection beyond your password. and its free to use DUO for up to 10 users.
3. close all inbound firewall ports on your server except RDP. this a good step, but something you will have to constantly review as both windows and Veeam love to fill the windows firewall with holes every time they update.
4. better yet put a physical firewall in front of the backup server if you can.
I'm glad this thread sort of blew up and other users echoed the concerns I had in response to the TC's commentaries. I 100% agree with your position, as it's impossible to claim it's not possible to be attacked if your machine is in any way connected. Just a few days after the original post, I came across this Hacker News Discussion:With all due respect dude, you're coming at me like I'm some sort of novice. I've been doing this for over 20 years. I have the correct security posture. You know as well as I do that zero day vulnerabilities (from an external port) are rare, just because there was one recently with Exchange doesn't mean they come around often. Look at the big picture.hoFFy wrote: ↑Jul 05, 2021 6:38 am Hi Rob,
I don't know how to tell you exsctly what I think, maybe because I'm no native speaker, but let's try.
I'm working for an IT service provider with very small customers up to up to customers with about 1000 employees. Over the last years I've seen and heard a lot. It starts from "why do I have to by anti-virus licenses for my servers? Only the clients are receiving emails and surfing the web" to "I only want to by 2 USB drives, they are so expensive. Two backups are enough".
At the moment I'm working with a new customer who called us in is darkes hours: Hafnium hit him hard. He patched his Exchange 2016 Server on March, 13th. But only with the CU19 from december and without installing any further security patches. Within days webshells have been deployd but nothing more happened (from the customers PoV). In mid-June he was wondering about slow server performance and called his old IT service provider who found out that he has been infected, but he couldn't help him "because lack of time"..... So the customer has a cyber security ensurance, which called a IT forensic company whose experts imediatly told him to shut down all systems. Thats where we joined the game. Whe had been called to extract all data for a deeper analysis and be some kind of remote hands for the forensic company. Together we found out that the customer was under attack since early April. Anti-Virus software has beend continuously deleeting web shells on the Exchange Server("why should I install AV on my servers?" you remember?). But one day the got through, were able to delete anti-virus software, extract admin credentials, walk through the servers and install crypto-mining software on every server. In my opinion the attackers had all aces in their hand and all they did was installing crypto miners, which worked with 100% CPU usage... only a matter of time until somebody wonders why servers are so slow.... Having a good backup might be enough by just doing a full system recovery and be happy. But we found out that the typical 14 days retention policy wasn't enough. Attackers where in the system even before the oldest restore point and at that time the domain controller already has fallen. EVERY serious security expert will tell you that its not enough to recover your data which was already infected, do a full system scan with one or two anti-vrus scanners and go online again. At the moment we are rebuilding his whole environment from scratch.....
I know this isn't exactly what you asked, but it's an example on how things can go, if you don't have a high retention policy. And even then: Only a few companies are able to recover from restores with month-old-data. Not only the loose of reputation by their customers when calling ("hello sir, yes we had an attack and lost all our data of the last 4 month. No, we also lost all construction data for our $$$-project. Yes we have to start fom scratch. Are we still friends?..") but also the loose of data, value and so on over the last month... all will be lost. Only a few can really think of what that will mean to their business.
Second example, which I only heard of: Backup server, not on the domain. Backup copy to a seperate place, in case of fire. Backup to cloud, because of... its better and the evil hackers.... All kind of security soft- and hardware. Small company, 30 employees, working 100% digital and relying completly on their IT. The firewall / router hasn't been patched agains a new vulnerability in its VPN-stack, attackers had been able to enter through the VPN login of the manager. By the use of software like Mimikatz they were able to extract cached login data and made their way through the servers with high privileges. It took them several weeks, but in the end thei made their way to the backup server (RDP from one of the domain joined servers which had been use by the IT service provider), changed the retention policy in the jobs to be sure that there will be no valid data, then deleted every backup they could find and had direct access to. Then they started to encrypt every data on every server. When finished they asked for Bitcoins worth 250.000$. They were so professional, one had to open a page in the dark net, start a chat with the attackers (which by the way hat their own opening hours for their chat so you had to wait for them) and then you had to talk to them. They told you very directly how they were able to get into your environment and what they found out about this company over the last weeks just do demonstrate that they have you by the balls.
This all ended up in paying the attackers (much lower than the original 250.000$), because no one was able to help them. They got an encryption key and got their data decrypted.
These stories should tell you something, may be you should try reading between the lines. Don't be so arrogant as you have been in this complete thread! There will ALWAYS be a zero day vulnerabiltiy in Windows, ESXi, your NAS box, firewall / router, all you can think of. Air gapped should mean there is no open port to nowhere. If you are running physical backup servers at all your small business companies, then this one should not be able to connect to by RDP, VNC, other software or even BMC. And if there is a NAS-box as abackup target: Only connect it to the backup server. To nothing else. Even not to a network switch. Or do you think a network switch is safe? They also have vulnerabilities or standard credentials like admin : admin allowing the attacker to use port mirroring to catch network packets which allow him to extract login data (when no encryption is being used). Your backup servers should also not make use of remote managemt software. Have you heard of SolarWinds...? In the case of having to get in touch whith the serves you should need to walk to them, connect a keyboard and mouse. There should be NO other option. And your retention policy.... long enough, but not that long... I already mentioned above. Customers always have to ake use of every possible security option, starting from anti-virus software, over MFA, strict security policies, etc. etc. . And in the end attackers will always be one step in front of us. The only thing you can do is to make live for them very hard, because they are trying to cath the low hanging fruits and make the most money out of them. Secure environments, where they have to circumvenct several lines of security, might be of lower interest of them passing by and attacking the next one.
But again: Never be so arrogant to think you are safe just because the server is not domain joined and your ESXi has been patched. There will be one day where the systems are not patched the same day, or patches are not already released. And then there is the real chance that you'll be hit. Have you heard of Microsofts printer nightmare? There's no patch at the moment, but one can create an privleged account an run malicious code.
One thing I'd like to point out about this, not to get too off topic.. Is that part of the blame for Hafnium is on Microsoft's shoulders. I bet LOTS of those Exchange admins weren't aware of the fact that Windows update does nothing for them if they aren't on the latest CU. It's a poor design in my opinion. Windows update is supposed to update all windows components and Microsoft software. That's what a lot of people believed, at least.
ESXi is not invulnerable - REvil RansomWare is now targeting ESXi.
I do not know the exact technical implications here, I just wanted to show that it is not impossible to attack ESXi. I don't think that there must be another linux system in the network - like someone already stated, attackers will infiltrate the system and then first keep their heads down and watch, check what they have, gather information (like credentials). And if they are nested on a Windows machine and then see an ESXi server, I guess they do not need a Linux machine to start their ESXi exploit. And this exploit is not patched yet as far as I know, it just appeared in the end of june.robg wrote: ↑Jul 06, 2021 7:05 am And how does the REvil ransomware get itself onto the ESXi server in the first place.. Does a negligent admin need to allow it to get there?
There's a new linux variant of revil. What if I don't run linux anywhere?
Or are you talking about the SLP-based ESXi vulns that were patched in February?
This is what I meant: you say it is a matter of perspective, I would say it is a matter of definition. You define air gap as "System that is very hard to access due to protected credentials". Most people in this thread define it as "System that is physically seperated from any network" - and the latter is the definition of "Air Gap" in RFC: "
(see https://datatracker.ietf.org/doc/html/rfc4949)air gap
(I) An interface between two systems at which (a) they are not
connected physically and (b) any logical connection is not
automated (i.e., data is transferred through the interface only
manually, under human control).
What I meant to say is that in my view your definition of Air Gap differs from my definition and - as I read it - form the RFC definition. You say it is impossible to access the backups from the network, "unless A, B, C, D, and E occurs". My Air Gap definition says it is impossible to acces the backups from the networs. Period. No "unless". Because it is not connected to the network. When I put the tapes back in the library, there is no air gap anymore - altough you could say that the hacker would have to take control over the library and the drive, which in my eyes is of course possible.
I never said that air gapped backups cannot be compromised! The can be stored under water pipes, next to the copy paper storage in the smoking area, or next to the hughe company electromagnet... I very well know that air gapped backups can be compromised or get lost - this has to be prevented as far as possible by appropriate measures. But this was not the topic here - the topic is if your design is air gapped or not.
I hear what you are saying. My point is still that those credentials can be exploited. With a true air-gap it doesn't matter if you have the keys to Fort Knox, you can't get to it over a network.
The only legitimate "way in" would be via the console, but hackers don't take that route, do they..Gostev wrote: ↑Jun 28, 2021 11:36 pm Except "properly deployed" hardened repository means disabling the SSH Server, which makes all your SSH key concerns irrelevant. With the only "way in" being the local server console, you can forget about any possibility of remote attacks, even if hackers have root credentials to your entire environment. They simply can't use them remotely, that's the whole idea!
. V11's chattr +i feature provides us one such tool which is indeed nice. The next should be B&R for Linux Doesn't repeating make a wish come true? so I would rather worry about bugs in that tiny immutability flags management process, which does run under root. However, it implement just 2-3 functions of a few lines of code each, which reduces a chance of vulnerabilities significantly... still without making them impossible, of course.I said "poor man's air gap" - not a literal one, but something that is almost as good in a pinch, if it's done correctly. I was pretty clear in how I defined it!
Sure, we can disagree on the wording, and I understand how precise IT people prefer to be, but I am more loose with my language. If something is "damn near impossible" in my eyes, I'll take the leap and say it's impossible.yakamoneye18 wrote: ↑Jul 07, 2021 7:39 am What I meant to say is that in my view your definition of Air Gap differs from my definition and - as I read it - form the RFC definition. You say it is impossible to access the backups from the network, "unless A, B, C, D, and E occurs". My Air Gap definition says it is impossible to acces the backups from the networs. Period. No "unless". Because it is not connected to the network. When I put the tapes back in the library, there is no air gap anymore - altough you could say that the hacker would have to take control over the library and the drive, which in my eyes is of course possible.
I never said that air gapped backups cannot be compromised! The can be stored under water pipes, next to the copy paper storage in the smoking area, or next to the hughe company electromagnet... I very well know that air gapped backups can be compromised or get lost - this has to be prevented as far as possible by appropriate measures. But this was not the topic here - the topic is if your design is air gapped or not.
And again - I am not saying your design is bad! Your design is much better than what I had to deal with before, when no money for tapes was available. I am not criticising your setup - I am only disagreeing with your wording.
Fantastic the air gap project!NaplesDave wrote: ↑Jun 28, 2021 1:24 am If you want a REAL AIR GAP device check out this https://www.techbyking.com/project-gallery-page. Scroll down for AIR GAP Device.
The device sits between your NAS or external backup drive and physically disconnects the ethernet from the device during your specified backup time.
)Users browsing this forum: Semrush [Bot] and 28 guests