power on just for backup

[ejenner]

Hi guys,

A while ago a solution for keeping backups safe from ransomware was suggested. Keeping the repository offline for the majority of the time and only turning it on for backup to take place.

Has anybody implemented this?

I'm interested in the different ways people have achieved this if anybody has done it.

[Rick.Vanover]

TL;DR: It works. Just automate repository up and down.

I do this all the time myself, and there are a number ways to do it. Here are some tips I've learned in doing this:

1. Automate the power-on of the repository for a small amount of time before the backup job. I do 10 minutes, meaning, repo is on 10 minutes before backup job (allows components to all be connected when job starts).
a. You can automate the power-off to be approximated to runtime, or put it in scripts after the job is done.
b. I am using managed power switches with a schedule.

2. Consider the backup target being a NAS in that situation. Meaning, you don't have as much hassle like Windows Updates coming during your startup that may interfere with the timing in #1.

3. Still have a backup job to a persistent repository. When I do the offline repository, I also have a backup to a regular repository that runs either the same times or more frequently. This is more for restores happening on-demand/ASAP.

What, in my opinion, seems to be one of the most helpful parts to something like this are the managed power ports and optionally scripts. The multiple jobs is a nice additional safeguard as well.

[Rick.Vanover]

I have also heard of some people accomplishing this on managed switches by setting time range settings on the port(s) needed to access a server or storage system. This may be slightly more desirable compared to powerup and powerdown cycles on a storage device, which I recommend RAID or other resiliency as a given however.

[ejenner]

Thanks for the tip on managed network ports and power switches.

I've picked up a few more ideas since yesterday:

1. Change the backup storage path. Don't use the default path to the backup storage.
2. Can the disks be encrypted or have restrictive permissions? Would that prevent forms of attack like deletion.
3. Set up a firewall so only the required ports are open. Can the default Veeam ports be changed?

[veremin]

All of those steps while making your environment less variable would not protect you 100%.

You need to consider true air gaped backups, such as ones that can be guaranteed by tapes or cloud repositories. More information can be found here.

Thanks!

[ejenner]

Some good info there.

I guess ultimate balance of convenience and protection would be to make sure only one process has the ability to read or write to your safe repository. It would be a Veeam process running on that server or maybe Veeam processes from other parts of the network. But only the Veeam process and not any version of malware.exe

[veremin]

This would not protect you against malicious administrator or against attackers that decide to log in to backup server and remove existing backups, though. Thanks!

[ejenner]

I think that's covered by the Computers Misuse Act 1990. i.e. if a known sys-admin goes ahead and severely damages a network he should expect to be prosecuted for a criminal act.

The only way I can think of technically countering the problem of a malicious administrator (who does not fear prosecution or is able to cover their tracks) is to make deletion a two step process which has to be handled by two separate people. For instance, having a backup location with a cloud provider configured so administrators on the client side cannot delete all the remotely stored data permanently. Being managed by a different company would create a checkpoint.

As long as you have administrators on the network with overall control, which is necessary then you have to trust they won't break the law.

[veremin]

Tapes and cloud repository with insider protection feature enabled answer those requirements. Thanks!

[ejenner]

Tapes = only if you have a manual process for taking tapes out of the autoloader
Insider protection = on Enterprise Plus with a CloudConnect SP/Tenant relationship? - i.e. if I am the tenant and the service provider I can delete whatever I want?

[veremin]

Correct, tapes would work, if exported regularly.

As to cloud repository, I meant a case where you're sending backups to SP of your choice, not the one when you're your own SP, of course.

Thanks!

[janezk]

Hi,

another scenario, that is not mentioned... And I don't know for sure if it can be achieved in this way? (maybe I'm missing something)
To have a physical server (could be rather old - recycled), with local storage that pulls backup jobs to its local storage from VBR storage.
It has to be invisible in the network... so no one basically knows that it exists! (can this even be done???)
Prerequisites:
- It's not joined to the domain
- has all ports closed, except the ones needed for pulling backups, just for the time it does the copy.
- network paths opened only in the desired direction for pulling backups... ( no updates or anything)
- admin has to access it trough local console in DC! ( YES, it's not old school, it's good for your health that you do a small walk from time to time... )
- ?

How to do this? different options could be implemented. Maybe the simplest one is some kind of linux distribution and some copy tool with a scheduled scrips...
or maybe a win server and robocopy and some script...

This idea has to be evolved and tested, as it's now only a concept... But I think it could be done in a way or another...

BR Janez

[ITP-Stan]

About your concept.
Most NAS devices have multiple network ports. Most servers have them too. So this may work if you use a NAS or a server as your backup repository.
A cross cable from your recycled server to a separate port on the NAS. Configured with static IP's without gateway.
The server has a software firewall prohibiting all incoming traffic.
The server then copies (pull) VBK/VIB files from the NAS to it's local disks and stores it in a folder with date/timestamp.
The server keeps x versions of these folders to protect from empty or corrupted VBK/VIB files being pulled in.
You would need lots of storage on the server to keep all the VBK's and VIB's. Unless you use reverse incremental, then you could only copy the VBK and not the VIB.

[DDIT]

Another easy trick is consider is a pre-job script and post-job script, to enable/disable NIC adapters, limiting the time the device is accessible.

[ejenner]

That would work quite well I think.

[OrangeWing]

Hey DDIT, do you have a copy of the scripts used for the NICs? That is what I was thinking of as far as limited time without having to power cycle everything.

[Rick.Vanover]

@OrangeWing I bet @DDIT is referring to, or at least one way to do it

netsh interface set interface "Ethernet" disable

(and corresponding enable)

Where "Ethernet" are the one or more interface names enumerated this netsh command:
netsh
interface
show interface

There may be PowerShell other network management options, but netsh is still a solid choice.

[DDIT]

Yup, pretty much a one-liner, but in PS:

Enable-NetAdapter -Name "NIC_Name" -Confirm:$false
and
Disable-NetAdapter -Name "NIC_Name" -Confirm:$false

I renamed my NICs before doing this so it made sense in my deployment.

A side point: Since moving to 9.5u4 and migrating my repo to SOBR (I'm not sure which of those two things triggers this) but I get a daily email from Veeam around 9am with a report of my SOBR showing status, capacity, used space, free space %, etc. I'm not sure how this report can query my SOBR if network access is off most of the time...confused.

[Rick.Vanover]

@DDIT - I believe it is based on last status of the repository. I've had this where repositories go away from my lab and such but remain in the configuration. There should be a corresponding indicator that it is missing/offline in the B&R UI also.

[nitramd]

@DDIT, follow this thread to eliminate the SOBR spam: veeam-backup-replication-f2/disable-dai ... ml#p313223

You'll be making a registry key entry.

[DDIT]

@nitramd, thanks! - very helpful

On a related point (and while I'm here), I also get an 8am daily email reporting on the status of my Veeam Agents Protection Group. I have 6 endpoints in here; 2 servers, 4 workstations (of which, 3 are PCs and 1 laptop).

The daily report either warns of my workstations is offline (it's a laptop - kinda expected), or, "Error" - one of the workstations needs a reboot as "Task failed. Error: System reboot is required to continue installation". What installation? This was working fine last week - no complaints! So yesterday, I asked the user of this workstation to reboot. This morning, I got the same error!? It's worth mentioning that the daily backup task on this workstation completes successfully so I'm not sure what this report is complaining about.

[nitramd]

A request for a reboot usually means that additional software has been installed - it's typically .NET that is installed.

[DDIT]

Well, having looked back at the emails I received, this "error" appears on each of the last 3 Tuesday's emails. Is there a .NET update released that frequently? I'll keep an eye on it and open a case if it gets annoying (I'm half way there)

Also, is there any way to get Veeam to not send a warning email when a workstation is offline? It seems likely that some workstations will be laptops off the network, or even PC's switched off. Or, at least choose the time this email is generated - default appears to be 8am, whereas most staff arrive at work after this.

[foggy]

The time for this report can be configured.

[DDIT]

@foggy, thanks. I just check these settings on the protection group - the notifications are NOT enabled - how does that explain the daily emails?!

Anyway, I have now enabled them and set for 12:00 noon so I'll see if this takes effect.

[DDIT]

Hello,

Just an update regarding the pre- and post- job scripts we were using. It seems they were the cause of random failed Backup Copy Jobs, which complained about RPC communication errors. Disabling the scripts for a few days confirmed these were issue. As support were already involved in troubleshooting these RPC issues, they suggested adding a 5 minute wait to both scripts to ensure the network is fully up before starting the job, and to allow time at the end of the job to finish completely before disabling the NIC.

However, we have overlapping Backup Copy Jobs, so its possible that two jobs are running concurrently, but the first to finish will disable the NIC, impacting the other job. Therefore I need the script to check whether any other BCJ's are running, if not, disable the NIC, if yes, then do nothing.

Having not done and PS scripting with Veeam, does anyone have a quick answer?

[lando_uk]

I was thinking, hardware snapshot on the SAN can protect your repositories against a lot of threats. Locked down, 2FA on the SAN admin account. Any hacker, even if they were on your system for a long time, figuring stuff out and collecting pw's probably wouldn't think about or find the SAN snapshots.

Protecting against rogue admins is another story.

[Rick.Vanover]

@lando_uk - I consider SAN snapshots "semi offline" - also floated replicated VMs as a similar technique (especially if things like different credentials on hypervisor are in place). In fact, got into an argument Gostev on it . He was right in the end, as usual, the absolute most resilient approach is something with air gap and offline.

[lando_uk]

For any enterprise over a certain size, the practicability of having to restore everything from tape would be mind blowing. You can't truly test it or run DR exercises like you can with a site failover etc. Its one of those things that might make you sleep better knowing you have all your data locked away in a mountain somewhere, but the reality of being offline for potentially weeks/months for a restore would put you out of business. It's why many orgs secretly just pay the ransom, even if they do have the backups.

Today, years after the first crypto, this kind of attack shouldn't be able to happen in 2019. Smart people should have solved the whole ransomware issue by now.