Feature Request: Malware predictive Backup - No more Ransomware problems

[Poweruser]

Hi Guys!

Today i would announce you my newest Feature Request which is a Malware Detector by making predictive Backups.
So whats the Use Case?

Lets imagine we got Ransomware or any Malware which destroys Files.
In most cases these Software tries to encrypt much data in a fast time before we can notice it.
If a Backup Software works normally it will backup the crap, too.

The "intelligent" variant of a normal backup would be a prediction of what would be plausible to be backed up. So let me explain how it works:
Lets define a State of our System which is backed up as A, B, C. Where A is the State of our System at Monday morning, B at Tuesday morning and C and Wednesday morning.
Monday everything was fine, so we made a Backup at the night to Tuesday from A to B, which can be described by a diff(A,B).
Tuesday we were infected by Ransom123 and 1/4 of our Data is destroyed. The Backup runs at night to Wednesday by doing a diff(B,C).
By doing diff(B,C) the Backup Software itself notices, that significant much more data has changed from B->C than from A->B. So it will fear, that we have been compromised.
So it has to ask the Backup Admin (e.g. by supplying a List oft changed Files) if this was okay or not. Backup Admin flags it between good, unsure or bad. Maybe he could flag good, maybe good, unsure, maybe bad, bad. If he flags bad, the last valid backup and some previous versions get freezed so they never wont be overwritten. if a fuzzy value comes out, backup software will observe if its right.
if it flags good, it ignores it. maybe he could also flag as "rare event" or as "regular event". rare events is a ms patchday which is admin driven, regular event is normal work which occurs much more then seldom.
In this case the backup software learns if its more black or more white.

A Simpler Version: Dont go fuzzy, just say: between last 10 backups, the current backup has more than 20% of data changed then the arithmetic mean of the last 10, so it could be bad. and everyone can set his own value like 20%, 50%,.. This will be very easy to implement, because we already should have much delta data available.

Please Note: adding files would not be counted for this logic. just changing or deleting files. also appending files when the first part of the files is bit identical and the file is just getting longer its also ok (a logfile for example). if you obey these, it should work fine.

So now please give Feedback if you love it or hate it!

[soncscy]

Monday everything was fine, so we made a Backup at the night to Tuesday from A to B, which can be described by a diff(A,B).
Tuesday we were infected by Ransom123 and 1/4 of our Data is destroyed. The Backup runs at night to Wednesday by doing a diff(B,C).
By doing diff(B,C) the Backup Software itself notices, that significant much more data has changed from B->C than from A->B.

I'm not sure I understand -- how would you actually diff this? Diff (just thinking about the Linux util) looks at two separate files to know the difference; how would I as a backup admin really know if the difference between diff(A,B) and diff(B,C) is significant? What if I do dedupe or some DB replication on my servers? Or Garbage Collection for an essential app? Likewise, my understanding of an incremental backup vs an full backup is that all such diffs will always show "statistically significant" results, even though the essentials are not that different.

I'm really not sure what this gets you -- I am confident when I say the following: "My users will notice their inability to access data far sooner than my Backup Admin will when comparing Diffs"

That is, the backups run daily, maybe twice or a few times daily for a few critical systems. Ransomware sits in the background and does volume encryption slowly but surely, right up to the point it has full control and it just cuts off access.

Similarly, I'm not sure that this has been done before, but this system wouldn't stop Malware that just takes a VSS snapshot and presents this to users (it's easy enough to see what's shared, then expose the VSS snapshot and replace the share) and the Users would be none-the-wiser until it's too late. And I don't believe for a second Users would report this; Windows' default behavior is to tell them the doc is read only and to save to another location, and that's just what they'll do because they have too much going on.

Honestly, this just sounds like added pageantry which doesn't really do anything.

1. No one is alerted any sooner in this system
2. There is no mitigative aspect to this, just awareness, which could be done with simple scripting techniques instead.

[ITP-Stan]

The delta is in the statistics of each job run.
I think this could be a good feature.
When the incremental job is almost finished it can look at the delta (file size) of the incremental and decide if it's unusual (with a threshold you can configure).
If it's flagged unusual, Veeam B&R can decide not to create a reverse incremental (if that is the chosen method) or not inject the oldest incremental in the full, or not create a virtual full or ...
The backup chain would then be locked from manipulation, but new incrementals can still be made, until the backup admin confirms.
This should be a default-off feature.

[ChrisGundry]

Interesting idea, but I don't think this would be a good solution.
1. Although it prevents a backup of the encrypted data and alerts you at that point, it is too late. I think I would rather have a backup of encrypted files than no backup.

2. VeeamONE and other tools provide a mechanism to detect encryption while it is happening, which is obviously a better time to review the issue.

3. I doubt Veeam would implement a feature that would require manual user interaction. They are all about automatic backups, even automatic backup testing etc. By putting a process in place where a backup can effectively not happen, due to changes within the guest OS, I don't see them doing it. This would mean that if there was large change % for genuine reason you would either have no backup until you said "this is OK" or perhaps you would have a preemptive method to 'allow' this as 'OK', bit either way it would be a manual process.

4. Veeam works on blocks, not files, so this would be based on % block changes for a VM or % files changed? % blocks could be a single large file like a DB having a lot of changed blocks, but in reality this is only a single file being modified/encrypted. If you want to look at % files, which would seem like the most relevant metric to me then you would need to look at files int he guest, which is going to add a lot more overhead to the process.

Just my thoughts on the idea

[Poweruser]

The Backup will always happen! It justs consumes more disc space in this case and will spontanously increase the last n kept images.
Its a fuzzy increase of backup generations! It will always backup, but it will keep more old versions!
admin can decider later if he wants to free space after investigation, keep it or restore.

[ejenner]

This is a question of how to automate ransomeware protection within Veeam. And this is about 1 idea of a possible approach. I probably wouldn't be thinking along those lines.

My preferred starting point for where to begin a design for this would be to create a situation where only certain actions are permissible on the repository. i.e. gain some awareness of activities being performed on the files which contain the backup data. This might be an agent on a Windows based repository perhaps. Then a gatekeeper which decides if the activity being performed on a file is a genuine Veeam process or something else. If it's a genuine Veeam process it can be allowed, if not it must be blocked. A whitelist would be possible, in case an important 3rd party process is being blocked.

There are all kinds of crazy things you could do to prevent ransomeware. I agree it would be a good idea for Veeam to think about officially taking up the challenge to produce a product feature which can deal with this kind of attack.

[Poweruser]

You must notice, that you cant use some repositories, in deed you have to grant admin access to some software to work right.
So every user can crash everything and admin cant do things before.
so here veeam b&r needs to detect this and offer an undo.

also if veeam is compromised, too, the repository should be protected.

[jmmarton]

Interesting feature request. One thing coming in Veeam ONE v10 is an alert to let you know if suddenly much more data is being backed up, which could be a sign of possible ransomware infection. And since Veeam ONE alerts allow you to write PowerShell scripts to be called as an action, the sky's the limit on what you can do when this alert detects possible ransomware.

Joe

[Poweruser]

Veeam ONE is to be pushed, but in my case its a useless tool.
For only one Machine its too much, too big, to useless.

Thats why it needs to be in B&R.

[veremin]

You can always use Veeam ONE Community Edition for everything less than 10 instances. Thanks!

[Poweruser]

i already own a VEEAM One License included in the package with B&R called Essentials.

I dont want to use a big software, because i want to keep the system slim. one seems to be very resource hungry and complex.
also one software more to update.

anyway one can check but not control the backup process?
i need an proactive detector who helps without beeing there..

[wishr]

Hi Poweruser,

Thank you for the FR! Please let me jump in and share my thoughts.

In modern cybersecurity, there are four types of security measures which could be implemented to lower security risks to an environment:
1. Prediction measures;
2. Prevention measures;
3. Detective measures;
4. Responsive measures.

Each of these domains includes a variety of things that could be implemented: from processes and procedures to applications and systems implemented on various layers of your organization. It's a tough process that should be planned, documented, executed, and most importantly - maintained daily. To become safe in a fight against ransomware you should be utilizing the maximum number of opportunities from each of the aforementioned domains, luckily all the information and required tools have become widespread and accessible during the last years. However, keep in mind that it is impossible to become 100% protected - 0-day vulnerabilities and other types of security flaws are regularly discovered and that will never stop. It's like evolution: systems evolve, hackers learn on the field and become more and more experienced, while tools and approached for cracking the systems evolve too. If somebody will directly target your organization of any size for exploitation - trust me, they will be able to succeed in their mission.

What you are asking here for is a big red button that must protect your assets from being backed up once they have been ALREADY hit by ransomware. Don't get me wrong, but this is not what you need. If you would like to protect your car against thieves there is a variety of measures to protect from different cracking techniques, right? Most of these are PREVENTION measures. Your environment is way more complex than a car is, but the need is the same.

Additionally, it's necessary to be mentioned that, each environment is unique - change rates could be different across various applications, systems, and data, so for some users your suggestion might work, while for others it will just cause lots of false-positives. Also, ransomware will keep evolving and while currently, in some cases, the change rates may be an indication of a malicious activity going in on in a system, with the time it might change, so the feature might become useless quickly.

Implementation. The requested system might be quite expensive to implement because there are lots of ice under the water and keeping in mind the last paragraph above might not pay off at the end.

The last thing I would like to say is I'm not saying we are not looking in this direction here, at Veeam, but this is not something that we will start rushing to plan or implement. I'm just trying to say that such a feature will not resolve the whole problem. In order to solve it professionally, you should implement lots of stuff on your end: both technical and procedural things, including security knowledge sharing and security awareness for your employees that will make them keep the security aspect in mind when they are coming to their workplaces with a cup of coffee in the early morning or leaving for a quick smoke break.

Once again, thank you for the FR. Definitely an interesting topic to discuss.

Regards,
Fedor

[Poweruser]

here in the forum we have one problem:
mostly all powerusers are users for BIG systems.

when i talk i talk about SMALL companies with ONE server running and an admin who is a single figther.
an admin who doesnt do 100% fulltime admin jobs.
an admin who is on holiday and can do sth. remote or cant.
an system which needs to be NOT complex but effective.

yes you can do all the candy stuff and for a >100 or more computer office, it may be right.
i talk about 10-50 users!

[Gostev]

Our vision for 10-50 users companies is that the majority of such customers will end up having their data protection (and more commonly, the entire IT) delegated to a managed service provider, as this is significantly cheaper than maintaining own IT staff and management software - classic economy of scale. And we're putting a lot of focus into better enabling MSPs to provide such services using Veeam products.

While MSP model will certainly not fit EVERY small company out there, MSPs is where the "10-50 users" market has been moving to in general. This model also works better for Veeam, because we make no profit off of small companies (as those tiny deal sizes simply don't cover our sale and support costs).

[ejenner]

I think larger companies could benefit from additional features as well.

The argument for doing this could be similar to the arguments made in favor of introducing the feature that scans data before restoring it to check for ransomeware. Why would it only be a good idea to think about malware before a restoration but at other times not consider it?

Logically Veeam is a product for data protection. There's scope within that definition to cover protection of the data once it has been acquired by Veeam and put into a repository. Similarly again that encryption is offered for protecting stored data. It's a shame data can be encrypted more than once or this would all be covered already

[Gostev]

Just to clarify, my response was in the context of "keeping the system slim" and not wanting to use our monitoring tool (Veeam ONE) for what is clearly a VM monitoring task. As Joe noted earlier, the requested feature is already implemented Veeam ONE v10. And the added benefit of this approach is that in making the conclusion, Veeam ONE can look at many more VM metrics that it already collects - rather than just the incremental backup size.

[Poweruser]

maybe veeam one is the perfect monitor.

but thats one's problem.
we dont need a monitor, we need an actor!
if we monitor and dont change backup policy in an event, monitoring is useless.
i needs to increase old backup storage on warning signs.

what if i get veeam one reports and i dont have the time to check it or iam on holiday.
even middle sized or large companies have 1-2 admins only.

except the big ones..

[veremin]

Create an Veeam ONE alarm, then, and specify whatever script (such as increasing volume size or something) you want to as post-activity for it. Thanks!

[jmmarton]

To Vladimir's point, one thing I've discussed with partners and v10 is that a script could be written that does something like shutdown backup repositories if this new alarm is triggered. That way you block the potential for an ongoing attack to encrypt backups. It can also do a number of other activities--imagination is the limit.

The same holds true of our current "possible ransomware activity" alert that's been in Veeam ONE since we shipped 9.5. Not only can it send an e-mail, but it could possibly even text you by e-mailing your phone (e.g. in the US with Verizon Wireless, the e-mail address of a phone is phonenumber@vtext.com).

Joe

[Poweruser]

yes i can create a script..
if i would like to create scripts, i would use linux. everything free, everything possible..
but iam a lazy windows user.
i prefer to make some clicks and then it should do it ;-)

[jmmarton]

If you don't want to create the scripts, try our PowerShell forum and ask for help from others who may have already done it.

Joe

[Poweruser]

maybe i added an feature request, because i wanted veeam todo this for customers? ;-)

[Gostev]

Look - you're keep being offered immediate solutions to your requirements. Something you can do right away, without having to wait - potentially for years - until it is released as a product feature (if ever). So I really don't understand why you're sticking to this stance of throwing endless objections - instead of showing at least some appreciation to people who are genuinely trying to help you address your needs.

I mean, if what you're asking for is truly important for you - then the detailed perspectives and workarounds provided above are way more useful than a template "thanks for your feedback, we will consider it for future releases" reply. But in hindsight, may be this would have been a better first response in this particular case

[Poweruser]

yes, its nice to get a solution.
but i dont have the time to realize it.
it is a nice to have feature, and important if the accident happens.

but if i have to install and update veaam one, too its not worth including scripting.

thanks anyway
but realizing it by myself is far too much