@A.J. - I agree with your feature requests. Keeping Malware events consolidated to a single Node I agree would be highly beneficial. Good one!
Gostev wrote: ↑Mar 08, 2024 11:19 am
Also there's finally some good news on the inline encryption detection diagnostic: the dev team seem to have found a reliable way to match encrypted block to a file and they will start building this tool next week.
@Gostev - REALLY looking forward to this being released...can't wait! Please thank the Devs for their efforts!
@BackItUp2020 - yeah...when marking restore points as clean after performing forensics/scans, it would be nice for subsequent backups to not trigger Malware alerts for the same file/event. I think this was discussed earlier in this post thread and the PMs are looking to add this feature request 'soon'. Agree with you there.
BackItUp2020 wrote: ↑Mar 11, 2024 4:29 pm
For us in particular, we would need to mark as "false positive" on a per-server basis and have those stick for the next job. Prior to disabling this feature, backup jobs would just set off the same 20 alerts every night, which would make our CS team have to look into every time, then do a scan with other scanners, and inevitably ignore it.
Can you please clarify what type of malware events you are getting? For indexing scan any change you've excluded the false positive extensions from monitoring? Thank you!
For us, its the malware extensions alert. We have a host of applications that use a bunch of these extensions legitimately. While we can exclude them, the exclusion is global which is not ideal. So, we just turned the malware stuff off until it's a little more mature.
BackItUp2020 wrote: ↑Mar 11, 2024 6:54 pmWhile we can exclude them, the exclusion is global which is not ideal. So, we just turned the malware stuff off
Apologies but I don't follow you here: if global exclusion of a just few extensions is a problem, then how is globally excluding thousand of extensions a solution? As this is what you did by disabling the feature entirely.
We have many other levels of malware detection in our system and the Veeam one is just too noisy as it sits, so we just rely on the other methods for now. I'd prefer to exclude by path per server, followed by per-server, followed by globally. I'd rather do that then have a ton of global exclusions. I do look forward to reenabling it at some point just to ensure things are happy at the repo level.
Gostev wrote: ↑Mar 11, 2024 8:04 pm
Apologies but I don't follow you here: if global exclusion of a just few extensions is a problem, then how is globally excluding thousand of extensions a solution? As this is what you did by disabling the feature entirely.
Alert fatigue is a real thing, same as crying wolf. Anyone who cares about this feature already has AV running on their servers so it's got to add value for them not just be another email to be deleted every morning.
Unfortunately, the license for the Free Edition is not sufficient for a report of our size. And using a purchase license is far too expensive and no one pays us managed service providers for that. In my opinion, the report must also be possible in Veeam itself WITHOUT Veeam One. Unfortunately, the price/performance ratio with Veeam One doesn't suit us as service providers.
Maybe Veeam should think about releasing a free Veeam One Edition for B&R customers that delivers exactly the backup reports... and ideally without MS SQL.
But I'm pleased that Veeam, as always, has an open ear for its customers and I can see from the postings here that something is moving and that we will certainly soon be out of the beta phase for malware detection.
This is also the reason that we don't want to do without Veeam even if we switch the hypervisor away from VMWare.
jandrewartha wrote: ↑Mar 12, 2024 2:52 am
Alert fatigue is a real thing, same as crying wolf. Anyone who cares about this feature already has AV running on their servers so it's got to add value for them not just be another email to be deleted every morning.
Thing is, simply excluding those few extensions belonging to legitimate apps in their environment would have stopped any and all emails. While still keeping another thousand of extensions under monitoring.
Also, I don't know many AVs which detect and flag random files getting encrypted, bulk file deletions etc. in any case? While a few do, the majority of AVs still focus on detecting threats in executables. And when they miss some zero day threat due to its signature not yet present in their definitions, or simply because the malicious executable runs on another machine and performs it's operations remotely, then Veeam will still detect encryption and/or mass file management operations.
Plus you always want your protection strategy to be multi-level in any case. As for example, the new malware reported by our research team last week disables AV software found on the machine as one of the first actions. So, it's a mistake to think AV is the ultimate answer and you don't need anything else.
SentinelOne absolutely detects cryptolockers and reverses it live, I've seen the demo. Or more generally it'll stop it before it even happens. CrowdStrike is just as good or so I've heard. Yes, for companies that don't pay for proper EDR, the email alerts in Veeam are probably useful, but they're not the ones disabling it because of false positives taking up SoC time.
Why spend time disabling parts of a feature when you don't trust the entire thing not to start wasting your time again? Easier to disable the whole thing and come back to it later once the functionality is improve.
Gostev wrote: ↑Mar 11, 2024 4:24 pm
Well, we happen to be a backup software, not an antivirus software however, could you give me some specific examples? I'm just curious to see how do they format the output, considering there are thousands upon thousands of files getting changed daily on every machine by all sorts of apps.
Well I know Acronis Cyber Protect (both Cloud and on-premises versions I believe) does have that ability, and they grew to offering that from at one point just being a backup software as well. So I do understand if Veeam doesn't do it yet, but perhaps in the future, with an in-VM (or physical computer) Agent it could be implemented. (Which is how Acronis does it.)
As for the specifics of how the format looks, I've never actually had such an alert for malicious activity by a specific process. So I can't say exactly how it looks myself. But there is an option to allow/block specific processes to modify/encrypt files in the ransomware detection settings. If you share a fairly specific test method, I could potentially do a test in our own testing environment to share the output. Though I suspect Veeam has licenses for competing softwares already for comparison purposes.
Thanks Tim. Right, for backup vendors coming from legacy agent-based backup world it's perfectly doable because they already use agents for most things. But 9 out of 10 workloads Veeam protects these days are protected in an agentless way. So requiring an agent just for the file activity monitoring just not going to cut it. Also, I would argue if you're ready to install an agent in every production workload for real-time threat monitoring, you're better of implementing a proper EDR solution for whom it's the main business, as opposed to a backup product with some rudimentary capabilities of real-time threat monitoring.
Because of this, we plan to expand our related functionality into a perpendicular direction (stuff you can NEVER do with EDR or AV solutions) vs. just becoming essentially a worse version of them with agent-based real-time monitoring.
jandrewartha wrote: ↑Mar 12, 2024 3:48 pmWhy spend time disabling parts of a feature when you don't trust the entire thing not to start wasting your time again?
Because it won't waste your time again the moment your exclude your legitimate business apps from monitoring. It's not like your LOB apps change every month.
Otherwise it's a bit like throwing out a phone just because it "wastes your time" asking for a PIN code to unlock by default, and refusing to instead simply disable the PIN code request in its settings. As well as change some other default settings to your liking.
Makes sense, the Acronis agentless malware detection (scanning of backups) does not have the ability to isolate modifications made by specific applications to my knowledge, for that particular feature, and the above mentioned reversal/blocking of ransomware that SentinelOne and Acronis have, the Agent is required. Agentless systems don't support those features.
So from that perspective, I agree, if Veeam's goal is just to be an additional layer of protection in a sort of "air gapped" manner, something malware active on an infected computer couldn't interfere with, and not to be a replacement for AV or EDR software, then it makes sense to keep along the same path things seem to be going now.
I do like seeing the additional functionality being added, though I do agree with some of the other comments about certain functionality regarding exclusions or marking files as safe needing to be improved a bit, but I would expect improvements to be needed of some sort in any newly released product, which this essentially is, new feature for an existing product anyways.
I do like seeing the additional functionality being added, though I do agree with some of the other comments about certain functionality regarding exclusions or marking files as safe needing to be improved a bit, but I would expect improvements to be needed of some sort in any newly released product, which this essentially is, new feature for an existing product anyways.
I'll contribute that I've got a VM backup that constantly alerts for potential ransomware note, but doesn't log the file/extension in the console, like other alert types.
I found in the C:\ProgamData... log file for that machine that it's related to a filetype *.wiki and have checked and in this instance the detection is a false positive.
Given that I can't exclude this extension for the one machine, and I don't wish to exclude the entire machine from scanning, the best alternative option appeared to be to add the *.wiki file type to the global exclusions for malware detection, under the trusted file extensions in file masks to monitor.
However, this doesn't seem to have worked at all, as I'll receive the same detection on the same VM when the job next runs.
Is this expected behaviour, or have I encountered a bug?
@VinnieNZ - file exclusions are only for File Indexing scans...not Inline Entropy scans. The Malware Detection event you received was for the Inline Entropy scan engine. There is no granularity with that particular engine at this point, sadly. As one who also uses Inline Entropy (I don't have guest indexing enabled in my jobs, so I don't use File Index scans), I'm curious what particular log file you were able to detect the file type in? The only log file for Inline Entropy scans is in C:\ProgramData\Veeam\Backup\Svc.VeeamDataAnalyzer.log file. And, from what I've been able to see in this file for several VMs Veeam has detected possible issues with, I've not seen any specific file listed in this log. I've only seen similar dialog as is shown in the Malware Event window from the History node.
Thanks.
I found in the C:\ProgamData... log file for that machine that it's related to a filetype *.wiki and have checked and in this instance the detection is a false positive.
Andrew, can you please share what type of the app is using wiki and how many files were detected?
The Malware Detection event you received was for the Inline Entropy scan engine.
Shane, inline scan does not provide information about extensions, so this one should be created by the indexing analysis.
Hi @Dima - Andrew shared he had a "Ransomware Note" event...which is an event for Inline Entropy. Yes, Inline Entropy does not display file extensions...I'm certainly aware as this has been a pain point for me in my environment ...and, which is why 1. his statement confused me, and 2. why I asked what log file he used to determine the extension which seemed to have caused his event because...as we know...Inline Entropy doesn't show extensions. Unless, his 2nd statement was in regards to a different event, but he didn't explicitly state so? And, this may be the case...he just didn't share he had a separate event for Guest Indexing?
All is correct. My logic is quite simple - whenever you can highlight the extension from the malware detection event in B&R, that's an Indexing Analytics causing disturbance in the force
@Dima - thank you! I pinged Anton on X to see if that was the case & awaiting his response. Can you confirm if the Inline Scan updates were applied we talked about?...specifically, if Inline Entropy events show a more detailed info (i.e. file or directory location)?
Thanks for all the work!
We’ve been working on a standalone utility to help you guys with inline false positive events investigation which should show the path to the suspicious files. The utility is being verified by QA team now, so we will publish it soon as a standalone tool with all the needed details / how-tos. Stay tuned, I’ll update this thread once we have it available.
this particular server is a new installation of server 2022
The false positive was raised on this installation after the update to 12.1.2, correct? Any user activity on this machine or that was a default installation from the default image? Thank you!
We’ve been working on a standalone utility to help you guys with inline false positive events investigation which should show the path to the suspicious files. The utility is being verified by QA team now, so we will publish it soon as a standalone tool with all the needed details / how-tos. Stay tuned, I’ll update this thread once we have it available.
Dima P. wrote: ↑May 23, 2024 12:18 pm
Hello SnakeSK,
Thank you for the feedback, investigating!
The false positive was raised on this installation after the update to 12.1.2, correct? Any user activity on this machine or that was a default installation from the default image? Thank you!
I dont think its a false positive per-se, it is definetely false positive since user cannot control WHAT website loads and what is stored in the cache, from a system standpoint, the .onion link is positive.
I just hope there would be a way to disable the .onion scanning and keeping the inline detection intact.
We scanned after upgrade, it may have been there before upgrade, for this particular account we have browser delete enabled in the gpo, so history, cache, passwords are not saved upon browser close (for all admin accounts), form of a DLP, yet the data were there. This cache folder also appeared under MSedge.
Thank you for the update! We've checked and onion links are not somehow included in any browser configurations or cache. In order for those to appear in the browser cache user must type in the onion link and access it in the browser.
user cannot control WHAT website loads
The ones that you've shared on the screenshot are used by different social networks specifically for anonymous access. For others it depends: we are aware of not many sites that redirect you to onion / tor network; those we know, I believe, should be reported as suspicious activity anyway.
I just hope there would be a way to disable the .onion scanning and keeping the inline detection intact.
Dima P. wrote: ↑May 22, 2024 7:59 am
The version 12.1.2 was just published and it contains lots of improvements based on your feedback. Thank you!
This is good news, especially seeing this in the release notes:
Malware Detection
Added the ability to exclude specific file paths from suspicious file system activity analysis.
To be clear: there was a lot of talk in the thread about false-positives, but the reason we can't just exclude certain extensions to deal with those false positives is because doing so creates blind spots. Those extensions were included in the scans and alerts for a reason, and turning off even one or two out of the many malicious extensions because we use them in legitimate ways in a few places leaves us needlessly blind if those extensions are eventually used elsewhere in malicious ways. We needed the ability to exempt just the specific known-valid uses.
Ideally we will also eventually get the ability to disable certain extensions on specific servers or within specific folders, so if we have an app that could create files with suspicious extensions repeatedly, but those files tend to live a specific area, we can exempt that extension generally just within that area.
Makes total sense and we add your vote to this feature request. Right now exclusions are global as we wanted to give you guys options or workarounds as soon as possible. Thank you for your feedback!
however, could you give me some specific examples? I'm just curious to see how do they format the output, considering there are thousands upon thousands of files getting changed daily on every machine by all sorts of apps.
...and, which is why 1. his statement confused me, and 2. why I asked what log file he used to determine the extension which seemed to have caused his event because...as we know...Inline Entropy doesn't show extensions. Unless, his 2nd statement was in regards to a different event, but he didn't explicitly state so? And, this may be the case...he just didn't share he had a separate event for Guest Indexing?