Veeam 12.1 & Suspicious files

[Dima P.]

Hello Scott,

Thank you for the feedback and glad to hear that we helped you to find some attack leftovers!

1) The ability to mark previously marked items as clean and IGNORE would be huge. Essentially take that Malware Detection file, and just ignore all those files in future scans. Many of these files are quite old and some have file types that won't be used again.

In the 12.1.1 we've added ability to exclude specific extesions directly from the malware event in B&R. In next versions we plan to add the ability to exclude specific file paths, allowing you to keep the extensions monitored but eliminating the false-positives caused by the trusted files.

2) the ability to disable specific file types per server, or per folder. Servers are hosting files for different reasons. Just because 1 server needs an exclusion, doesn't mean I want it excluded everywhere to stay protected.

Good request, added your vote!

3) showing the files in the GUI or "NEW" files in the GUI vs the log file.

Understood, also a good request we keep track of. For now the goal is to provide path exclusions globally (i.e. specific path will be excluded from detection across all machines), after it's done we can start working on the precise per machine exclusions!

Hello HDI92,

Is it possible to see a list with the path and name of the detected files? I can't find it. Thank you

It's possible to check the dedicated log files on the B&R server here: Programdata\Veeam\Backup\Malware_Detection_Logs\

[B.T.]

It's possible to check the dedicated log files on the B&R server here: Programdata\Veeam\Backup\Malware_Detection_Logs\

We have no such folder in our 12.1 B&R Windows server system. Is there another location to find these results? I feel like we need to disable this feature with all the alerts, our EDR is not reporting issues. Please advise, thank you.

[Andreas Neufert]

It is a hidden folder on the c drive usually

[B.T.]

I checked in the location mentioned and there is nothing there. Can you please provide the location path?

[damiengm]

Hi,
Wondering if this has actually been implemented yet in 12.1.1.5, is the ability to exclude a filename: we have a file that exists in many users accounts on our server: 30018.techstreet.com.sea. I've added this file name under Inventory->Malware Detection->Settings->File Masks to Monitor...->Trusted files: but am still getting Malware warnings "Potential malware activity detected:*.sea: 5". I guess I could set it as an exclusion wildcard (a StuffIt archive infecting a Windows server, possible but unlikely) but better to have narrow exclusions that wide if we are able to.
Regards
Damien

[Dima P.]

I checked in the location mentioned and there is nothing there. Can you please provide the location path?

Hello, B.T. Login to your Veeam B&R server, and in the file explorer type in this path C:\ProgramData\Veeam\Backup\Malware_Detection_Logs\

If you cannot see the folder on your B&R server with the latest build and malware detection based on guest index analysis enabled please reach out our support team.

Wondering if this has actually been implemented yet in 12.1.1.5, is the ability to exclude a filename

Hello Damien, with latest build it's still not possible to exclude path to a file or folder but we plan to deliver this functionality as soon as we can. Thank you for your feedback!

[coolsport00]

After a couple weeks of upgrading to v12.1 CP1 and running Malware Detection, as well as seeing a couple posts on the Comm Hub, I'm just now coming across this Forum thread. And it's been great! I'll provide my feedback & questions below...

@B.T. - I think @Dima can confirm this, but if you're VBR Malware config is like mine, 1. you don't have your jobs configured for guest indexing and thus 2. you also don't have File System Analysis enabled. So, the "Malware_Detection_Logs" folder isn't available..it's only for File System Analysis. If you did configure File System Analysis & your jobs aren't configured for Guest Indexing, Veeam wouldn't be able to report anything anyway (i.e. create the Malware logs directory). If this is the case (you only have Entropy enabled), the only file I'm aware of to investigate is C:\ProgramData\Veeam\Backup\Svc.VeeamDataAnalyzer.log file. But, this log file doesn't really say much more than the 'details' window you can open from within the Console > History node. This leads to my feedback below...

@Dima @Gostev - for Inline Entropy scans only, logging is sparse. I think I did see 1 specific comment in this thread about this specific feature. But, can a dedicated log for Entropy scans be created like the one for File system? BTW..I've been working with support on all the questions/concerns I've been having since I turned on Malware Det (case#07128707). The details window is really too vague to get any idea what VBR "sees" as an issue within my VMs. I've had 6 VMs flagged in 2wks, 4 of which are not only Linux VMs, but appliances at that, so am confident they're good (I excluded those VMs globablly). My 2 Windows VMs I A/V scanned and they came back ok, as well. That said, could you all provide more detail on what file and/or location VBR sees as being 'suspicious'? Being this is a block-level scan and not file, I'm not sure what 'magic' you all can do here, but this level of scan is lacking a lot (may be outside your control?).

Also, the Guide says for Entropy scans, there's a file created on the Proxy in a temp location in RIDX format, but doesn't say what path. Is this deleted upon job completion? Is the file viewable for flag investigation purposes?

Lastly, and I may need to finish this with Support, when I go to attempt to scan my backups for A/V or YARA, it doesn't scan...it starts, but then just hangs at 0%. At first I thought this may be a network L3 issue, but then remembered all VBR is doing here is scanning the backup files, not the VMs (correct?). So, all should be ok. Thoughts on this?

Thanks for all the comments everyone!

[Dima P.]

Hello Shane,

Thank you for your feedback!

If you did configure File System Analysis & your jobs aren't configured for Guest Indexing, Veeam wouldn't be able to report anything anyway (i.e. create the Malware logs directory). If this is the case (you only have Entropy enabled), the only file I'm aware of to investigate is C:\ProgramData\Veeam\Backup\Svc.VeeamDataAnalyzer.log file

Absolutely correct, this engine requires guest file indexing in the job.

I think I did see 1 specific comment in this thread about this specific feature. But, can a dedicated log for Entropy scans be created like the one for File system?

Unfortunately not in the current version, but we are investigating how this feedback can be addressed.

BTW..I've been working with support on all the questions/concerns I've been having since I turned on Malware Det (case#07128707).

Thank you for sharing, we will review the support thread and all the questions!

That said, could you all provide more detail on what file and/or location VBR sees as being 'suspicious'? Being this is a block-level scan and not file, I'm not sure what 'magic' you all can do here, but this level of scan is lacking a lot (may be outside your control?).

For inline scan engine B&R proxies analyze raw data while it's being transferred, so right now we cannot match the block to a file / file path on the file system.

Also, the Guide says for Entropy scans, there's a file created on the Proxy in a temp location in RIDX format, but doesn't say what path. Is this deleted upon job completion? Is the file viewable for flag investigation purposes?

Wont help unfortunately as such temp logs or files contains metrix for entire backup, not files, paths or anything file system 'centric'

Lastly, and I may need to finish this with Support, when I go to attempt to scan my backups for A/V or YARA, it doesn't scan...it starts, but then just hangs at 0%. At first I thought this may be a network L3 issue, but then remembered all VBR is doing here is scanning the backup files, not the VMs (correct?). So, all should be ok. Thoughts on this?

There were issues with backup locks preventing Scan Backup to start on the vanilla 12.1 release. You are running the latest path, right?

[coolsport00]

Hi @Dima -
Thanks for the responses. I will anticipate what the future releases can do to enhance this great feature.

As far as my Scan Backup process...I have a 'feature enhancement' request for it. It turns out this does work for me. The issue I was (am) experiencing is, when I look at the 'Running jobs' area for this process, it only shows 0% complete for the entire scan process. So when I went to run this task, I'd close the process window & basically just monitor the Scan Backup in the 'Running jobs' area. The particular VM I was scannng was 1TB in size. And, I'd never ran this type of task before (i.e. Secure Restore type task), so wasn't really aware how long the task actually does take to complete (hours for a VM of the size I was scanning). So, in working with support, they noticed the same behavior as me (0%), but the task finished successfully. I re-attempted a scan for a smaller sized VM (90GB) and it took 56min to complete. That being said...I wasn't patient enough, it seems, for the larger VM So my feature request is to see if you all can actually increment that 0% status in the Running Jobs section. And, within the process window itself, it doesn't provide any progress status either...just states "[C:\] Scanning...". Would be beneficial to see an actual incremented progress status if possible.
Thank you.

[dweide]

+ 1 vote for this request:

2) the ability to disable specific file types per server, or per folder. Servers are hosting files for different reasons. Just because 1 server needs an exclusion, doesn't mean I want it excluded everywhere to stay protected.

[molsena]

Hi
I have activated suspicious detection for us
And a potential malware activity was detected with the note *.wch: 3
I ran a manual scan on the machine in question using the antivirus product Withsecure, but unfortunately nothing was found.
A backup scan carried out with Veeam showed a "Failed" after 14 hours and no indication and also did not find a clean restore point. Does malware detection work reliably? Between backups with suspicious activities, there are backups that show as OK. How do I find out where and what file is behind suspicious activity?

[Mildur]

Hello Andreas

Welcome to the forum.

Please see the previous answers in this post.
*.wch looks like it could be related to products from "Corel".
You may exclude those file endings from further scans.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[molsena]

Hi Fabian
Thanks for the answer, but when I scan the server in question for this *.wch file, it is not found. Should I still set up an exception for the file extension?

[coolsport00]

@molsena -
I would suggest to set up the extension exception 1. since your on-host A/V scan turned out ok; and 2. if you don't want to receive future Malware Detection events for that file extension type, if you're sure those types are fairly 'safe' in your environment. It really is best up to you to on whether adding the exception is best for your environment or not.

Best,
Shane

[LThibx]

Last nights backup had an alert of suspicious files on one of my machines. 'Potential malicious activity detected too many .zip, .conf files have been detected or renamed since last backup...etc. I looked in the C:\ProgramData\Veeam\Backup\Malware_Detection_Logs folder, but there is no log file for that backup. How can I determined where the backup encountered these files?

[coolsport00]

@LTibx -
Did you look in the History node of the Console? If you double-click the machine affected, you should see a Malware event window that, hopefully, has the file and/or path to the potential risk.

[LThibx]

@coolsport00,
Yes, that is where I see the alert. It shows the machine it is on, but only the message:
"Potential malicious activity detected: too many .zip,.conf files have been deleted or renamed since last backup, ensure this activity is legitimate"
No indication of where it found this activity.

[coolsport00]

@LThibx - ah, ok; hmm...would think a folder at least would have been shown in there. I will say...investigation/forensics of these events are going to be a challenge at times. For me, it's a huge challenge because I don't use Guest Indexing, and thus no File System Analysis configuration for my Malware Detection. Inline scans have no info - just the event name, e.g. Encrypted Data (what data??), Ransomware Note (what note??), Onion Link (in what file/location??).

Does this particular server have a folder/directory you're aware of with a lot of zips? Do you have cleanup tasks for it? Those are just some of the things to think about when performing your event investigation strategies.

[Dima P.]

Hello folks,

"Potential malicious activity detected: too many .zip,.conf files have been deleted or renamed since last backup, ensure this activity is legitimate"

This event is created based on index analysis. There is no text log for the bulk rename event yet, but we will add it in the upcoming minor patch. Thanks!

[A.J.]

Hi,
I have a few suggestions for improvement:

- It would be good if the detailed log files were integrated directly into the mail reports OR viewable in the Backup Console . Due to the security guidelines, I can't easily access the management machine to get the logs manually from C:.
- It would be good if "Malware Detection" and "Malware Events" were available consolidated in the Backup Console and not distributed across Inventory and History. Now you have to jump back and forth over several trees.
- The "Potential malware activity detected" event is completely useless because you can't tell what to look for on the machine. Or I haven't found the necessary log yet?
- Whitelisting needs to be more granular. If I exclude, for example, *.1251.EXT because I only want to whitelist these files, they will still be recognized because the *.EXT extension is checked by default, which is what would be wanted. This means that the *.EXT check beats all other variants of it in the whitelist.
- I would like to see an executive report about the malware detections and actions carried out such as whitelisting, "Mark as Clean" etc. Audit and reporting obligations are becoming increasingly important for us service providers towards our customers and also authorities.

Basically, I think these new features are a sensible approach. However, at the moment they tend to lead to extra work and uncertainty due to many false positives.
You should also consider making this feature available in the lowest Veeam edition and not just in Enterprise Plus.

[Dima P.]

Hello A.J.,
Thank you for your feedback and feature requests!

It would be good if the detailed log files were integrated directly into the mail reports OR viewable in the Backup Console

Good idea, noted as improvement request!

It would be good if "Malware Detection" and "Malware Events" were available consolidated in the Backup Console

Agreed that the need to switch between tabs is not continent, we will discuss possible improvements with the team.

The "Potential malware activity detected" event is completely useless because you can't tell what to look for on the machine

If event is created by inline scan engine, unfortunately right now we do not have any logs to determine the exact file path.

- Whitelisting needs to be more granular. If I exclude, for example, *.1251.EXT because I only want to whitelist these files, they will still be recognized because the *.EXT extension is checked by default

Great feedback, thank you!

- I would like to see an executive report about the malware detections and actions carried out such as whitelisting, "Mark as Clean" etc

It's possible today with VeeamOne solution, please take a look: Veeam Backup Monitoring > Malware Detection

Basically, I think these new features are a sensible approach. However, at the moment they tend to lead to extra work and uncertainty due to many false positives.

Please let us know about any false-positive reports and we will gladly investigate those! You can open a support case or post here to let us know which extensions/files created false positives (also, if possible, we need to know the amount of files and what app is using such extension on your workloads).

Thank you!

[Jalapaca]

Veeam B&R Version 12.1.1.56

Enabled Malware Detection about a week ago, got an alert last night with no information in it other than "POSSIBLE MALWARE" which is quite useless IMO. There are no logs in the Veeam console other than, "Hey we 'think' there might be malware, good luck!" I'm not sure how this passed into Stable with the serious lack of logging and information available.

I had to Google the alert message, find a Reddit thread that linked me to this thread, and then spend time looking through various logs and folders to find out that the logging is still only, ".txt:-31" at the most. I'm sure glad I know how many files were deleted, not like I need to know WHICH ones.

I've checked the "C:\ProgramData\Veeam\Backup\Malware_Detection_Logs\" folder and it's empty. I had to review the "C:\ProgramData\Veeam\Backup\Svc.VeeamDataAnalyzer.log" file to get anything slightly useful regarding the malware scans and it only tells me how many files were added/removed.

I love Veeam and everything it does, but the Malware Detection feature is not even half baked, it's still raw sitting in a lukewarm oven. We shouldn't be paying to alpha test features and get sent on wild goose chases with the only info being "there MIGHT be a goose somewhere in the city, please find it." I guess I will be disabling this feature until it's actually fleshed out with proper logging.

[Dima P.]

Hello,

I had to Google the alert message, find a Reddit thread that linked me to this thread, and then spend time looking through various logs and folders to find out that the logging is still only, ".txt:-31" at the most.

It's not correct, most of the events based on guest index analysis has the dedicated log which can lead you to the exact file location. However, we are still investigating how to provide advanced logging for inline scan option.

Enabled Malware Detection about a week ago, got an alert last night with no information in it other than "POSSIBLE MALWARE" which is quite useless IMO.

Could you share the event type and the event details from B&R console?

[SnakeSK]

I disabled the feature too, I was getting possible malware and ransomware notes, with no logs to diagnose, support wasnt helpful either (case 07110723 )

[Gostev]

Really appreciate you reporting your events to support. Overall there has been a great progress specifically due to all this field knowledge rolling in. Most customers are happily sharing not only the false positives they get, but also useful information about the app that caused them. Knowing the specific usage of files with monitored extensions by legitimate apps, we're able to do a few different things. For example, if they are config files, then we don't exclude them from monitoring - but instead raise a threshold for a number of files before their presence will be seen as a problem. And in other cases, when a legitimate app is expected to generate many of such files, we just remove the extension from monitoring completely.

Needless to say, all these changes are applied to our online malware definitions right away, so everyone who has their auto-update enabled will benefit of these changes immediately. While the rest are highly encouraged to script updating the malware definition file, or just do it manually periodically (see KB4514). This is critically important to do even just because newly born malware strains get added there every week, and they are much more dangerous than those well-known strains of the past.

Also there's finally some good news on the inline encryption detection diagnostic: the dev team seem to have found a reliable way to match encrypted block to a file and they will start building this tool next week.

[BackupBytesTim]

Curious if it would be possible, in an effort to simplify exclusions, to determine what applications (processes) are modifying or creating files, and exclude any modifications by specific applications regardless of file location or extension? Or would that be too far outside the current functionality to be feasible anytime soon?

Full disclosure, I do not use the feature anywhere, even for testing, but I've been following the thread as I do try to keep up with the status of new features.

[Gostev]

We don't have an in-guest monitoring agent so we can't track which app has modified or deleted which file.

[BackupBytesTim]

Well, I certainly understand that limitation then. Definitely could be a nice feature to have for future consideration though, it is a feature other anti-malware/anti-ransomware software has.

[Gostev]

Well, we happen to be a backup software, not an antivirus software however, could you give me some specific examples? I'm just curious to see how do they format the output, considering there are thousands upon thousands of files getting changed daily on every machine by all sorts of apps.

[BackItUp2020]

Even with the latest Veeam update, we had to resort to disable the malware features. Just too noisy and not enough tweaking to make it useful. I think a little half-baked at this point, but good for marketing.

For us in particular, we would need to mark as "false positive" on a per-server basis and have those stick for the next job. Prior to disabling this feature, backup jobs would just set off the same 20 alerts every night, which would make our CS team have to look into every time, then do a scan with other scanners, and inevitably ignore it.

Looking forward to some improvements for sure.