Veeam 12.1 & Suspicious files

[Dima P.]

Dima, I cannot determine which pattern or extensions Veeam isn't happy with. The message in VBR is just "...too many files have had their names changed...". Is there a log file of the list of extensions that triggered this warning?

Thank you for posting the details, can you please a screenshot? We usually add information about the affected extension, so fill event details and a screenshot would help a lot to understand the issue.

[Dima P.]

I wish there was a way to exclude certain files, i.e. i know that file x on volume y is clean and is acceptable, so that my entire vm is not flagged as infected. My choices are either have it flagged or dont scan it, no middle ground.

Hello Jason,
Welcome to the R&D forums and thank you for your feedback. I've merged your post to existing thread. I've added your vote to this feature request but meanwhile please take a look at this post: right now you can exclude entire extension from processing to make sure it wont generate false-positive malware events for extensions that are normal for your environment. Thank you!

[jeffshead]

I'm not understanding something. I added exclusions for all of the file extensions that were reported as being suspicious in the 12/12/2023 backups. I pulled those extensions from the suspicious_files_23-12-12.log file.

Exclusions:

Code: Select all

*.oxr
*.dop
*.afd
*.ttt
*.wiki
*.fixed

I also marked all of those backups as clean. Veeam flagged the 12/13/2023 backups as being infected but when I examine C:\ProgramData\Veeam\Backup\Malware_Detection_Logs\, there's no log file (i.e., suspicious_files_23-12-13.log) for last night's backups so to me, that indicates nothing was found or that they were excluded. So why did last night's backups get flagged as infected if I excluded those extensions and there's no "suspicious_files" log file?

[Dima P.]

Hello Jeff,

The exclusions are set correctly, that's the way these extensions are listed in SuspiciousFiles.xml. Have you marked the corresponding restore points as clean too? suspicious_files_23-12-13.log contains information only about detected items, so it seems that the event has been raised based on previous activity.

[jeffshead]

Early yesterday morning, I marked the corresponding restore points as clean for all three machines.


I also thought it was strange that I received the following email alert. I'm not understanding the inconsistency. It shows zero for clean and suspicious yet one of the three machines that had false-positives is missing from the alert.

Email alert:

[Dima P.]

Thank for your the update Jeff, we need some debug logs here. Can you please raise a support case, share the log bundle via support case and share the case ID with us?

[jeffshead]

This is a home lab with NFR license. Can I raise a support case? The company that I work for is a ProPartner with Veeam but I'm out in the field and this lab is for personal, educational use.

[galbitz]

new one today. it ran clean for a few days after i exempted a problematic file extension and deleted some files. Reg file extension exclusion this is not sufficient long term. My original issue i mentioned was it was flagging exe's in my visual studio project folders. I was able to delete those exes because they are a result of compiled code that i can re compile if i ever need the project again, in that scenario excluding exe isnt viable so we definitely need the ability to exclude specific files and not file extensions. I believe it was held up on some exes i made that hook and record keyboard events, fine i get it, but still its supposed to be there and i wouldnt want to live with the warnings all the time and also cant exempt all exes..

anyways about today i received this one:

Potential malicious activity detected: too many .rtf,.html,.sql,.ini,.jpg,.png,.gif,.svg,.wav,.cab,.h,.js,.css,.xml,.config,.ps1 files have been deleted or renamed since last backup, ensure this activity is legitimate

First off i know what this is, i copied a large directory and iso to a machine, ran the install, and then deleted it yesterday. C:\ProgramData\Veeam\Backup\Malware_Detection_Logs is empty though, whereas previously it wasnt in past incidents. Is this logged elsewhere? I am not seeing a list of files in question. For an error like this we need a who, what, where, or it isnt viable for production environments. I am testing this in a lab where I am the only user of the env so its very easy for me to write these off as things ive done. At work, with 15 IT employees, i would be spending a week on incident response to appease my boss if he received an alert like this and we had no history =)

I am hoping the history is somewhere else and i just need to look, but having this all in one place seems like a better approach.

[Gostev]

I think logging potentially millions of files is off the table. So let's brainstorm here, what kind of reasonable minimal log entry would be useful for you? Would for example first 10 file names per extension help and be enough?

[Dima P.]

This is a home lab with NFR license. Can I raise a support case? The company that I work for is a ProPartner with Veeam but I'm out in the field and this lab is for personal, educational use.

Stay tuned, I've asked QA team to reproduce the issue. Possibly it's related to email reporting and not to the detection engine itself.

[cgsm]

Thank you for posting the details, can you please a screenshot? We usually add information about the affected extension, so fill event details and a screenshot would help a lot to understand the issue.

Dima, see below. There isn't any detail in the History > Malware Detection > Malware Event logging in the GUI. I also do not have any suspicious_files log files in C:\ProgramData\Veeam\Backup\Malware_Detection_Logs that are on the same date as the detections noted in my screenshot.

[galbitz]

I think logging potentially millions of files is off the table. So let's brainstorm here, what kind of reasonable minimal log entry would be useful for you? Would for example first 10 file names per extension help and be enough?

everyones opinion is going to vary, but if we are supposed to investigate a potential issue i know i personally would need the entire list. I think at the moment we are all not dealing with a large number of log entries. Mostly because none of us are responding to a real incident. So the idea that not logging these because it could be too big seems to be overly cautious.

There doesn't seem to be any log entry for the too many files have changed events, which I find a little odd considering you must have already compiled the list to detect the changes..As it stands that error with what seems to be 0 logging isnt useful and as implemented would be better off not existing. What would be the expected next steps when that alert is thrown? I wouldnt know what files were changed, when they changed and who might have changed them.

If people are concerned maybe there can be a setting for max error log size or something as the risk you seem to be trying to mitigate is causing the veeam server to run out of space. While a 10gb log file might seem like a bad idea, if its going to fully document all of the changes ransomware just caused it is likely going to be extremely valuable. Long term you could maybe look to utilize it as the starting point for a restore task etc.. However in a real scenario, most people are probably going to restore the entire server etc.

[tyler.jurgens]

Why not log the directory where those files have been deleted or renamed? It would be obvious to anyone who *knew* what caused it, and if they didn't know they could go check that folder. You still may get hundreds/thousands of folders, but its better than logging every individual file, and you are pointed to where to start investigating.

[Gostev]

Good idea! I would still include some limited number of sample file names just because they are gone from production at this point.

I too am not sure there's any value listing every single changed file because in case of mass changes they are all changed for the same reason and by the same process. As opposed to each file for it's own, unique reason. And yes, the concern here is very high amount of data that would potentially need to be logged.

I also don't believe there will ever be scenario where a living person would physically go and read every line in a dump of 100K changed files when troubleshooting something. This is just not going to happen, ever. So I question the very "need the entire list" unless Veeam Backup & Replication is now seen as a "file system audit" tool that can be used for actual investigations of what is going on in production. We certainly don't want to become one, even just because there are plenty of existing purpose-built solutions for this which are very robust (Windows has one built-in). So given all that, can we try to find some reasonable middle ground here?

[Gostev]

P.S. I think it's also worth reminding here that V12 added the Compare with Production functionality to the file-level recovery Backup Browser. This mode displays changed/deleted items only within the selected scope and thus might be helpful in investigations too.

[Dima P.]

This is a home lab with NFR license. Can I raise a support case? The company that I work for is a ProPartner with Veeam but I'm out in the field and this lab is for personal, educational use.

I've asked support team to reach you and help to collect the logs, looks like we still need those. They will reach you out via your forum's email address.

[Dima P.]

There isn't any detail in the History > Malware Detection > Malware Event logging in the GUI. I also do not have any suspicious_files log files in C:\ProgramData\Veeam\Backup\Malware_Detection_Logs that are on the same date as the detections noted in my screenshot.

cgsm,

Unfortunately we have to ask you to raise a support case too as we need debug logs to continue the investigation. Is it possible to raise it and share the case id with us? Thank you!

[cgsm]

@dima
Case # 07050157

[Dima P.]

cgsm,

Thank you! Passed the case to QA folks!

[dabrigo]

Good idea! I would still include some limited number of sample file names just because they are gone from production at this point.

Adding to this: what about a flag in the preferences so that we can toggle the logging of all suspicious files, default is disabled and can be enabled when an in-depth analysis is required.

[Dima P.]

Hello Davide,

We've decided to add detailed logs containing the full paths with the files removed between job runs on a first place but such log will be created only if malware event is raised. Thank you for your feedback!

[cgsm]

@Dima
Will the new/enhanced logging also show the path to the file that triggered the "ransomware note found" warning?

[Dima P.]

Hi cgsm,

Enhanced logs will cover the malware events raised by guest file index analysis.

We are investigating the possibility to add path information for inline malware detection events like ransomware note, encrypted data or onion link (do not mix this one with .onion extension event). Unfortunately it's quite complex, so I cannot provide any ETA at the moment. Thank you!

[Gostev]

To clarify the added complexity, these particular detections are not performed on a file system level but rather directly in the data stream.

[LThibx]

I'd like to chime in here with a bit of feedback on this subject.
I just installed VB&R 12.1 last weekend and was monitoring the Malware Detections.
Yesterday 12-18-2023, before the nightly run, I put in some file exclusions. One being *.biz. Last night's run did exclude those files.

But other exclusions where I specified complete file names, those did not get excluded:
_pg.cve - Actual complete file name (part of Sage 100)
_pr.get - Actual complete file name (part of Sage 100)
.purge - Actual complete file name (part of Sophos AutoUpdate
OPS311.EXX
MCADAM.SEA
Calendar.ccc

I add my vote to other's suggestions about being able to provide path expressions or path names for exclusions.
If an extension (.biz for example) is associated with a known threat, it would be better to be able to provide path exceptions. That way if files with that extension were found outside of the exceptions, it would be caught.

LThibx

[Dima P.]

Hello LThibx,

For now unfortunately exclusions must match the syntax of predefined extensions in the SuspiciousFiles.xml but I've added your vote to this feature request! Thank you for your feedback!

[LThibx]

Hi Dima,
Thanks for the response.
Yea, I read that post earlier and re-read it again.
So at this point the only way to not detect all these false positives is to specify it as *.<ext>
IMO, not a very clever implementation of applying exclusions.
Exclusions need to become a bit more versatile.

Ok...got it.

Thanks
LThibx

[Dima P.]

Lonnie,

Sure, there is always a room for improvement and we totally understand the need to exclude particular file paths while entire extension being monitored. We've already noted a couple of improvement requests related to this area for next versions. Thank you!

[jeffshead]

Several days have passed since I turned on Malware protection. I continue to get new false-positives and with every new backup (clean or infected), the previously 'Marked as clean' backups are remarked as infected.

Having to manually browse log files in order to determine what triggered an alert which results in excluding a known, popular malware file extension just for a single file is not good practice. Especially when you consider that you have to manually make sure that your exclusion exactly matches what is predefined in SuspiciousFiles.xml. Furthermore, when a new false positive is triggered, previously exempted file extensions are also included in the Veeam One email alerts even though they were excluded several days prior and as I stated earlier, the corresponding backups which were marked as clean get remarked as infected.

Undoubtedly, Veeam will refine and improve this great feature but speaking strictly for me, I don't see the cost/benefit (time/risk) leaning in my favor with this current implementation. I'll revisit this after Veeam has had time to use feedback to improve and tune the Malware protection and alerts. Thank you Veeam for your business model and utilizing input from all users

[Dima P.]

Hello Jeff,

Several improvements to address the mentioned issues are already planned for upcoming releases. Thank you for the honest feedback, Merry Christmas and Happy New Year!