Veeam 12.1 & Suspicious files

[damiengm]

Malware Detection

Added the ability to exclude specific file paths from suspicious file system activity analysis.

Hi, this wasn't quite what I was expecting for our use case: we have a DRM software which creates a file in each user's directory with the extension .sea, but its the same filename for all users. Its not a compression file as far as I can tell. I was hoping for just the filename could be on a exclude list, but I can use the above and enter in the full path as so far I only have 5 users affected so its just 5 paths to enter. The number of total users we have is low so I can always manage this manually, but I'd image if there were 100's of users this would be more problematic. With Joel's suggestion I would add to that the ability to add wildcards or path macros e.g. %HOMEDIR%/dir/filename.fir or %APPDATA%/otherdir/*.dss. I understand some of those system variables are also user specific so that may be hard to retrieve out of a VM.
Regards
Damien

[sdv]

Hello Joel,

Makes total sense and we add your vote to this feature request. Right now exclusions are global as we wanted to give you guys options or workarounds as soon as possible. Thank you for your feedback!

Excluding file/paths and extensions are still configured globally in 12.3.

Is there a new release coming up to make this less global?
Thus make malware exclusions and trusted objects VM specific.

[doktornotor]

Not sure this is the proper rant topic - but: there are blatant false positives in the malware detection which are super-easily reproduced, such as

Potential malware activity detected: too many files have had their names changed since last backup, ensure they were not encrypted by ransomware

I can reproduce this any time by just wiping Google Chrome or Edge cache / history. Really does not make sense to detect those, rather annoying. Some sample from the logs:

Code: Select all

[\\?\C:\Users\<redacted>\AppData\Local\Google\Chrome\User Data\Default\Cache\Cache_Data]
New files: 945 Deleted files: 577
New files: 
f_016f72, f_017296, f_017297, f_017298, f_0176e4, f_017299, f_0172ad, f_01725b, f_017541, f_01729a, f_017ad4, f_01729b, f_0172ae, f_017213, f_017464, f_017a54, f_0172af, f_017259, f_017ad5, f_017253, f_017b78, f_017aab, f_017552, f_017b27, f_01771b, f_017ad6, 
... <snipped>,
Deleted files: 
f_016b94, f_016c94, f_014fec, f_015e5b, f_016cd9, f_016b4a, f_015e55, f_016bac, f_016cb3, f_015e62, f_016bcf, f_0161db, f_015e5e, f_015e25, f_016b00, f_015e81, f_015e5f, f_016afc, f_016bef, f_015e42, f_015e60, f_015e43, f_015e5c, f_015eff, f_014fac, f_016b89, 
... <snipped>,

[\\?\C:\Users\<redacted>\AppData\Local\Google\Chrome\User Data\GrShaderCache]
New files: 111 Deleted files: 115
New files: 
f_004a0e, f_004a25, f_004a26, f_004a3f, f_004a30, f_0049b2, f_004a0f, f_0049e5, f_004a27, f_004a3c, f_004a31, f_004a21, f_0049eb, f_0049b7, f_004a10, f_004a1f, f_0049d6, f_004a22, f_0049ec, f_0049ee, f_004a42, f_0049f3, f_0049ed, f_004a18, f_004a11, f_0049e9, Deleted files: 
f_004994, f_00498d, f_0048cf, f_004965, f_00497c, f_0048f1, f_004966, f_00498e, f_004983, f_004967, f_0049a5, f_00497f, f_004980, f_004981, f_0049a8, f_004975, f_0049a9, f_004982, f_004953, f_00497d, f_00497e, f_004976, f_004977, f_00493a, f_004978, f_00498f, 
... <snipped>,

[\\?\C:\Users\<redacted>\AppData\Local\Google\Chrome\User Data\Default\Code Cache\js]
New files: 1372 Deleted files: 1256
New files: 
ddfbb02afbd26420_0, 1be07a4678d08021_0, 97cdbda0a603a967_0, 828118f2a19624aa_0, 0284cb5a09ff1061_0, f71fba7d99a66536_0, 52db9a92dfceb4bc_0, 8668876dd8c373e9_0, 09e188f3ab4fe451_0, 8870f87bd65c6849_0, ba9d8ed9a9ef4a22_0, b288cebb5113aa83_0, 
... <snipped>,
Deleted files: 
a7cf60cb8934f782_0, 937a8545e742ab3e_0, 30b1747e964e26dd_0, befddea146dca5fd_0, 7f125a8dbbea6478_0, 7dcb585914e593af_0, 76969a0061c1e5d1_0, f69cbd5b57c1d0a0_0, e285000faa75caab_0, f8041f6f86a3733e_0, 44a3aa3f725f0ffe_0, 7605fef948ae782c_0, 
... <snipped>,

Some more things to avoid nagging admins with:

Code: Select all

[\\?\C:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18]
[\\?\C:\ProgramData\Microsoft\Windows\Caches]

Really produces just useless FP noise.

[doktornotor]

Ok, after some more testing, I disabled the feature altogether. Uninstall a single application on a PC (PowerChute Serial Shutdown) -> boom: suspicious activity alert on next backup. Is someone really using this in production? This is just a test lab with trial license, seriously cannot imagine babysitting the feature with hundreds of computers and manually marking all those FPs. You cannot assume the machines are pretty much static and produce knee-jerk alerts on any change to the state.

Should I decide to give it a try again in future, a checkbox to disable the file deletion "logic" altogether would be very much required. Waste of time otherwise.

[jeffshead]

@doktornotor - I've come to the same conclusion and agree with your entire post. I like innovation but Veeam is a backup solutions company, not an antivirus/malware company. It should be much more granular, easier to investigate/create exclusions and to submit false-positives feedback for frequent updates to deal with false-positives.

Veeam is a great backup solution that I still highly praise and sell but its malware detection feature is NOT a selling point that I use.

[Gostev]

Is someone really using this in production?

Yes and it already helped to prevent many actual cyber-attacks. Just last week one of our system engineers has shared a feedback from the customer when Veeam was able to detect an attack that passed every single purpose-built security solution in the environment, leaving the customer's security team completely blown away that it was a backup solution that was able to detect the attack.

It's just that, as always, people rarely come to forums to share the positive experience, they are only quick to come here to complain when they don't like something. This is perfectly normal but its becomes even more complex topic when it comes to security, as most admins don't like to talk about successful attacks on their production environment in principle.

In any case, if you don't find it useful for your specific environment, for sure you can just disable the feature as this is a completely optional functionality.

[Gostev]

And just a day later after I posted this, Veeam has detected an in-process cyberattack at a huge (Fortune 500) manufacturer and allowed their security team to remediate it in time. While the customer is obviously thrilled with the net result, they will likewise never come here to talk about this attack in public.

[doktornotor]

Well I certainly appreciate some of the features may be useful and innovative / different enough (compared to the usual cybersecurity products) to make it worth using. The "mass deletion" detection however is not one of them - not in the current state for sure. The problem is - the configuration is pretty much bare-bones or even non-existent, as @jeffshead noted. If I cannot disable a feature that just doesn't work in my environment, the rest is sadly unusable as well.

[Gostev]

Actually, it's all configurable and technically you can set "mass deletion" thresholds high enough so it does not trigger at all. Plus I believe the default thresholds were already increased in the later versions after 12.1 discussed here, right @Dima P.? What are the current ones and using what configuration file can they be increased further?

Also, keep in mind that from 12.3 onwards, when you do "Mark as clean" then the mass deletion thresholds will be adjusted accordingly for this specific machine. This helps, for example to easily adjust "useful file types" deletion thresholds for a server hosting user home folders, as it is more likely to see many useful file types deleted in 24 hours.

[Dima P.]

Sure, here is the detailed instruction how to add / remove specific extensions or adjust the specific thresholds Multiple Deleted Files. Such event will be raised if at least 100 files or 50% of all files or a certain type is deleted.

The "mass deletion" detection however is not one of them - not in the current state for sure. The problem is - the configuration is pretty much bare-bones or even non-existent, as @jeffshead noted. If I cannot disable a feature that just doesn't work in my environment, the rest is sadly unusable as well.

Please make sure you are with the latest 12.3 build. Additionally can you please elaborate what type of extension is causing issue (...and if possible let us know what type of activity is raising such false positive i.e. software upgrades / user integrations)? Thank you!

[doktornotor]

Ok, I posted two super-easily reproducible examples of perpetual "mass deletion" FPs above which occur with the very latest 12.3 build

- clearing browser/OS/ other caches/temporary junk: post538251.html#p538251
- simply uninstalling an application: post539868.html#p539868 (that particular app mentioned bundles JRE and lots of html/js etc. static files (documentation).

"Mark as clean" does not help - these are coming back over and over again.

Dunno, it even triggers on some "garbage collection" in Veeam itself, looking at one of the logs:

Code: Select all

C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{a6ad56c2-b509-4e6c-bb19-49d8f43532f0}\{db8a6eaf-8379-438e-ad63-689ebc5822dc}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}\{9849391f-f53a-4990-96e5-85b23580d7e9}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{be000cbe-11fe-4426-9c58-531aa6355fc4}\{54219aa7-8caa-42c1-906e-26b2009f5852}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{542da469-d3e1-473c-9f4f-7847f01fc64f}\{fbfea13e-51b7-4578-a54b-3d4f80b909f7}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{59b1f0cf-90ef-465f-9609-6ca8b2938366}\{7136db3f-aa39-41f4-83b9-9b83722318bd}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}\{eb93324b-5baa-4889-ad25-95dc6521e9c6}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{41db4dbf-6046-470e-8ad5-d5081dfb1b70}\{0ef9ff96-21d0-4fab-b396-79ece4441114}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{2a40fd15-dfca-4aa8-a654-1f8c654603f6}\{1e76c9d7-fa33-4962-bc34-d2b7fd6a6137}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{cd3f2362-8bef-46c7-9181-d62844cdc0b2}\{f30658c3-4b0c-4d7e-83ed-6dab443acd76}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{afbab4a2-367d-4d15-a586-71dbb18f8485}\{ebb7bd6d-5c42-4a68-b858-6a41da087162}\MetadataDoc.xml
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_12_53_10.zip
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_00_32_07.zip
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_06_42_08.zip
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_19_03_53.zip

Will play with the manual exclusions and threshold settings, thanks for the link! Just noting, there are some default exclusions obviously missing, as debated above.

[Dima P.]

Thank you doktornotor!

We would love investigate this issue but need the debug logs to work with. Can you please raise a support case and share the case ID with me?

[doktornotor]

Well as noted, this is a test lab with a trial license. Never had any luck with tickets for those cases (very much expected). I'm planning to set up some small test jobs with a sample of machines on some site with a "proper" license (once those are upgraded to 12.3) and raise a support case then. Thanks.

[Andreas Neufert]

If you open a ticket with trial and share here the number, the PM team can forward the logs as needed.

[Dima P.]

C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{afbab4a2-367d-4d15-a586-71dbb18f8485}\{ebb7bd6d-5c42-4a68-b858-6a41da087162}\MetadataDoc.xml
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_12_53_10.zip

We've reviewed this issue and whitelisted the data created by our Veeam Agent for Windows and Veeam One agent. It should not affect guest file analytics metrics anylonger.

- clearing browser/OS/ other caches/temporary junk: post538251.html#p538251

Also addressed the issue mentioned it this post. Several Google Chrome cache locations will be excluded from file analytics metrics collection.

Thank you for the report!

[sdv]

Excluding file/paths and extensions are still configured globally in 12.3.

Is there a new release coming up to make this less global?
Thus make malware exclusions and trusted objects VM specific.

Any news to this?

[Dima P.]

Hello Stefan,

Thank you for your post. This feature is on the radar, I'll add your vote to this feature request!

[jeffshead]

I was getting a lot of false-positives and I kept adding exclusions. I was getting frustrated and feeling like I was spending more time than should be required "baby sitting" the malware functionality of Veeam but... I recently started over from scratch, due to a couple of unrelated issues. I rebuilt the Veeam server, proxies, etc and all of the backup jobs. Since doing this a week ago, I haven't received a single false-positive, so far!

Veeam's malware detection is maturing in the right direction! I still had to manually add some exclusions but several of my previous exclusions were already excluded via omissions in the C:\Program Files\Veeam\Backup and Replication\Backup\SuspiciousFiles.xml file. Great job, Veeam Team!

[Gostev]

Right, suspicious files definitions are updated up to a few times per week. Most updates are adding newly released malware definitions, but some are removing common false positives reported here and via Support. Thanks everyone for your feedback!

[BackupBytesTim]

Regarding that last comment, are those definition updates handled by the regular manual update process or are those updated automatically separate from the VBR software itself?

[Gostev]

They are updated automatically every 24 hours as far as I remember, so long as your backup server is connected to the Internet and access to Veeam Update Server endpoints is not blocked by external firewall.

@Egor Yakovlev @Dima P. how can one easily check that suspicious files definitions have been successfully autoupdated recently?

[Egor Yakovlev]

You can check SuspiciousFiles.xml file modified date or go Backup Console > History > Malware Detection and check last Malware Detection session - it has a line with malware signatures update status.

[Dima P.]

Btw the Suspicious File List aka SuspiciousFiles.xml was updated today again with the fixes and exclusions based on latest false positive reports. Thank you!

Veeam's malware detection is maturing in the right direction! I still had to manually add some exclusions but several of my previous exclusions were already excluded via omissions in the C:\Program Files\Veeam\Backup and Replication\Backup\SuspiciousFiles.xml file. Great job, Veeam Team!

Thank you so much for the kind words and additional thank you for sharing all the issues with us!

[VinnieNZ]

@VinnieNZ - file exclusions are only for File Indexing scans...not Inline Entropy scans. The Malware Detection event you received was for the Inline Entropy scan engine. There is no granularity with that particular engine at this point, sadly. As one who also uses Inline Entropy (I don't have guest indexing enabled in my jobs, so I don't use File Index scans), I'm curious what particular log file you were able to detect the file type in? The only log file for Inline Entropy scans is in C:\ProgramData\Veeam\Backup\Svc.VeeamDataAnalyzer.log file. And, from what I've been able to see in this file for several VMs Veeam has detected possible issues with, I've not seen any specific file listed in this log. I've only seen similar dialog as is shown in the Malware Event window from the History node.
Thanks.

@coolsport00 - I'm so sorry, I've got no idea how I missed this over a year ago. Either way, that data is now excluded, since 12.2 possibly - I can't remember when it started working.

[VinnieNZ]

Hi Team,

I've got another interesting issue with the exclusions and Malware scanning - I've got a backup from a Macbook residing on a server that I backup with Veeam and it does Malware scanning over.

I've got 82 alerts for the *.frag extension (and 2 for *.mkp), all related to the location of the Macbook backup data on the server.

The server is Linux based.

For examples sake, on the server, the path to one of the *.frag files is:
/mnt/backup/macbook/Applications/iBooks.app/Contents/PlugIns/iBAReaderKit.bundle/Contents/Resources/KNBuildAnvilSmoke.frag

There's quite a few other paths thrown in as well, all under that same /mnt/backup/macbook/ location.

I've added a trusted objects path under File system activity analysis > Suspicious files to monitor, which is /mnt/backup/macbook/, which based on my understanding of the "Add path" dialogue, should mean that whole macbook path is trusted - under Example: "C:\Folder\ will exclude folder and all it's content.".

However, this does not seem to be the case, as every backup run, I'm still getting suspicious alerts in Malware Events for the excluded path.

Any advice on this one?

[Dima P.]

Andrew,

Thanks for the report. Investigating it now with the RnD team!