Veeam 12.1 & Suspicious files

[damiengm]

Malware Detection

Added the ability to exclude specific file paths from suspicious file system activity analysis.

Hi, this wasn't quite what I was expecting for our use case: we have a DRM software which creates a file in each user's directory with the extension .sea, but its the same filename for all users. Its not a compression file as far as I can tell. I was hoping for just the filename could be on a exclude list, but I can use the above and enter in the full path as so far I only have 5 users affected so its just 5 paths to enter. The number of total users we have is low so I can always manage this manually, but I'd image if there were 100's of users this would be more problematic. With Joel's suggestion I would add to that the ability to add wildcards or path macros e.g. %HOMEDIR%/dir/filename.fir or %APPDATA%/otherdir/*.dss. I understand some of those system variables are also user specific so that may be hard to retrieve out of a VM.
Regards
Damien

[sdv]

Hello Joel,

Makes total sense and we add your vote to this feature request. Right now exclusions are global as we wanted to give you guys options or workarounds as soon as possible. Thank you for your feedback!

Excluding file/paths and extensions are still configured globally in 12.3.

Is there a new release coming up to make this less global?
Thus make malware exclusions and trusted objects VM specific.

[doktornotor]

Not sure this is the proper rant topic - but: there are blatant false positives in the malware detection which are super-easily reproduced, such as

Potential malware activity detected: too many files have had their names changed since last backup, ensure they were not encrypted by ransomware

I can reproduce this any time by just wiping Google Chrome or Edge cache / history. Really does not make sense to detect those, rather annoying. Some sample from the logs:

Code: Select all

[\\?\C:\Users\<redacted>\AppData\Local\Google\Chrome\User Data\Default\Cache\Cache_Data]
New files: 945 Deleted files: 577
New files: 
f_016f72, f_017296, f_017297, f_017298, f_0176e4, f_017299, f_0172ad, f_01725b, f_017541, f_01729a, f_017ad4, f_01729b, f_0172ae, f_017213, f_017464, f_017a54, f_0172af, f_017259, f_017ad5, f_017253, f_017b78, f_017aab, f_017552, f_017b27, f_01771b, f_017ad6, 
... <snipped>,
Deleted files: 
f_016b94, f_016c94, f_014fec, f_015e5b, f_016cd9, f_016b4a, f_015e55, f_016bac, f_016cb3, f_015e62, f_016bcf, f_0161db, f_015e5e, f_015e25, f_016b00, f_015e81, f_015e5f, f_016afc, f_016bef, f_015e42, f_015e60, f_015e43, f_015e5c, f_015eff, f_014fac, f_016b89, 
... <snipped>,

[\\?\C:\Users\<redacted>\AppData\Local\Google\Chrome\User Data\GrShaderCache]
New files: 111 Deleted files: 115
New files: 
f_004a0e, f_004a25, f_004a26, f_004a3f, f_004a30, f_0049b2, f_004a0f, f_0049e5, f_004a27, f_004a3c, f_004a31, f_004a21, f_0049eb, f_0049b7, f_004a10, f_004a1f, f_0049d6, f_004a22, f_0049ec, f_0049ee, f_004a42, f_0049f3, f_0049ed, f_004a18, f_004a11, f_0049e9, Deleted files: 
f_004994, f_00498d, f_0048cf, f_004965, f_00497c, f_0048f1, f_004966, f_00498e, f_004983, f_004967, f_0049a5, f_00497f, f_004980, f_004981, f_0049a8, f_004975, f_0049a9, f_004982, f_004953, f_00497d, f_00497e, f_004976, f_004977, f_00493a, f_004978, f_00498f, 
... <snipped>,

[\\?\C:\Users\<redacted>\AppData\Local\Google\Chrome\User Data\Default\Code Cache\js]
New files: 1372 Deleted files: 1256
New files: 
ddfbb02afbd26420_0, 1be07a4678d08021_0, 97cdbda0a603a967_0, 828118f2a19624aa_0, 0284cb5a09ff1061_0, f71fba7d99a66536_0, 52db9a92dfceb4bc_0, 8668876dd8c373e9_0, 09e188f3ab4fe451_0, 8870f87bd65c6849_0, ba9d8ed9a9ef4a22_0, b288cebb5113aa83_0, 
... <snipped>,
Deleted files: 
a7cf60cb8934f782_0, 937a8545e742ab3e_0, 30b1747e964e26dd_0, befddea146dca5fd_0, 7f125a8dbbea6478_0, 7dcb585914e593af_0, 76969a0061c1e5d1_0, f69cbd5b57c1d0a0_0, e285000faa75caab_0, f8041f6f86a3733e_0, 44a3aa3f725f0ffe_0, 7605fef948ae782c_0, 
... <snipped>,

Some more things to avoid nagging admins with:

Code: Select all

[\\?\C:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18]
[\\?\C:\ProgramData\Microsoft\Windows\Caches]

Really produces just useless FP noise.

[doktornotor]

Ok, after some more testing, I disabled the feature altogether. Uninstall a single application on a PC (PowerChute Serial Shutdown) -> boom: suspicious activity alert on next backup. Is someone really using this in production? This is just a test lab with trial license, seriously cannot imagine babysitting the feature with hundreds of computers and manually marking all those FPs. You cannot assume the machines are pretty much static and produce knee-jerk alerts on any change to the state.

Should I decide to give it a try again in future, a checkbox to disable the file deletion "logic" altogether would be very much required. Waste of time otherwise.

[jeffshead]

@doktornotor - I've come to the same conclusion and agree with your entire post. I like innovation but Veeam is a backup solutions company, not an antivirus/malware company. It should be much more granular, easier to investigate/create exclusions and to submit false-positives feedback for frequent updates to deal with false-positives.

Veeam is a great backup solution that I still highly praise and sell but its malware detection feature is NOT a selling point that I use.

[Gostev]

Is someone really using this in production?

Yes and it already helped to prevent many actual cyber-attacks. Just last week one of our system engineers has shared a feedback from the customer when Veeam was able to detect an attack that passed every single purpose-built security solution in the environment, leaving the customer's security team completely blown away that it was a backup solution that was able to detect the attack.

It's just that, as always, people rarely come to forums to share the positive experience, they are only quick to come here to complain when they don't like something. This is perfectly normal but its becomes even more complex topic when it comes to security, as most admins don't like to talk about successful attacks on their production environment in principle.

In any case, if you don't find it useful for your specific environment, for sure you can just disable the feature as this is a completely optional functionality.

[Gostev]

And just a day later after I posted this, Veeam has detected an in-process cyberattack at a huge (Fortune 500) manufacturer and allowed their security team to remediate it in time. While the customer is obviously thrilled with the net result, they will likewise never come here to talk about this attack in public.

[doktornotor]

Well I certainly appreciate some of the features may be useful and innovative / different enough (compared to the usual cybersecurity products) to make it worth using. The "mass deletion" detection however is not one of them - not in the current state for sure. The problem is - the configuration is pretty much bare-bones or even non-existent, as @jeffshead noted. If I cannot disable a feature that just doesn't work in my environment, the rest is sadly unusable as well.

[Gostev]

Actually, it's all configurable and technically you can set "mass deletion" thresholds high enough so it does not trigger at all. Plus I believe the default thresholds were already increased in the later versions after 12.1 discussed here, right @Dima P.? What are the current ones and using what configuration file can they be increased further?

Also, keep in mind that from 12.3 onwards, when you do "Mark as clean" then the mass deletion thresholds will be adjusted accordingly for this specific machine. This helps, for example to easily adjust "useful file types" deletion thresholds for a server hosting user home folders, as it is more likely to see many useful file types deleted in 24 hours.

[Dima P.]

Sure, here is the detailed instruction how to add / remove specific extensions or adjust the specific thresholds Multiple Deleted Files. Such event will be raised if at least 100 files or 50% of all files or a certain type is deleted.

The "mass deletion" detection however is not one of them - not in the current state for sure. The problem is - the configuration is pretty much bare-bones or even non-existent, as @jeffshead noted. If I cannot disable a feature that just doesn't work in my environment, the rest is sadly unusable as well.

Please make sure you are with the latest 12.3 build. Additionally can you please elaborate what type of extension is causing issue (...and if possible let us know what type of activity is raising such false positive i.e. software upgrades / user integrations)? Thank you!

[doktornotor]

Ok, I posted two super-easily reproducible examples of perpetual "mass deletion" FPs above which occur with the very latest 12.3 build

- clearing browser/OS/ other caches/temporary junk: post538251.html#p538251
- simply uninstalling an application: post539868.html#p539868 (that particular app mentioned bundles JRE and lots of html/js etc. static files (documentation).

"Mark as clean" does not help - these are coming back over and over again.

Dunno, it even triggers on some "garbage collection" in Veeam itself, looking at one of the logs:

Code: Select all

C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{a6ad56c2-b509-4e6c-bb19-49d8f43532f0}\{db8a6eaf-8379-438e-ad63-689ebc5822dc}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}\{9849391f-f53a-4990-96e5-85b23580d7e9}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{be000cbe-11fe-4426-9c58-531aa6355fc4}\{54219aa7-8caa-42c1-906e-26b2009f5852}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{542da469-d3e1-473c-9f4f-7847f01fc64f}\{fbfea13e-51b7-4578-a54b-3d4f80b909f7}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{59b1f0cf-90ef-465f-9609-6ca8b2938366}\{7136db3f-aa39-41f4-83b9-9b83722318bd}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}\{eb93324b-5baa-4889-ad25-95dc6521e9c6}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{41db4dbf-6046-470e-8ad5-d5081dfb1b70}\{0ef9ff96-21d0-4fab-b396-79ece4441114}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{2a40fd15-dfca-4aa8-a654-1f8c654603f6}\{1e76c9d7-fa33-4962-bc34-d2b7fd6a6137}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{cd3f2362-8bef-46c7-9181-d62844cdc0b2}\{f30658c3-4b0c-4d7e-83ed-6dab443acd76}\MetadataDoc.xml
C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{afbab4a2-367d-4d15-a586-71dbb18f8485}\{ebb7bd6d-5c42-4a68-b858-6a41da087162}\MetadataDoc.xml
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_12_53_10.zip
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_00_32_07.zip
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_06_42_08.zip
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_19_03_53.zip

Will play with the manual exclusions and threshold settings, thanks for the link! Just noting, there are some default exclusions obviously missing, as debated above.

[Dima P.]

Thank you doktornotor!

We would love investigate this issue but need the debug logs to work with. Can you please raise a support case and share the case ID with me?

[doktornotor]

Well as noted, this is a test lab with a trial license. Never had any luck with tickets for those cases (very much expected). I'm planning to set up some small test jobs with a sample of machines on some site with a "proper" license (once those are upgraded to 12.3) and raise a support case then. Thanks.

[Andreas Neufert]

If you open a ticket with trial and share here the number, the PM team can forward the logs as needed.

[Dima P.]

C:\Program Files\Veeam\Endpoint Backup\metadata\VSS\WritersData\{afbab4a2-367d-4d15-a586-71dbb18f8485}\{ebb7bd6d-5c42-4a68-b858-6a41da087162}\MetadataDoc.xml
C:\ProgramData\Veeam\OneAgent\Log\e2aeb2a5-0e3a-4fa9-addd-7a1d6e928e70\VbrRuntimeServicePackage_2025_02_12_12_53_10.zip

We've reviewed this issue and whitelisted the data created by our Veeam Agent for Windows and Veeam One agent. It should not affect guest file analytics metrics anylonger.

- clearing browser/OS/ other caches/temporary junk: post538251.html#p538251

Also addressed the issue mentioned it this post. Several Google Chrome cache locations will be excluded from file analytics metrics collection.

Thank you for the report!