VM restore point deletion detection/alerting

[Brad Schubring]

Hello! Depending on where the solution lies, this topic could belong in the Veeam B&R, or Powershell forums. I was hoping for a Veeam One alert capability, so I'm starting here.

I'm interested in figuring out if it is possible to alert on malicious admin console/admin powershell activity that deletes backup data. (not the automated expiration of GFS backups of course)

I've reviewed the Veeam One alarms, and have not found anything yet, opened a case to ask support (Case #05917421), and reviewed the Windows events here.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

I'm not finding anything that is reporting/alerting on administrator activity yet. Has anyone else accomplished this? If so, I'd be interested in hearing the solution.

[RomanK]

Hello Brad,

Veeam ONE is able to collect most of the events from the VBR Veeam Backup event log. These events could be seen on the Tasks & Events tab in the UI. It is possible to create an alarm for such events.

Veeam ONE is also able to analyze the infrastructure and provide its own predefined alarms based on that. For example "VM with no backup", "Unusual job duration", "Suspicious incremental backup size".

Both features cover the most popular cases. Please let me know if you are interested in alarms for some specific console/admin activity.

Thanks

[Brad Schubring]

Hi, yes. Sorry for the delayed response, I thought I had email alerts set up for replies.

I am looking for an alarm that covers manual restore point deletion by an admin/operator. This would most likely occur through the console, but could also be done via Veeam powershell. As it stands, I was asked "If an administrator was to log into the console right now and delete backups, how long would it take us to notice?" Immutable backups are helpful, but we'd like to be alerted in this kind of scenario, not reply on immutable backups and us noticing the restore points are missing in time.

[jorgedlcruz]

Hello Brad,
I have not checked on PowerShell, but if you use the VBR Console, the next event will be triggered 10050. It appears like this in Windows:


And like this on Veeam ONE:


What I would probably do is take that current event in VONE, and create an alarm:






Here you have the easy way to send your alarms, the ones you prefer, to Slack/Teams additionally to an email, for extra awareness.

Let me check on Monday regarding deletions from PowerShell.

[Brad Schubring]

Thank you for the detailed reply. It's exactly what I'm looking for, and was the original direction I was looking when I opened my ticket.

My next challenge in regard to using this method that I stated in my ticket (but not here, sorry) is that I do not see event 10050 when I delete a restore point from the repo via the Files tab. I just deleted the oldest VBK full I had for a non-prod server, and I did not receive an event in the Veeam Backup log. I only see restore point creations, start/stopping of jobs, and automated restore point removals using 10050. I also checked the Events view in VeeamOne and verified the deletion was not captured there.

[Brad Schubring]

Hi, I thought of something I don't fully understand here aside from the fact that I don't see 10050 for manual deletions. 10050 is generated for automatic restore point deletions. Does the above method filter the automatic deletions somehow?

[jorgedlcruz]

Hello,
As you can see on my lab, I triggered the delete of the point manually, removing a point automatically at 09.42:42 would be strange, could happen, but no it was manually triggered.

Can you please explain from where exactly you removed the point, from here?:


If that is so, Veeam ONE does not get an alarm, as VBR currently does not have a mechanism to understand you have deleted a file from there. Same as if you delete it directly in the OS.

To avoid these kind of manual (or automated/ransomware/unexpected) deletions I would recommend to leverage the vast range of immutability that Veeam allows today.

Perhaps a periodic check of what is in disk vs DB could help here, but either way it will be reactive rather than proactive like following the 3-2-1 rule with immutability.

Please let us know.

[Brad Schubring]

I understand that preventing the deletion, or mitigating it via immutability is the solution. However, I would argue that knowing that the restore point was deleted is an important part of recovering. Immutability is not typically as lengthy of a time period as overall retention. This leaves a gap for backups to age out of immutability, but still be in retention if you don't realize that the deletion happened.

I'm going to make a Feature Request post in the B&R forum around this, if the deletion itself through the console cannot create an event, perhaps an event for when a backup occurs and part of the expected backup chain is missing could suffice.

I appreciate your time and expertise in this explanation!

[jorgedlcruz]

Thank you, Brad. Just saw your FR on the other thread. Great stuff, if the Console creates an event for deletion from the Files tab, and if they introduce that chain-check of files, we will definitely add them to VONE.

Thank you!