-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Oct 27, 2017 4:28 pm
- Full Name: Brad Schubring
- Contact:
VM restore point deletion detection/alerting
Hello! Depending on where the solution lies, this topic could belong in the Veeam B&R, or Powershell forums. I was hoping for a Veeam One alert capability, so I'm starting here.
I'm interested in figuring out if it is possible to alert on malicious admin console/admin powershell activity that deletes backup data. (not the automated expiration of GFS backups of course)
I've reviewed the Veeam One alarms, and have not found anything yet, opened a case to ask support (Case #05917421), and reviewed the Windows events here.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
I'm not finding anything that is reporting/alerting on administrator activity yet. Has anyone else accomplished this? If so, I'd be interested in hearing the solution.
I'm interested in figuring out if it is possible to alert on malicious admin console/admin powershell activity that deletes backup data. (not the automated expiration of GFS backups of course)
I've reviewed the Veeam One alarms, and have not found anything yet, opened a case to ask support (Case #05917421), and reviewed the Windows events here.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
I'm not finding anything that is reporting/alerting on administrator activity yet. Has anyone else accomplished this? If so, I'd be interested in hearing the solution.
-
- Veeam Software
- Posts: 701
- Liked: 179 times
- Joined: Nov 01, 2016 11:26 am
- Contact:
Re: VM restore point deletion detection/alerting
Hello Brad,
Veeam ONE is able to collect most of the events from the VBR Veeam Backup event log. These events could be seen on the Tasks & Events tab in the UI. It is possible to create an alarm for such events.
Veeam ONE is also able to analyze the infrastructure and provide its own predefined alarms based on that. For example "VM with no backup", "Unusual job duration", "Suspicious incremental backup size".
Both features cover the most popular cases. Please let me know if you are interested in alarms for some specific console/admin activity.
Thanks
Veeam ONE is able to collect most of the events from the VBR Veeam Backup event log. These events could be seen on the Tasks & Events tab in the UI. It is possible to create an alarm for such events.
Veeam ONE is also able to analyze the infrastructure and provide its own predefined alarms based on that. For example "VM with no backup", "Unusual job duration", "Suspicious incremental backup size".
Both features cover the most popular cases. Please let me know if you are interested in alarms for some specific console/admin activity.
Thanks
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Oct 27, 2017 4:28 pm
- Full Name: Brad Schubring
- Contact:
Re: VM restore point deletion detection/alerting
Hi, yes. Sorry for the delayed response, I thought I had email alerts set up for replies.
I am looking for an alarm that covers manual restore point deletion by an admin/operator. This would most likely occur through the console, but could also be done via Veeam powershell. As it stands, I was asked "If an administrator was to log into the console right now and delete backups, how long would it take us to notice?" Immutable backups are helpful, but we'd like to be alerted in this kind of scenario, not reply on immutable backups and us noticing the restore points are missing in time.
I am looking for an alarm that covers manual restore point deletion by an admin/operator. This would most likely occur through the console, but could also be done via Veeam powershell. As it stands, I was asked "If an administrator was to log into the console right now and delete backups, how long would it take us to notice?" Immutable backups are helpful, but we'd like to be alerted in this kind of scenario, not reply on immutable backups and us noticing the restore points are missing in time.
-
- Veeam Software
- Posts: 1434
- Liked: 634 times
- Joined: Jul 17, 2015 6:54 pm
- Full Name: Jorge de la Cruz
- Contact:
Re: VM restore point deletion detection/alerting
Hello Brad,
I have not checked on PowerShell, but if you use the VBR Console, the next event will be triggered 10050. It appears like this in Windows:
![Image](https://www.dropbox.com/s/jacicd018atwkfi/2023-03-10_21-43-25.jpg?dl=1)
And like this on Veeam ONE:
![Image](https://www.dropbox.com/s/8mp9vyyfz1owq8a/2023-03-10_21-45-52.jpg?dl=1)
What I would probably do is take that current event in VONE, and create an alarm:
![Image](https://www.dropbox.com/s/r96n6uf87malq49/2023-03-10_21-47-12.jpg?dl=1)
![Image](https://www.dropbox.com/s/z5s92rtravsnh8w/2023-03-10_21-51-11.jpg?dl=1)
![Image](https://www.dropbox.com/s/myqf1w9tfdjaex7/2023-03-10_21-51-40.jpg?dl=1)
![Image](https://www.dropbox.com/s/okrcn6gpa6plcsd/2023-03-10_21-52-08.jpg?dl=1)
![Image](https://www.dropbox.com/s/vj5oybm7ns07k2z/2023-03-10_21-52-39.jpg?dl=1)
Here you have the easy way to send your alarms, the ones you prefer, to Slack/Teams additionally to an email, for extra awareness.
Let me check on Monday regarding deletions from PowerShell.
I have not checked on PowerShell, but if you use the VBR Console, the next event will be triggered 10050. It appears like this in Windows:
![Image](https://www.dropbox.com/s/jacicd018atwkfi/2023-03-10_21-43-25.jpg?dl=1)
And like this on Veeam ONE:
![Image](https://www.dropbox.com/s/8mp9vyyfz1owq8a/2023-03-10_21-45-52.jpg?dl=1)
What I would probably do is take that current event in VONE, and create an alarm:
![Image](https://www.dropbox.com/s/r96n6uf87malq49/2023-03-10_21-47-12.jpg?dl=1)
![Image](https://www.dropbox.com/s/z5s92rtravsnh8w/2023-03-10_21-51-11.jpg?dl=1)
![Image](https://www.dropbox.com/s/myqf1w9tfdjaex7/2023-03-10_21-51-40.jpg?dl=1)
![Image](https://www.dropbox.com/s/okrcn6gpa6plcsd/2023-03-10_21-52-08.jpg?dl=1)
![Image](https://www.dropbox.com/s/vj5oybm7ns07k2z/2023-03-10_21-52-39.jpg?dl=1)
Here you have the easy way to send your alarms, the ones you prefer, to Slack/Teams additionally to an email, for extra awareness.
Let me check on Monday regarding deletions from PowerShell.
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Oct 27, 2017 4:28 pm
- Full Name: Brad Schubring
- Contact:
Re: VM restore point deletion detection/alerting
Thank you for the detailed reply. It's exactly what I'm looking for, and was the original direction I was looking when I opened my ticket.
My next challenge in regard to using this method that I stated in my ticket (but not here, sorry) is that I do not see event 10050 when I delete a restore point from the repo via the Files tab. I just deleted the oldest VBK full I had for a non-prod server, and I did not receive an event in the Veeam Backup log. I only see restore point creations, start/stopping of jobs, and automated restore point removals using 10050. I also checked the Events view in VeeamOne and verified the deletion was not captured there.
My next challenge in regard to using this method that I stated in my ticket (but not here, sorry) is that I do not see event 10050 when I delete a restore point from the repo via the Files tab. I just deleted the oldest VBK full I had for a non-prod server, and I did not receive an event in the Veeam Backup log. I only see restore point creations, start/stopping of jobs, and automated restore point removals using 10050. I also checked the Events view in VeeamOne and verified the deletion was not captured there.
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Oct 27, 2017 4:28 pm
- Full Name: Brad Schubring
- Contact:
Re: VM restore point deletion detection/alerting
Hi, I thought of something I don't fully understand here aside from the fact that I don't see 10050 for manual deletions. 10050 is generated for automatic restore point deletions. Does the above method filter the automatic deletions somehow?
-
- Veeam Software
- Posts: 1434
- Liked: 634 times
- Joined: Jul 17, 2015 6:54 pm
- Full Name: Jorge de la Cruz
- Contact:
Re: VM restore point deletion detection/alerting
Hello,
As you can see on my lab, I triggered the delete of the point manually, removing a point automatically at 09.42:42 would be strange, could happen, but no it was manually triggered.
Can you please explain from where exactly you removed the point, from here?:
![Image](https://www.dropbox.com/s/l3vvniaxnfpa6fx/2023-03-14_16-46-09.jpg?dl=1)
If that is so, Veeam ONE does not get an alarm, as VBR currently does not have a mechanism to understand you have deleted a file from there. Same as if you delete it directly in the OS.
To avoid these kind of manual (or automated/ransomware/unexpected) deletions I would recommend to leverage the vast range of immutability that Veeam allows today.
Perhaps a periodic check of what is in disk vs DB could help here, but either way it will be reactive rather than proactive like following the 3-2-1 rule with immutability.
Please let us know.
As you can see on my lab, I triggered the delete of the point manually, removing a point automatically at 09.42:42 would be strange, could happen, but no it was manually triggered.
Can you please explain from where exactly you removed the point, from here?:
![Image](https://www.dropbox.com/s/l3vvniaxnfpa6fx/2023-03-14_16-46-09.jpg?dl=1)
If that is so, Veeam ONE does not get an alarm, as VBR currently does not have a mechanism to understand you have deleted a file from there. Same as if you delete it directly in the OS.
To avoid these kind of manual (or automated/ransomware/unexpected) deletions I would recommend to leverage the vast range of immutability that Veeam allows today.
Perhaps a periodic check of what is in disk vs DB could help here, but either way it will be reactive rather than proactive like following the 3-2-1 rule with immutability.
Please let us know.
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software
@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Senior Product Manager | Veeam ONE @ Veeam Software
@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Oct 27, 2017 4:28 pm
- Full Name: Brad Schubring
- Contact:
Re: VM restore point deletion detection/alerting
I understand that preventing the deletion, or mitigating it via immutability is the solution. However, I would argue that knowing that the restore point was deleted is an important part of recovering. Immutability is not typically as lengthy of a time period as overall retention. This leaves a gap for backups to age out of immutability, but still be in retention if you don't realize that the deletion happened.
I'm going to make a Feature Request post in the B&R forum around this, if the deletion itself through the console cannot create an event, perhaps an event for when a backup occurs and part of the expected backup chain is missing could suffice.
I appreciate your time and expertise in this explanation!
I'm going to make a Feature Request post in the B&R forum around this, if the deletion itself through the console cannot create an event, perhaps an event for when a backup occurs and part of the expected backup chain is missing could suffice.
I appreciate your time and expertise in this explanation!
-
- Veeam Software
- Posts: 1434
- Liked: 634 times
- Joined: Jul 17, 2015 6:54 pm
- Full Name: Jorge de la Cruz
- Contact:
Re: VM restore point deletion detection/alerting
Thank you, Brad. Just saw your FR on the other thread. Great stuff, if the Console creates an event for deletion from the Files tab, and if they introduce that chain-check of files, we will definitely add them to VONE.
Thank you!
Thank you!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software
@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Senior Product Manager | Veeam ONE @ Veeam Software
@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Who is online
Users browsing this forum: No registered users and 12 guests