Monitoring and reporting for Veeam Data Platform
Post Reply
kevdpc
Influencer
Posts: 20
Liked: 2 times
Joined: Feb 18, 2020 5:45 pm
Full Name: Kevin Chubb
Contact:

Suspicious incremental backup size - how do you investigate this?

Post by kevdpc »

Sometimes I get the alert for "Suspicious incremental backup size" when I don't expect any significant changes like a large amount of data added to a VM in the job, adding VMs to the job, etc.

How do you investigate this if you're not sure what caused the large incremental backup?

Do others get these unexpectedly?

What are some other things that could trigger this alarm other than ransomware?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Suspicious incremental backup size - how do you investigate this?

Post by wishr »

Hi Kevin,

You may use the VM Change Rate History and Job History reports to get more information about the VM size changes and act from there. You may also want to involve the owner of the applications running on this VM to get more understanding of the situation once it's understood when exactly changes that caused the alarm to trigger happened.

It may be natural for a VM to have a high change rate, depending on what kind of applications are running there. In this case, you may adjust alarm thresholds accordingly or exclude that VM from the original alarm and create a separate individual alarm for it with specific thresholds.

Thanks
bkain1
Expert
Posts: 137
Liked: 8 times
Joined: Dec 23, 2020 4:43 pm
Full Name: Becki Kain
Contact:

Re: Suspicious incremental backup size - how do you investigate this?

Post by bkain1 »

Interesting. I have the same issue but the suspicious backup size is on the SQL backend to the VeeamOne box. Ideas?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Suspicious incremental backup size - how do you investigate this?

Post by wishr »

Hi Backi,

Nothing to add to my post above. Do you have any particular questions?

Thanks
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Suspicious incremental backup size - how do you investigate this?

Post by Vitaliy S. »

SQL Server is a highly transactional application, so looks like you had some activity on that server at the time when that alarm was triggered.
dspringer
Enthusiast
Posts: 32
Liked: 3 times
Joined: Feb 01, 2022 10:57 am
Full Name: David Springer
Contact:

[MERGED]Re: Dealing with the "possible Ransomware" Alarms

Post by dspringer »

I found this topic matching my concern and of course didn't want to make a new one.

I am currently testing VeeamOne in an unlicensed version to show the advantages of the software to my colleagues. I myself am already enthusiastic. Even though my SQL Express is already being admonished, where I only have the one main backup server with its various remote backup proxy connections and just 3 of our 10 vCenters connected *grin*.

My concern is this, which actually fits with the first post. Full and incremental backups are running. For servers where little changes, the amount of incremental file can of course change every now and then. But why is this interpreted as an error? Because in the case of ransomware, files are encrypted, so they may be recognised as 0kb and thus the total backup amount fluctuates? But what is the best way to set the warning threshold? I still have the standard with "Below" warning at 80% and error at 70%.

For example my Error was "Size of incremental backup created by "Jobname" job (31.1%) is below the configured threshold (70.0%)
Incremental backup creation time 2022-06-15 23:00:47 (UTC+2:00)"
RomanK
Veeam Software
Posts: 628
Liked: 164 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Suspicious incremental backup size - how do you investigate this?

Post by RomanK »

Hello David,

"Possible ransomware activity" alarm detected suspicious activity on the virtual machine by evaluating CPU usage, datastore write rate and network transmit rate. While the "Suspicious incremental backup size" alarm evaluates the size of increments. That is why I moved your post to this thread.

Over time, incremental backup points are fairly consistent in terms of storage consumption. However, ransomware could affect the size. Unexpected file system encryption or mass deletion would be easily recognized. The best way to start an investigation would be the second post, please take a look.

However, there is no best way to set up this particular alarm for everyone. You should adjust the rules of this alarm to fit your environment.

Thanks
ValiantMartian
Enthusiast
Posts: 36
Liked: 5 times
Joined: Oct 28, 2019 6:02 pm
Contact:

Re: Suspicious incremental backup size - how do you investigate this?

Post by ValiantMartian »

Just wanted to point out this blog post and tool to identify WHAT is causing unusual incremental backup sizes.
https://www.veeam.com/blog/big-incremental-backup.html
Post Reply

Who is online

Users browsing this forum: No registered users and 15 guests