Alert on Specific Veeam Backup for M365 Warning

[EricinIT]

Hi,

I want to alert when a specific Warning occurs in any Veeam Backup for M365 job. I've got Veeam ONE integrated with Veeam Backup for M365. When the job ends in a warning and the reason for the warning is "Item may have a virus reported by the virus scanner plug-in", I need to send this over to the M365 team to have them investigate. I looked at the Veeam ONE alarms and I can email on any warning with the "Backup job state (Veeam Backup for Microsoft 365)" Alarm. However, I don't need to send every warning to the M365 team. Also, I tried Event-Based rules but Veeam ONE doesn't seem to trigger an event for this.

Am I missing something or does anyone have ideas how to accomplish this?

[jorgedlcruz]

Hello Eric,
I do not think this is possible today as the alarms are somewhat rigid.

However, I think by either using Syslog, and on the syslog system filter by that specific text, or any other, and alarm your team could work.

Another option is to use the variables per alarm, to capture this very specific alarm on warning, and only when the text you want appears, then trigger an email, all of this will be a mix of PowerShell and VONE, I can help you with all of this tomorrow, if you want to pursue this avenue.

Meanwhile let me discuss internally this FR for future releases.

Thank you

[EricinIT]

Yes, I'd be interested in the PowerShell with VONE to get an email to send off with the trigger text "Item may have a virus reported by the virus scanner plug-in".

Eric

[jorgedlcruz]

Hello Eric,
So I have been doing research, and sadly at the moment we are not collecting the necessary data.

Let me explain, we of course do collect ton of great data for VB365, that is how we can produce very comprehensive reports, alarms and dashboards.

It is just that for the Job Policy VB365 has a few endpoints for the end result:

• https://127.0.0.1:4443/v7/JobSessions
• https://127.0.0.1:4443/v7/JobSessions/1 ... e/LogItems

Currently, we rely for the alarm, report, etc. On just the initial endpoint, which result is the next one:

Code: Select all

{
  "offset": 0,
  "limit": 30,
  "_links": {
    "self": {
      "href": "/v7/JobSessions?jobid=810513da-5cd0-4c83-9bae-03d558569333&offset=0&limit=30"
    },
    "next": {
      "href": "/v7/JobSessions?jobid=810513da-5cd0-4c83-9bae-03d558569333&offset=30&limit=30"
    }
  },
  "results": [
    {
      "id": "1dd0dcfc-65b4-4e5a-8de1-ef145da9b081",
      "repositoryId": "fb37151b-4a83-4c1e-8e79-df3cd67330a5",
      "mainSessionId": "b8df056e-2dd3-4303-b01f-283d127ec124",
      "details": "Some objects were processed with warnings",
      "creationTime": "2023-12-18T13:01:50.3119559Z",
      "endTime": "2023-12-18T13:03:01.9408366Z",
      "retryCount": 2,
      "jobWillBeRetried": true,
      "progress": 0,
      "jobType": "Backup",
      "status": "Warning",
      "statistics": {
        "processingRateBytesPS": 19,
        "processingRateItemsPS": 0,
        "readRateBytesPS": 159637,
        "writeRateBytesPS": 0,
        "transferredDataBytes": 1399,
        "processedObjects": 0,
        "bottleneck": "Target"
      },
      "_links": {
        "self": {
          "href": "/v7/jobsessions/1dd0dcfc-65b4-4e5a-8de1-ef145da9b081"
        },
        "log": {
          "href": "/v7/jobsessions/1dd0dcfc-65b4-4e5a-8de1-ef145da9b081/logitems"
        },
        "job": {
          "href": "/v7/jobs/810513da-5cd0-4c83-9bae-03d558569333"
        }
      }
    },
    

The other endpoint, the LogItems, has the details we are looking for:

Code: Select all

...
    {
      "id": "018e10f5-12c0-40ef-a474-086b56bc1e28",
      "usn": 1001,
      "title": "[Warning] Processing site https://jorgedelacruz.sharepoint.com/sites/TestTeamsSite finished with warning: Failed to backup item: /sites/TestTeamsSite/Shared Documents/eicar.com.txt, Item may have a virus reported by the virus scanner plug-in",
      "creationTime": "2023-12-18T13:13:03.9464581Z",
      "endTime": "2023-12-18T13:14:13.2915674Z",
      "_links": {
        "self": {
          "href": "/v7/jobsessions/1fc06510-834f-470f-b189-d6cba3904fde/logitems/018e10f5-12c0-40ef-a474-086b56bc1e28"
        },
        "jobsessions": {
          "href": "/v7/jobsessions/1fc06510-834f-470f-b189-d6cba3904fde"
        }
      }
    },
    ...

So, what Veeam ONE will display is just that:


Next steps
From my side, I have included a feature request for the future to capture the extended information and add it as part of the alarm/reports. I can not promise any date, but surely it will made it in the future.

Workarounds
There are a few, I am not sure which one can be useful for you, so I hope to give you some ideas:

• Parsing Logs, from logs to Email/Ticket System: With the raise of Grafana Loki, or any other technology that can parse millions of log lines and show it to you, like Splunk, Elastic, etc. This is super easy. You should pass the directory C:\ProgramData\Veeam\Backup365\Logs\YOUR ORGANIZATION\YOURJOB and then once ingested, on the system create Alarms on specific lines, like these ones on the log:

  Code: Select all

  [19.12.2023 08:40:21.615]  191 (5672) Warning: Failed to backup item: /sites/TestTeamsSite/Shared Documents/eicarcom2.zip, Item may have a virus reported by the virus scanner plug-in
  

  The extra good news is that the logs even have the affected files, etc. You can parse them too:

  Code: Select all

  [19.12.2023 08:40:21.615]  191 (5672) Received an infected item warning: /sites/TestTeamsSite/Shared Documents/eicar_com.zip

• API to Email: Another idea could be to have a PS script that checks the endpoints I shared every x minutes, or hours, depending of your job frequency, looking for that specific line "Item may have a virus reported by the virus scanner plug-in" on the title of the LogItems endpoint.

I understand this is not all plug and play, so I hope we can bring it in the future, and meanwhile you can explore some of those options.

Thank you

[EricinIT]

Hi Jorge,

Thank you for detailed explanation! I appreciate the feature request and I'll use a workaround for the time being.

Eric

[rtheseeker]

I use a script to get this data everyday and use task scheduler to send it to respective teams as we have both Mailbox team and OneDrive team

Code: Select all

$lastNight = (Get-Date).AddDays(-1).ToString("MM/dd/yyyy")

# Get all job sessions that ran last night with warnings
$lastNightSessions = Get-VBOJobSession | Where-Object {
    $_.CreationTime -ge $lastNight -and
    $_.CreationTime -lt (Get-Date).ToString("MM/dd/yyyy") -and $_.JobName -like '*OneDrive*'
}

# Initialize $logs before using it
$logs = @()
$jobs =@()
$filePath = 'C:\temp\OneDriveerror.txt'

foreach ($lastNightSession in $lastNightSessions) {
    $logs += $lastNightSession.Log
    $jobs += $lastNightSession.JobName
}

$i = 0
$logArray = @()

# Display only warning logs excluding specific warnings
foreach ($log in $logs) {
    if ($log.title -match 'error|warning' -and 
        $log.title -notlike '*OneDrive was not found*' -and 
        $log.title -notlike '*does not have a valid Microsoft 365 license with SharePoint plan enabled*' -and 
        $log.title -notlike '*Item may have a virus reported by the virus scanner plug-in*' -and 
        $log.title -notlike '*Unable to send email notification*' -and 
        $log.title -notlike '* Job finished at *') {

        $formattedLog = "{0} {1}" -f $log.title, $log.description
        $logArray += $formattedLog
    }
}

# Convert the log array to HTML
$htmlContent = $logArray -join "<br>"

# Define the file path where you want to save the HTML content
$htmlFilePath = 'C:\temp\error.html'
    
    
# Save the HTML content to a file
    $htmlContent | Set-Content -Path $htmlFilePath

[jorgedlcruz]

Thank you Rajeev,
Hope it helps others. There are as well other options like using VB365 API, I've created as well a simple script that does all of this for you:
https://jorgedelacruz.uk/2021/08/03/vee ... y-project/

Maybe with some modifications, and having an script checking on VB365 events, can perhaps notify them in real-time.

Hope we can add this in the future so it becomes less complex.

Thanks for sharing.