WillBeRetried property through REST api

[M4rco]

We are monitoring the result of our Veeam backup jobs using Zabbix. Currently it depends on Powershell scripts which queries job status and presents it in a form so Zabbix can work with it.

Now we want to move away from the Powershell setup and use the Veeam VBR REST api for determining job status of the latest backup run.

All our backup jobs have a retry scheme configured (retryCount = 3, awaitMinutes=10).

Now when I query /api/v1/jobs/states endpoint I can get the content of property lastResult which tells me if a backup job run succeeded or failed. But if a job fails the retry scheme kicks in and the job may still succeed.

We don't want our monitoring to alert for a failed job while it is still in its retry scheme. With Powershell this is accomplished like this:

$job = Get-VBRJob -Name MKL-rest-test
$joblastsession=$job.FindLastSession()
$joblastsession.WillBeRetried => this property tells me if a retry is scheduled for this job

How can I determine the retry status of a job using the VBR REST api? I also checked the Veeam Enterprise Manager REST api but could not find it there either.

Veeam 11.01.1261 P20220302
REST version V1-REV2

[jorgedlcruz]

Good morning, and thanks for the post. I genuinely like Zabbix, and I have seen so many Customers using it.

So, I have been taking a look, and so far:

• If you query the jobs API - https://YOURVBRIP:9419/api/v1/jobs and then do a jq of the response to .data[].schedule.retry you will see something like this per Job. Meaning you can go deep and assign each of those values to variables and save it for later on:

  Code: Select all

  {
    "isEnabled": true,
    "retryCount": 3,
    "awaitMinutes": 10
  }
  

• For the retries on the Job, please give it a try in the session logs? I think the retries are there; this URL https://YOURVBRIP:9419/api/v1/sessions/ ... 61217/logs I see something like this for an snapshot job:

  Code: Select all

  {
    "totalRecords": 8,
    "records": [
      {
        "status": "Failed",
        "id": 8,
        "startTime": "2022-05-10T02:06:13.1618675+01:00",
        "updateTime": "2022-05-10T02:06:13.1618675+01:00",
        "title": "Job finished with error at 5/10/2022 2:06:13 AM",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 7,
        "startTime": "2022-05-10T02:05:25.1129755+01:00",
        "updateTime": "2022-05-10T02:05:25.128603+01:00",
        "title": "VM 'NGINX-004' is outdated and its restore points will be subjected to deleted VMs retention",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 6,
        "startTime": "2022-05-10T01:57:22.1704122+01:00",
        "updateTime": "2022-05-10T02:05:23.8629896+01:00",
        "title": "All VMs have been prepared for storage snapshot",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 5,
        "startTime": "2022-05-10T01:57:22.139191+01:00",
        "updateTime": "2022-05-10T01:57:22.139191+01:00",
        "title": "Required backup infrastructure resources have been assigned",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 4,
        "startTime": "2022-05-10T01:57:22.1079401+01:00",
        "updateTime": "2022-05-10T01:57:22.1079401+01:00",
        "title": "Queued for processing at 5/10/2022 1:57:22 AM",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 3,
        "startTime": "2022-05-10T01:57:21.9516866+01:00",
        "updateTime": "2022-05-10T01:57:21.9516866+01:00",
        "title": "NGINX-004 is no longer processed by this job. Make sure this change is intentional",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 2,
        "startTime": "2022-05-10T01:57:14.8110469+01:00",
        "updateTime": "2022-05-10T01:57:20.061048+01:00",
        "title": "Building list of machines to process",
        "description": ""
      },
      {
        "status": "Succeeded",
        "id": 1,
        "startTime": "2022-05-10T01:57:14.6235471+01:00",
        "updateTime": "2022-05-10T01:57:14.6235471+01:00",
        "title": "Job started at 5/10/2022 1:57:08 AM",
        "description": ""
      }
    ]
  }

Give it a quick try and let us know if this good enough for you.

[M4rco]

Hi Jorge,

Thank you for your response. I have been using the Swagger interface to try the different endpoints.

https://YOURVBRIP:9419/api/v1/jobs => this endpoint gives the job configuration, there is no information related to actual retry status

https://YOURVBRIP:9419/api/v1/sessions/{id}/logs => I have tried this and I have these findings:

1) during initial failure there is nothing in the session logs showing the session failed and it will be retried
2) during first retry there is nothing showing in the session logs this is a new session due to a retry

so it seems this path is a dead end?

[bcram]

Interested as well

[jorgedlcruz]

Hello, are you using the v1-rev2 from the top option? That should have the retry parameter I've shared.


What is the result of your job session, using the logs call using the v1-rev2, the same?

[M4rco]

hi Jorge,

Yes we are using REST version V1-REV2. I am aware of possible differences caused by different REST versions so for that reason I mentioned my used REST version in my opening post.

here is an example of the last entries from the sessions logs endpoint for my failing backup session

Code: Select all

{
  "totalRecords": 12,
  "records": [
    {
      "status": "Failed",
      "id": 12,
      "startTime": "2022-05-17T15:35:11.3998177+02:00",
      "updateTime": "2022-05-17T15:35:11.3998177+02:00",
      "title": "Job finished with error at 17-5-2022 15:35:11",
      "description": ""
    },
    {
      "status": "Succeeded",
      "id": 11,
      "startTime": "2022-05-17T15:34:42.3824716+02:00",
      "updateTime": "2022-05-17T15:34:42.8043418+02:00",
      "title": "Deleting storage snapshots",
      "description": ""
    },
    {
      "status": "Failed",
      "id": 9,
      "startTime": "2022-05-17T15:32:27.6932181+02:00",
      "updateTime": "2022-05-17T15:34:41.8512182+02:00",
      "title": "Processing dc-ans-p01 Error: Failed to connect to the port [dc-rpo-p0x.company.com:2501].\n--tr:Failed to connect to target endpoint.\n--tr:Client failed to process the command. Command: [connectByIPs].\n--tr:event:1:\n--tr:event:3:\n",
      "description": ""
    },

No more log entries were generated after this failure. As you can see there really is no indication this session failed and the corresponding job will be retried. After this Veeam initiated a retry of the backup job, which resulted in a new job session, but again no retry indicator.

I have doublechecked the output you shared (from sessions/{id}/logs endpoint) as well and also in there there is no retry information...

[oleg.feoktistov]

If we are talking about an analogue for CBackupSession.WillBeRetried property from powershell, it is definitely not a session log member, but a session's. Overall, unfortunately, currently there is no clear indicator in REST if a session will be retried. So, thanks for your feedback. I noted it for further enhancements in SessionModel of VBR REST spec. Cheers!

[M4rco]

Hi Oleg,

Thank you for confirming this property is not available through any of the REST methods of the VBR REST api. I will search no longer.

As said I also tried finding this property through Veeam Enterprise Manager REST api but could not find it either. Can you confirm it is also not available via that REST api?

[oleg.feoktistov]

Hi,

Correct, there is no such indication in EM REST either. There is IsRetry property inside backup session object, but it reflects whether the session is a retry itself (analogue of CBackupSession.IsRetryMode in powershell).

Thanks,
Oleg

[M4rco]

Again, thank you for confirming.

Now I have to think of a different way of deducting if a failed job may still be in its retry schedule.

[oleg.feoktistov]

I'm afraid, for now the only way is powershell. You made a great point though, I'll make sure we add WillBeRetried and IsRetry properties to SessionModel spec for VBR REST in one of the next product releases. Thanks!