Relate VM objects to jobs

[pdrangeid]

I am trying to validate what VMs SHOULD be configured for Veeam, and which Are, and which backups have success within x days.

Basically validate that we have configured the correct machines for backup jobs, and they are indeed running.

Do do this I have all my machines in our asset management DB, and pulled that into a graphDB (neo4j). I am now using the VeeamAPI to query and validate that those VMs are indeed getting backed up.

I queried jobs via api/jobs
I queried a list of all the VMs using api/lookupSvc (got .Links where .HierarchyItemList ends with =Vm)
I queried restorepoints by api/query?type=VmRestorePoint&format=Entities&filter=HierarchyObjRef==%22"+vvm.id+"%22;CreationTime%3E"+backupdate

So now I know WHICH vms have backups completed successfully, and I put them into buckets (last 12 hours, last 1 day, 2 days, 7 days, 14 days, longer, or never)

The problem is that those VMs that have "NEVER" been backed up, I'm not sure if they are a part of any Veeam jobs. The VMs that are collected, appear to be all VMs that veeam "saw" through the integration with VMware. I tried digging into the api/jobs/{ID}/includes endpoint, but it seems to only show me the folder objects that define the job, and not the individual VMs that become a part of the backup.

Is there any way to query the api to determine which VMs are a part of which jobs?

[jhoughes]

The '/jobs/{ID}/includes' endpoint will show you the exact object which is targeted for the VM job, including any resource containers which may have been selected.

This means that if your jobs are targeting resource containers such as clusters/resource pools/VM folders/datastores/vSphere tags, then those are the HierarchyObjRef objects which will be returned by the endpoint.

In my case, for a job targeting a tag, this is what gets returned (trimmed for space):

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<ObjectsInJob xmlns="http://www.veeam.com/ent/v1.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <ObjectInJob Href="http://ausveeambem.lab.fullstackgeek.net:9399/api/jobs/c56fc07e-38e6-4672-902f-87b834d73305/includes/8d75eba8-cebc-4b11-b31c-818a49a03002" Type="ObjectInJob">
        <ObjectInJobId>8d75eba8-cebc-4b11-b31c-818a49a03002</ObjectInJobId>
        <HierarchyObjRef>urn:VMware:Tag:fdd958d1-0a04-410a-964f-0f8f9c13a5fe.urn:vmomi:InventoryServiceTag:512c257c-4a66-4cce-9578-34bb8df48f4b:GLOBAL</HierarchyObjRef>
        <Name>VeeamTagTest</Name>
        <DisplayName>VeeamTagTest</DisplayName>
        <Order>0</Order>
        <GuestProcessingOptions>
        </GuestProcessingOptions>
    </ObjectInJob>
</ObjectsInJob>

Within this reference object, there is a specific type designation (a 'Tag' in my case), as documented here: https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

You would need to query your vSphere environment to determine the VMs that are contained within each specific reference objects (in your case a VM folder), and therefore which job those VMs are contained within.

[pdrangeid]

So, I cannot determine (from Veeam API alone) which VMs a Veeam job is protecting? and instead requires me to then query a VMware API?

This seems very strange, as Veeam is highly aware of VMs. (I'm able to find them with the /api/query tool) but it doesn't track which VMs ended up attached to a particular job?

[jhoughes]

If you are wanting to at least check the last run of the backup job to see which VMs were included, you could follow the '/backupSessions' link to get all sessions of that specific job.

From there, you can parse out the most recent session, or any session on a particular date if you want to catch all retries as well.

From any specific backup session entity, you will get a '/taskSessions' link which will give you each task session for that particular backup job run.

Within the task session list, you'll have an entity for each VM which was included in the job run.

This would probably be the closest to a list of VMs that you will get for your intended purposes, at least off the top of my head. That list may still only be valid for a specific date or job run, as if a VM has moved within your environment then it may no longer a candidate within a backup job, once the vSphere inventory data is updated.

[pdrangeid]

OK - that makes sense... I'll have to dig into the task session list.

Another question - I use folders, and will create 'VEEAM EXCLUDE" folders, which I then "exclude" from the job. How can I tell via the API that the folder within a job is marked for EXCLUSION vs INCLUSION.. they all looked the same at first glance when I saw the folder objects appearing in the "/job/include" query

[jhoughes]

So, I cannot determine (from Veeam API alone) which VMs a Veeam job is protecting? and instead requires me to then query a VMware API?

This seems very strange, as Veeam is highly aware of VMs. (I'm able to find them with the /api/query tool) but it doesn't track which VMs ended up attached to a particular job?

The job is not targeting individual VMs, it is targeting a container (or sorts). There is a dynamic list of child objects within the job, which gets updated from the cache held by the broker service. The final list of VMs for the job run is created when the job session begins so that the session includes the proper child resources.

The information being presented back from the API endpoint is the actual resource included within the job, it just does not include child resources.

[chris.arceneaux]

I’d be remiss if I didn’t mention another method for tracking unprotected VMs. If the Backup Jobs used vSphere tags as that container Joe mentioned, then you’d only need to find out which VMs are lacking a tag.

Another option to mention is that Veeam ONE has a pre-built report with this information.

[pdrangeid]

Ok - getting really close.

I've got all the VMs via: base-veeam-api-url/lookupSvc
I get the Links where type=HierarchyItemList and retrieve the VM name and the vmid and I create unique nodes for each (:Veeamvm). I also store the .ObjectRef as a property for reference.

Then I'm querying the JobSession via base-veeam-api-url/query?type=BackupJobSession&format=Entities and then delve .Entities.BackupJobSessions.BackupJobSessions to get the unique session data.
Then I query the .Links (where .Type='BackupServerReference') and query that.

Finally I unwind .Refs (where .Type='BackupTaskSessionReference' ) with the intent to match a (:Veeamvm) So I can map a relationship [:INCLUDED_IN_VEEAM_JOB].

Here's the problem. The data in the .Links where .Type = "BackupTaskSessionReference" only "slightly" indicates the VM name:
"Name": "my.vm.fqdn@2019-10-24 05:39:46"

Unfortunately this doesn't use any unique "VM ids (not the vm-###, nor the veeam stil UID like urn:VMware:Vm:8708fde2-61ad-42ee-bed9-94ccf994a3a0.vm-487)

So if I ever collected 2 VMs with the same name (even in different folders, resource groups, etc) - I would have no way to identify WHICH vm the BackupTaskSessionReference is actually referring to.
the Href for the backuptasksessions/{id} at this point seems to be self-referring and just shows the same .Links data.
Shouldn't there be another property within the BackupTaskSessionReference link that provides the .uid or objid of the VM we are talking about?

[chris.arceneaux]

Hi Paul,

Kudos on the effort! Looks like you've come pretty far. The method you've mentioned ends with the caveat that you mentioned in that you can't match to a unique ID. I have a method that should work for you though and provide you the unique ID to compare against.

1) /lookup?.... - You're already doing this to gather list of VMs with ObjectRef and placing them in an array.

2) /query?type=VmRestorePoint&format=Entities&pageSize=1000&page=1 - This API call will you give you restore points for every VM that Enterprise Manager knows about. Depending on the amount of restore points you have, the results will be paginated so keep that in mind in your code. Included in this call is the VM ObjectRef that you got from your first call. Added bonus: You also get a time stamp so you can exclude restore points older than XX number of days.

3) Loop through restore points and remove VMs with matching ObjectRefs from your lookupSvc array you created.

4) What's remaining in the lookupSvc array are VMs with no restore point so they're not protected.

[pdrangeid]

Aah - forgot to mention - I actually had already collected the VM lists from the VMRestorePoint... I've got a great relationship between those.. I'm trying to add the job-to-vm relationship, and having a hard time get those mapped (it works, but using VM name only is going to be easy to mis-connect a vm that has the same vm name (I know not terribly likely, but still could happen)

[pdrangeid]

FYI This is a quick example of the knowledge graph I am generating with this tool:

[chris.arceneaux]

Nice! To relate definitively VMs to Backup Jobs (without using vSphere APIs), the restore points would be the way to go. To demonstrate...

1) /vmRestorePoints?format=Entity - You're doing this already either through this call or using the QuerySvc.

2) One of the links in each vmRestorePoint will have a type of RestorePointReference.



3) Making a GET request of the RestorePointReference will tell you the Backup Job name along with the job's unique ID. This information is found in the link with a type of BackupReference.



With all this said, this approach will essentially add an extra API call for every vmRestorePoint you analyze. Depending on the size of your environment, this could add a good bit of time to your automation which might be a problem for you. If you need to optimize to reduce the run time of your automation, I recommend you leverage the vSphere API using the method Joe previously outlined.

[pdrangeid]

Ok - I think I got that working..

How can I tell if a job is a "backup" or a "copy" job?

Also how can I tell if a folder item within a job include is INCLUDING or EXCLUDING the contents?

Thanks!

[chris.arceneaux]

How can I tell if a job is a "backup" or a "copy" job?

Running GET /jobs?format=Entity will present you with the JobType field. You probably already have this information in a variable. If not, just put it in a variable early in your script for use later on.

Backup for a Backup Job and BackupCopy for a Backup Copy Job.

Also how can I tell if a folder item within a job include is INCLUDING or EXCLUDING the contents?

Can you provide a little more context?

[pdrangeid]

http://<Enterprise-Manager>:9399/api/jobs/{ID}/includes/{ID}

I see VMware folders I created (like VeeamExclude) that I use to exclude the VMs contained within from the job.
How can Identify that these folders indicate EXCLUSION and not INCLUSION of the job?

[chris.arceneaux]

I'm asking about this internally. I'll follow-up soon.

[veremin]

Currently, this information regarding exclusions is not available via RESTful API. Thanks!

[pdrangeid]

I've got my script running in production now. Seems to be working pretty well. I'm not using this to troubleshoot veeam jobs (existing alerting works find for that) - this is more about validation that the correct VMs are indeed configured for backups, and have successfully completed according to a backup policy/decision that is assigned at the asset management level. This will support us in building workflows to identify VMs that are missing backups or not properly configured. If anyone is interested in building this in their own graph database, let me know and I will post the github link to the import scripts I'm using to accomplish this.

Thanks to the Veeam folks for their help!

[chris.arceneaux]

Good job @pdrangeid! I know I'd like the GitHub link.

Also, what tool are you using to generate the graph?

[pdrangeid]

I'm using Neo4j GraphDB (https://neo4j.com/), and a combination of Powershell and .cypher (the native neo4j graph query language) scripts.

Here's a how-to for setting up using the free community edition of Neo4j:
https://blog.graphcommit.com/2019/11/co ... e-are.html

If you also want to "relate" the protected VMs to "VMware assets" this is the process to import your vCenter infrastructure into a graph (which you can then use to tie to the Veeam data):
https://blog.graphcommit.com/2019/11/im ... -into.html

I'll need to add a little bit of documentation to the Veeam import scripts, and I'll post something back here soon.

[infraerik]

Just adding a vote to add the Exclusions object list into the results under <JobInfo>. I have some clients who default to backing up everything in a datacenter or a cluster, and then handle the odd ones by manually adding them to the exclusions list. Being able to include that in the reporting would be very useful.

[oleg.feoktistov]

Hi @infraerik. Thank you for your feedback! Marked it in my notes as a topic for further discussions.

[pdrangeid]

Ok - I finally got the scripts posted online. Here's the how-to:

https://blog.graphcommit.com/2019/11/cr ... h-via.html
And the git repo for those scripts: https://github.com/pdrangeid/veeam-maint

Here's the pre-requisite (this configures a CYPHER interpreter for powershell, and gives you a secure way to store the credentials in the registry for automation):
https://blog.graphcommit.com/2019/12/gr ... oject.html
and the git repo for those scripts: https://github.com/pdrangeid/graph-commit

And finally, the pre-pre-requisite (if you are COMPLETELY new to Neo4j graph databases) This will get you your N4J server setup and running with the plugins you will need to run the project above:
https://blog.graphcommit.com/2019/11/co ... e-are.html

[chris.arceneaux]

Thanks for sharing @pdrangeid!

[pwinkeler]

This business where backupTaskSessions only reference the vmDisplayName and not a full qualified identifier to locate the VM is becoming a major pain. We backup multiple vCenter datacenter with the same Veeam master and indeed of course have the same VM Name appear more than once across this collection. Internally these VM objects are tracked using their hierarchyObjectReferences I am sure. We really need that attribute added in the API call returns.

[jhoughes]

Querying the backupTaskSessions entity (/backupTaskSessions/{ID}?format=Entity) will return the full VM HeirarchyObjRefType value and enough information to perform this mapping.

The results returned will include the HeirarchyRootID as the second to last portion (this will map to your vCenter server), and the last portion being the MoRef ID from vCenter.

These are the results from a backTaskSession out of my lab, querying against API v1_5 (current v_latest):

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<BackupTaskSession xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.veeam.com/ent/v1.0" Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/backupTaskSessions/080b580b-584d-4efd-bae6-fc2a8227df55?format=Entity" Type="BackupTaskSession" Name="ausjump01@2020-05-16 09:20:38" UID="urn:veeam:BackupTaskSession:080b580b-584d-4efd-bae6-fc2a8227df55" VmDisplayName="ausjump01">
  <Links>
    <Link Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/backupServers/4eca212a-e998-47cb-b8ac-8d68f43b97d0" Name="ausveeambr.lab.fullstackgeek.net" Type="BackupServerReference" Rel="Up"/>
    <Link Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/backupSessions/a9c71f25-3d1c-4e9c-afdd-72454be4fd14" Name="AUSVCENTER-NoTag@2020-05-16 09:05:06" Type="BackupJobSessionReference" Rel="Up"/>
    <Link Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/backupTaskSessions/080b580b-584d-4efd-bae6-fc2a8227df55" Name="ausjump01@2020-05-16 09:20:38" Type="BackupTaskSessionReference" Rel="Alternate"/>
  </Links>
  <JobSessionUid>urn:veeam:BackupJobSession:a9c71f25-3d1c-4e9c-afdd-72454be4fd14</JobSessionUid>
  <CreationTimeUTC>2020-05-16T09:20:38.62Z</CreationTimeUTC>
  <EndTimeUTC>2020-05-16T09:23:41.943Z</EndTimeUTC>
  <State>Completed</State>
  <Result>Success</Result>
  <Reason/>
  <TotalSize>42949672960</TotalSize>
  <VmUid>urn:VMware:Vm:5000abb5-f13d-4434-a48d-bec0763f68f3.vm-451</VmUid>
</BackupTaskSession>

Relevant data:
<VmUid>urn:VMware:Vm:5000abb5-f13d-4434-a48d-bec0763f68f3.vm-451</VmUid>


This is the matching entity from my list of my HeirarchyRoots within Enterprise Manager:

Code: Select all

  <Ref UID="urn:veeam:HierarchyRoot:5000abb5-f13d-4434-a48d-bec0763f68f3" Name="ausvcenter.fsglab.local" Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/hierarchyRoots/5000abb5-f13d-4434-a48d-bec0763f68f3" Type="HierarchyRootReference">
    <Links>
      <Link Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/backupServers/4eca212a-e998-47cb-b8ac-8d68f43b97d0" Name="ausveeambr.lab.fullstackgeek.net" Type="BackupServerReference" Rel="Up"/>
      <Link Href="https://ausveeambem.lab.fullstackgeek.net:9398/api/hierarchyRoots/5000abb5-f13d-4434-a48d-bec0763f68f3?format=Entity" Name="ausvcenter.fsglab.local" Type="HierarchyRoot" Rel="Alternate"/>
    </Links>
  </Ref>

Relevant data:
5000abb5-f13d-4434-a48d-bec0763f68f3

This is the matching VM queried from Center:

Code: Select all

(Get-VM ausjump01).ExtensionData.MoRef

Type           Value
----           -----
VirtualMachine vm-451

Get-VM ausjump01 | Select-Object Name, @{n='MoRefID';e={$_.ExtensionData.MoRef.Value}}

Name      MoRefID
----      -------
ausjump01 vm-451

Relevant data:
vm-451


Hope this helps you out.

[pwinkeler]

Aha - that hint was enough! I am now pulling in the MoRefID of each VM and can further disambiguate using the hierarchy object reference to the vCenter in which it lives.