SOBR Data locality policy not applied

[b.vanhaastrecht]

We recently run rappidly out of SOBR repository space. One of four child reps who was running out of space, therefore chains were diverted to other reps. Backup data landed on other volumes, so ReFS could not do block cloning and backup size exploded.

We want to monitor if Veeam starts to place data outside the Data locality policy. This is an indication we need to re-balance/add more space. Are there any specific log entries which indicates locality policy could not be applied?

[Gostev]

Yes, I am quite positive this results in warning logged into the job sessions, because placement policy violation is sort of a big deal. I still remember the corresponding discussions back when we designed SOBR. Please check out the job's action log for the run when policy violation happened?

[b.vanhaastrecht]

Sorry for late reply, had my holiday in between.

I've parked a log of a backup job from I'm certain it did not follow it's preferred data location policy. This is confirmed with the synthetic full not using fast clone, but I'm unable to find some log text clarifying why data is diverted to an alternative repository. Would it be possible if you or an tech could help me out on this and take a look at the log? I can drop it somewhere and PM you the link.

[billcouper]

Hi b.vanhaastrecht,

I don't have any jobs where the SOBR placement policy has been violated, so I am not 100% certain of a few things in this script example.
#1) I assume the backup job ends up as 'Warning' status
#2) I assume that the job Session ends up as 'EWarning' status (based on failed jobs show as EFailed here)
#3) I assume that the job detail for the policy placement warning includes the word 'placement'

What the script does is:
a) Connect to Veeam server (can skip the Connect-VBRServer part if running script on VBR server itself - I always run script from remote)
b) Create an array with details of every primary backup job
c) Create an array with session details of all jobs that ended as 'warning'
d) For each job that ended in a warning, it finds the warning message from the session details, and if the warning includes the word 'placement' it adds the job name to an array
e) Sends an email with a list of jobs that appear to have placement policy warnings

As I said I don't have any jobs that had placement policy violation, or even any job with a warning, to really test this code... but it could give you a hint to a solution for reporting, just run the script on a schedule once it works. And of course, this is a very rudimentary code that ideally would have a lot of error checking

Code: Select all

Add-PSSnapin veeampssnapin
$Creds = get-credential
Connect-VBRServer -Server VBRSERVERNAME -Credential $Creds
$VBRJobs = Get-VBRJob | ? { $_.IsBackup }
$Sessions = Get-VBRBackupSession | ? { $_.result -eq "Warning" }

$WarningJobs = @()
foreach ($job in $VBRJobs) {
    if ($job.GetLastResult() -eq "Warning") {
        $WarningJobs += $job
    }
}

$PlacementWarnings = @()
foreach ($job in $WarningJobs) {
    $jobSession = $Sessions | ? { $_.JobName -imatch $job.name } | sort CreationTimeUTC | select -last 1
    [array]$jobWarnings = $jobSession.Logger.GetLog().UpdatedRecords | ? { $_.Status -eq "EWarning" } | select Title
    if ($jobWarnings -imatch "placement") {
        $PlacementWarnings += $job.Name
    }
}

if ($PlacementWarnings.Count -ge 1) {
    $MessageSubject = "Placement Policy Violations"
    Send-MailMessage -SmtpServer mail.server.com -From veeam@server.com -To admin@server.com -Subject $MessageSubject -Body $($PlacementWarnings -join ", ")
}

[b.vanhaastrecht]

Hi Bill, thanks for this.

I've scoped the particular job and fetched the '.Logger.GetLog().UpdatedRecords' . Unfortunately there's no entry from SOBR placement policy.

In the UI, there's no warning what so ever regarding this.

[billcouper]

If you look through the History tab in the GUI for instances of this job that had a 'Warning' result you should be able to find it somewhere?
The code I wrote only looks at the most recent backup session. It may (or may not) pick it up if/when it happens again. Perhaps someone from Veeam could comment on the code?

edit: If you remove this text "| select -last 1" it will look for the warning in every backup session, not just the most recent one, and could help to identify the warning string correctly (if it's even there at all)

[b.vanhaastrecht]

No, there's no message. Not in GUI, not in the session. SOBR locality policy errors/warnings are not logged there. That was the reason I created this post in the first place

edit: I used parts of your script, but like I wrote I scoped the particular job and went thru all its logger lines.

[billcouper]

The code was just an example and I'm glad it was useful at all. That's a bummer that the warning can't be found. The session that had to span to a new extent should have definitely mentioned it in the GUI's simplified job log, and therefore should also show in the session 'Logger' records of the job. Is your GUI set to show all history? Depending how many backups jobs you have (I have 127 jobs) the default settings won't go back very far and you might not be able to see the session history any more. If you increase the settings in the GUI you can see further back in the history view. But if you can't find it in the GUI then it sounds like a bug.

I would suggest automated monitoring/alerting on the extent capacity usage to avoid similar issues in future

Using ReFS myself, I can say you need a way to infinitely expand extents. Because my repository servers are virtualized and there are limitations with the underlying flash storage maximum LUN sizes, I've resorted to creating 'Dynamic Disks' in disk management (inside the repo vm's) to join multiple VMDK's into a single volume if I get close to the LUN size limit on any extent. Also, I mount my ReFS extent volumes to directories (mount points) because using drive letters would only allow a maximum of 25 extents on each repo server. I have my ReFS extent volumes mounted to C:\Mount\SSDxx and of course C: is an NTFS volume - Veeam correctly detects the mounted volume as ReFS and it's all happy days. Lots of smaller extents (to spread out the VM chains) can work better than fewer large extents, but it depends on how many VM's and how large they are.

edit: if you haven't already, collect the job log for the day(s) that it spilled over and give it to support.

[billcouper]

b.vanhaastrecht I have spent some time looking at the powershell method to collect data placement policy warnings, and I have some revised code for you to try.
Again this code looks at the most recent job session, so you may have to scope it while testing.

edit: The timing couldn't have been better, one of our customers that runs Veeam on premise had a data placement policy breach last night, somebody screwed up monitoring the extent sizes... which meant I was able to test this code and confirm it definitely captures the placement policy warnings!

Code: Select all

# Load the Veeam PS Snapin
Add-PSSnapin veeampssnapin

# Collect VBR Job and Session info
$VBRJobs = Get-VBRJob | ? { $_.IsBackup }
$Sessions = Get-VBRBackupSession | ? { $_.result -eq "Warning" }

# Create an array of jobs that finished with a Warning
$WarningJobs = @()
foreach ($job in $VBRJobs) {
    if ($job.GetLastResult() -eq "Warning") {
        $WarningJobs += $job
    }
}

# Create an array to hold a list of VM's that were unable to meet data placement policy
$PlacementWarnings = @()
foreach ($job in $WarningJobs) {

    # Get the most recent job session for the job
    $jobSession = $Sessions | ? { $_.JobName -imatch $job.name } | sort CreationTimeUTC | select -last 1

    # Get a list of all the warnings that this job experienced
    $ThisReport = [Veeam.Backup.core.CBackupTaskSession]::GetByJobSession($jobsession.id) | ? { $_.Status -eq "Warning" }

    # If any of the warnings include the word placement
    if ($ThisReport.Reason -imatch "placement") {
        
        # Loop through each of the placement warnings
        $ThisReport | ? { $_.Reason -imatch "placement" } | % {
            
            # Create an object with details of the placement warning
            $objWarning = [pscustomobject]@{
                Job = $job.Name
                Date = $jobsession.CreationTime
                Object = $_.ObjectName
                Warning = $_.Reason
            }
            
            # Add the object to the array
            $PlacementWarnings += $objWarning
        }
    }
}