SureBackup & Exit Codes

[theadamlion]

Good afternoon all, I have been chasing an issue with a SureBackup PowerShell script that will always return 0 when run via the SureBackup job, but it will return the proper/expected exit code when run manually. I searched and saw a few other threads on this topic but none of them helped with a resolution unfortunately.

The actual code that I am running in the SureBackup job is as follows:

Code: Select all

Param($TestVmIP)
    $ReturnCode = 1
    $CredObject = Import-Clixml -Path C:\ps\CredObject.xml
    $ReturnCode = Invoke-Command -Credential $CredObject -Computername $TestVmIP -ErrorAction Stop -ScriptBlock{
  

# create an array of paths to search through
$folders = 'E:\Foldername\*'
# create an array of file names to look for
$files = 'budget2024.xlsx', 'exec-contacts.docx', 'pw.txt', 'payroll.docx'

Get-ChildItem -Path $folders -Include $files -File |
       Get-FileHash -Algorithm MD5 | 
       Export-Csv -Path 'C:\ps\newhashes.csv' -UseCulture -NoTypeInformation

$oldHashes = Import-Csv "C:\ps\hashes.csv"
$newHashes = Import-Csv "C:\ps\newhashes.csv"
$compare = Compare-Object $oldHashes $newHashes -Property Hash, Path 
	if ($compare.SideIndicator -eq "<=" -or $compare.SideIndicator -eq "=>"){
		$ReturnCode = 1
		Write-Host "$ReturnCode"
		Exit 1
	}
	else{
		$ReturnCode = 0
		Write-Host "$ReturnCode"
		Exit 0
	}
}
#exit $ReturnCode ## THIS WAS COMMENTED OUT TO TEST, NO CHANGES AS A RESULT

When run, this code will compute the MD5 hash of a few dummy files and compare them to a known, good list of MD5 hashes. If there are no differences the script should exit with a 0, and a 1 if there are any changes. This works perfectly when running this manually from the VBR server against the SB VM which is running in troubleshooting mode.

I am testing the manual run here by running these commands and calling the same PS script with a few lines commented out:

Code: Select all

$CredObject = Import-Clixml -Path C:\ps\CredObject.xml
Invoke-Command -ComputerName 172.30.253.3 -FilePath c:\ps\SB_manual.ps1 -credential $CredObject

The modified PS Script being called in the above command (commented out the first few lines so I could call the CredObject.xml and Invoke-Command cmdlet manually but otherwise identical to the above script:

Code: Select all

Param($TestVmIP)
    $ReturnCode = 1
    #$CredObject = Import-Clixml -Path C:\ps\CredObject.xml
	#$ReturnCode = Invoke-Command -Credential $CredObject -Computername $TestVmIP -ErrorAction Stop -ScriptBlock{
  

# create an array of paths to search through
$folders = 'E:\ShareData\*'
# create an array of file names to look for
$files = 'budget2024.xlsx', 'exec-contacts.docx', 'pw.txt', 'payroll.docx'

Get-ChildItem -Path $folders -Include $files -File |
       Get-FileHash -Algorithm MD5 | 
       Export-Csv -Path 'C:\ps\newhashes.csv' -UseCulture -NoTypeInformation

$oldHashes = Import-Csv "C:\ps\hashes.csv"
$newHashes = Import-Csv "C:\ps\newhashes.csv"
$compare = Compare-Object $oldHashes $newHashes -Property Hash, Path 
	if ($compare.SideIndicator -eq "<=" -or $compare.SideIndicator -eq "=>"){
		$ReturnCode = 1
		Write-Host "$ReturnCode"
		Exit 1		
	}
	else{
		$ReturnCode = 0
		Write-Host "$ReturnCode"
		Exit 0
	}
#}
#exit $ReturnCode

From the troubleshooting session, if I make the files identical resultings in matching hashes I am returned 0. If I change anything with one of those dummy files which does change the MD5 hash and subsequently updated the newhashes.csv file as expected, I am returned 1.

Regardless of that, running the script from the SB job itself will always return 0/success.

If anyone has any ideas or pointers, I would gladly welcome them as I cannot figure out what may be the cause here. Thanks all!

-Adam

[david.domask]

Hi Adam,

If I remember right, you're hitting this nuance of Powershell since Veeam Backup and Replication will call the script as a Command:

https://learn.microsoft.com/en-us/power ... stexitcode

Basically, the output will just be 0 or 1, but you can potentially manipulate this a bit by using $? and setting it's value to $false if you have a mismatch in your hash-check, and then exit after setting $? to $false.

Alternatively I think it may work if you add ; exit $LASTEXITCODE to the arguments line in the SureBackup job, but I haven't tested this; that however should make the execution look like:

yourscript.ps1; exit $LASTEXITCODE

But try one of those to start with and see if it helps with your goal.

[theadamlion]

Hi David, thanks for the reply and info on this one. I tested appending the "; exit $LASTEXITCODE" to the end of the arguments line in the SB job setup and got some syntax errors, so I shifted over to trying to manipulate and set the $? automatic variable and I had some promising testing.

I did notice that we're still on PowerShell 5.1 in this environment which doesn't have the ability to directly set the $? variable, but I read that calling the Write-Error cmdlet outside of a function will set $? to false after it is executed (from https://learn.microsoft.com/en-us/power ... #section-1)

"The Write-Error cmdlet always sets $? to False immediately after it's executed, but won't set $? to False for a function calling it:"

After some trial and error and manual testing, I added this line to my if statement that checks for mismatched hashes and exits with 1 if they're mismatched, and this seems to force $? to false:

Code: Select all

Write-Error "nope" | out-null

Manual testing and verification of the $? value all went as planned, I'm going to run the proper test in the morning and will report back with results.

As always David, thank you so much for your help and contributions on this board, have a great evening.

-Adam

[david.domask]

Hi Adam!

You've very welcome; thanks for testing on the exit $Lastexitcode part, regrettably I just did not have time today -- I know elsewhere in the product that trick works (e.g., for pre/post job scripts, you can add arguments to the script path and Veeam will run the script with the arguments), but maybe it's limited here.

Glad that $? is working though via Write-Error, look forward to hearing the results of the final test.

> As always David, thank you so much for your help and contributions on this board, have a great evening.

You're very welcome!

[theadamlion]

David, we are in business! I'm sure someone more skilled with PowerShell may know a better/cleaner way to do this but the Write-Error trick seems to have worked! I tested it both from manual runs and by running the SureBackup job and we are seeing the results we expect.

Thanks again man, have a lovely weekend!

-Adam

[david.domask]

Hello Adam!

Thank you very much for the confirmation and glad to hear we're in business!

I will ask that we please add some notes to the User Guide on this subject so it's a bit easier to know the expected behavior.

[Matts N]

I did notice that we're still on PowerShell 5.1 in this environment which doesn't have the ability to directly set the $? variable, but I read that calling the Write-Error cmdlet outside of a function will set $? to false after it is executed (from https://learn.microsoft.com/en-us/power ... #section-1)

Running PowerShell starting it with "ps", which is default for most applications, it will use standard version that is shipped with the OS which often matches 5.1. If you install a newer version of powershell try starting it with "pwsh" instead of "ps". That should hopefully start the newer version of PowerShell.

// Matts

[theadamlion]

Matt, that is a very helpful tip and thank you for the info. I really appreciate the assistance and the community at large here, thanks all!

-Adam

[david.domask]

Thank you Adam and also Matt for the testing and the commentaries.

We've updated the User Guide to reflect these behaviors: https://helpcenter.veeam.com/docs/backu ... t-settings

(See the note after the Test Script Settings section)