Custom script SureBackup on a Veeam B&R non-domain joined server

[drop_table]

Hi,

Currenlty i'm wrestling with a issue i'm facing when i try to use custom scripts to do some tests on a server during a SureBackup job.

The situation:

I followed the best practice guide from Veeam. One of the practices is to not domain join the Veeam B&R Server. So that is the case, my VEEAM B&R server is not domain joined.
What i want to do is to some more checks on a VM that is in a SureBackup application group. I found the knowledge base article to do some checks on particular services (https://www.veeam.com/blog/automate-rec ... ackup.html).

Now during some testing, i noticed that this wouldn't work, because my server is not domain joined, and trying to powershell to a domain joined machine and trying to use domain credentials simply just would'nt compute on my veeam server. So i tried it via another direction; WinRM over HTTPS.

So i made sure that WinRM over HTTPS was correctly setup on my destination machine (that i would script against during the surebackup job). Did some tests with my script from a test machine, to make sure everything would work and so fort.

Looking at my script, my eye couldn't let go the fact that my password whas in plain text just sitting there. To resolve this, I encrypted it.

Code: Select all

<# Set and encrypt credentials to file using default method #>

$credential = Get-Credential
$credentialfile = Read-Host -Prompt 'Voer hier de gewenste bestandsnaam in'

$credential.Password | ConvertFrom-SecureString | Set-Content "E:\Scripts\Credentials\$($credentialfile)_encrypted_password.txt"

But here is the thing. The user encrypting the password, has the key to un-encrypt it; so i made sure that the user that is running the script (svc_veeam_scripts , the local user account specifically made on the veeam server to run the scripts) encrypted the credential and putted it in the file.

So after doing that, i ran PowerShell ISE as another user (svc_veeam_scripts) and tested the script and it worked. Test it again with my admin account, and as expected it failed (due to the fact that the encrypted credential could'nt be decrypted with my admin account). So far so good.

But now here's my problem. During the Surebackup job, it failed again. And it looks like that the user running the script couldn't decrypt the credential But why?

• - I created the user on the Veeam server locally
  - I created the user in Veeam itself with the same credentials
  - I made sure that the credentials were encrypted with the same user
  - I tested the script with the same user; no errors at all.

LOGS DELETED

And here is my script:

Code: Select all

param(
[string] $ip #IP address van server
)


$username = "DOMAIN\user"
$secpasswdfile = "E:\Scripts\Credentials\svc_domainuser_encrypted_password.txt"

$secpasswd = Get-Content $secpasswdfile | ConvertTo-SecureString
$credentials = New-Object System.Management.Automation.PSCredential ($username, $secpasswd)
$soptions = New-PSSessionOption -SkipCACheck -SkipRevocationCheck -SkipCNCheck 

# Start services
Invoke-Command -ComputerName $ip -UseSSL -SessionOption $soptions -Credential $credentials -ScriptBlock `
{
Start-Service -InputObject (Get-Service -Name IAS)
#Write-Output (Get-Service -Name IAS)
#Start-Sleep -s 60

# Check services status

$checkservice = (get-service -ComputerName $ip -Name IAS -ErrorAction SilentlyContinue)
if($checkservice.status -ne "Running"){$host.SetShouldExit(1)}

exit
}
#En de batch file waar SureBackup draait:

#powershell.exe -noninteractive -noprofile -command "& {E:\Scripts\Check-Services_XXX.ps1 %1 }"
#EXIT /B %errorlevel%

I searched the forum and came across a thread (powershell-f26/surebackup-custom-powers ... 63179.html) that Better Credentials module would somehow work beter? But the thread was inconclusive about this.

Hope that anyone can help me out with this one.

Kind regards,
David

[drop_table]

Case #05223286 — Custom script SureBackup on a Veeam B&R non-domain joined server

[oleg.feoktistov]

Hi David,

I'll take some time to analyze your scenario and support case. Please refrain, however, from inserting logs into your posts - it is against our forum rules. Besides, we are able to see your logs in a case given that you shared them there.

Thanks,
Oleg

[drop_table]

Ah sorry Oleg about that. Will remember it for the next time.

[drop_table]

Well, with help from StackOverflow.com, i got the script working now withing a SureBackup job.

The only thing with this, is that i have to use the password in PLAIN TEXT (!!!!), i hope that Veeam can figure this out for us, because this is one humungous red flag ofcourse.

This is the BAT file that i'm using:

Code: Select all

powershell.exe -noprofile -File "E:\Scripts\Check-Service.ps1" "%1" "%2"
EXIT /B %errorlevel%

And this is the PowerShell script:

Code: Select all

param(
[string] $ip,       #IP address of checked server
[string] $service ) #Service name

$username = "DOMAIN\Domainuser"
#$secpasswdfile = "E:\Scripts\Credentials\domainuser_encrypted_password.txt"  # Doesn't work, just commented out until Veeam fixes this.

#$secpasswd = Get-Content $secpasswdfile | ConvertTo-SecureString   # Doesn't work, just commented out until Veeam fixes this.
 $secpasswd = ConvertTo-SecureString "Password123" -AsPlainText -Force   #WARNING, BIG RED FLAG! BEWARE OF THIS!
$credentials = New-Object System.Management.Automation.PSCredential ($username, $secpasswd)
$soptions = New-PSSessionOption -SkipCACheck -SkipRevocationCheck -SkipCNCheck
$session = New-PSSession -ComputerName $ip -UseSSL -SessionOption $soptions -Credential $credentials

# Start services
Invoke-Command -Session $session -ScriptBlock { Start-Service -Name $using:service }

# Check services status
$checkservice = Invoke-Command -Session $session { Get-Service -name $using:service | where status -eq running }
if (! $checkservice) { 
  write-output ("Error 1, Service '" + $service + "' not running or not found.")
  exit 1
}

In the article https://www.veeam.com/blog/automate-rec ... ackup.html you will find some differences.

Difference with BAT file
Change your invocation of powershell.exe to use the -File CLI parameter:

Code: Select all

powershell.exe -NoProfile -File "E:\Scripts\Check-Services_XXX.ps1" "%1"
EXIT /B %errorlevel%

That way, the .ps1 script's exit code is properly relayed as powershell.exe's exit code.

Why I used another Exit way

To use $host.SetShouldExit(1) call out of the Invoke-Command script block, given that the latter executes remotely. For the reasons explained below, exit 1 is preferable.

Generally speaking:

There's no reason to use the -Command (-c) CLI parameter with "& { ... }" in order to invoke code - just use "..." directly. Older versions of the CLI documentation erroneously suggested that & { ... } is required, but this has since been corrected.

Not only is "& { ... }" unnecessary, it invariably resets the exit code to 0.

As for your use of $host.SetShouldExit(1) to request exiting with an exit code of 1 (leaving aside that in a remote call it isn't effective):

[drop_table]

*******SOLUTION**********

Ok, the last problem is now also solved. I've got it all working now.

Now my non-domain joined Veeam Backup and Replication server, within a SureBackup job, can connect to a domain joined machine with credentials from the domain.
Which are encrypted under the service account that Veeam backup & replication service is running on.

So it's completely safe now, at least as much as possible; it would be great if Veeam made its own solution for this within the application. Because I'm just following the best practice instruction not to put the Veeam server in the domain... anyway.



If your Veeam Backup service is still running under the Local system account; then do the following:

• Create a new local account on your Veeam server.

• Make sure this account is member of the local administrators group

• Make sure this account gets rights to Log on as a Service.

• Then adjust the credentials of Veeam Backup service to this account and restart the service (make sure nothing is running at that moment of course)

Then open PowerShell ISE 'as another user', and then enter the credentials of the new local account you just created (AND NOT SOMETHING ELSE

Then run the following script, and adjust the parameters as needed:
MAKE SURE THAT WHEN THE CREDENTIAL PROMPT COMES, YOU FILL IN THE DOMAIN CREDENTIALS THAT HAS RIGHTS ON YOUR DOMAIN MACHINE

Code: Select all

<# Set and encrypt credentials to file using default method #>

$credential = Get-Credential
$credentialfile = Read-Host -Prompt 'Enter the desired file name here'

$credential.Password | ConvertFrom-SecureString | Set-Content "E:\Scripts\Credentials\$($credentialfile)_encrypted_password.txt"

Then open the script I last posted above and uncomment the following parts:

Code: Select all

$secpasswdfile = "E:\Scripts\Credentials\DOMAINUSER_encrypted_password.txt"
$secpasswd = Get-Content $secpasswdfile | ConvertTo-SecureString

Also remove the following line:

Code: Select all

#$secpasswd = ConvertTo-SecureString "Password123" -AsPlainText -Force

Your Check-Service.ps1 script should look like this:

Code: Select all

param(
[string] $ip, #IP address of checked server
[string] $service ) #Service name

$username = "DOMAIN\Domainuser"
$secpasswdfile = "E:\Scripts\Credentials\domainuser_encrypted_password.txt"

$secpasswd = Get-Content $secpasswdfile | ConvertTo-SecureString
$credentials = New-Object System.Management.Automation.PSCredential ($username, $secpasswd)
$soptions = New-PSSessionOption -SkipCACheck -SkipRevocationCheck -SkipCNCheck
$session = New-PSSession -ComputerName $ip -UseSSL -SessionOption $soptions -Credential $credentials

# Start services
Invoke-Command -Session $session -ScriptBlock { Start-Service -Name $using:service }

# Check service status
$checkservice = Invoke-Command -Session $session { Get-Service -name $using:service | where status -eq running }
if (! $checkservice) {
  write-output ("Error 1, Service '" + $service + "' not running or not found.")
  exit 1
}

#And the batch file where SureBackup is running:

#powershell.exe -noprofile -File "E:\Scripts\Check-Services_XXX.ps1" "%1" "%2"
#EXIT /B %errorlevel%

Cheers!
David

[oleg.feoktistov]

Hi David,

Sorry for such a long run with this scenario. We have finally had time to check it with QA and came to the conclusion that these difficulties you encountered are because if you use custom scripts to test out remote VMs, credentials indicated in this tab are not recognized, so you need to use invoke cmdlets in your script and configure creds along with encryption/descryption manually.
I noted a feature request to allow using credentials tab I mentioned above for scenarios like yours so that there is no need to hardcode it.

Thanks,
Oleg

[mike.anderson]

I recently ran into the exact same issues. The initial reason it didn't work was because most of the ways of exporting keys in Windows Powershell uses DPAPI which is only useful on the machine the credentials were generated on.

In my testing I didn't need to change the service account the Veeam service ran under, but I did need to exports the credentials to AES to have them work.

Not sure of the rules on external linking here, but this tutorial covers it extremely well - https://www.pdq.com/blog/secure-passwor ... ls-part-2/

I would like to add to the feature request also, as being able to implement the credentials from the credentials tab would have potentially greatly simplified the task.