Powershell - Create S3 Compatible Repo and folders

[kavaa]

Hi,
We are in the works of scripting some of the clicks we have to do to create new repos for S3 compatible Storage.

We have found a way to create a new bucket in our TrueNAS server with S3 Powershell tools;

Code: Select all

Install-Module -Name AWSPowerShell -Force

Import-Module -Name AWSPowershell

Set-AWSCredential -AccessKey ACCESSKEYHERE -SecretKey SECRETKEYHERE -StoreAs default

New-S3Bucket -BucketName ictwebsolution -EndpointUrl https://URL:9000

So now we have an new bucket but than, we need to create the Repo in Veeam and Veeam needs to create "its" folder to store the data in.
But that's where the commands / guids end; https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=50

For example in their guide they speak about VBOAmazonS3Folder but others is example: VBOAmazonS3CompatibleConnectionSettings commands...

So we miss something I think, support is also checking it, but it takes to long.

Maybe someone already has a script and is willing to share.

So what we have after creating the bucket and run into issues now;

Code: Select all

$name = Read-Host -Prompt 'Input the name like S3-NAME'
$bucketname = Read-Host -Prompt 'Input the name for the folder in lowercase like: name'
$foldername = "veeam"

$account = Get-VBOAmazonS3CompatibleAccount -Id "S3AccountName"
$connection = New-VBOAmazonS3CompatibleConnectionSettings -Account $account -ServicePoint "ENDPOINTURL:9000"
$bucket = Get-VBOAmazonS3Bucket -AmazonS3CompatibleConnectionSettings $Connection -Name namehere
$folder = Add-VBOAmazonS3Folder -Bucket bucketnamehere -Name veeam

I've removed the $variables from the lines for testing.

As you can see the commands for getting the bucket are different than the first two.

Please if you have a solution share!

[Mildur]

Hi Kay

After you have created the folder, you must add the bucket as an object storage to vbo365. When this is done, you can create a vbo365 backup repository with a local cache path and your added object storage. See my sample script:

Code: Select all

### declare variables ###
$s3accountid = "Put in the guid for the s3 account"
$s3serviceendpoint = "Put in the S3 service endpoint url"
$bucketname = "Put in your bucket name"
$foldername = "Put in your folder name on the bucket"
$objectstoragename = "Put in the name for the object storage in Veeam"
$vbobackupreponame = "Put in the name for the VBO Backup repository"
$cachepath = "Put in The Path for the Cache Folder"
$proxyname = "Put in the vbo365 proxies hostname or IP address which you want to use to connect to the bucket"

### get the Connection data ###
$account = Get-VBOAmazonS3CompatibleAccount -Id $s3accountid
$connection = New-VBOAmazonS3CompatibleConnectionSettings -Account $account -ServicePoint $s3serviceendpoint

### Connect to the Bucket and create the Folder ###
$bucket = Get-VBOAmazonS3Bucket -AmazonS3CompatibleConnectionSettings $Connection -Name $bucketname
$folder = Add-VBOAmazonS3Folder -Bucket $bucket -Name $foldername

### Connect the created folder as an object storage backup repository to vbo365 ###
$objectstorage = Add-VBOAmazonS3CompatibleObjectStorageRepository -Folder $folder -Name $objectstoragename

### Create the VBO Backup repository with Cache disk and object storage. 1 Year Retention, Snapshot Based ###
$VboProxy = Get-VBOProxy -hostname $proxyname
Add-VBORepository -Proxy $VboProxy -Path $cachepath -Name $vbobackupreponame -ObjectStorageRepository $objectStorage -RetentionPeriod Year1 -RetentionType SnapshotBased -RetentionFrequencyType Daily -DailyTime 12:00:00 -DailyType Everyday

[kavaa]

@Mildur,

Thanks, but I get the same error.

Code: Select all

Add-VBORepository -Proxy $VboProxy -Path $cachepath -Name $vbobackupreponame -ObjectStorageRepository $objectStorage -RetentionPeriod Year1 -RetentionType SnapshotBased -RetentionFrequencyType Daily -DailyTime 12:00:00 -DailyType Everyday
Add-VBOAmazonS3Folder : Cannot validate argument on parameter 'Bucket'. The argument is null. Provide a valid value for the argument, and then try runni
ng the command again.
At line:17 char:41
+ $folder = Add-VBOAmazonS3Folder -Bucket $bucket -Name $foldername
+                                         ~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Add-VBOAmazonS3Folder], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Veeam.Archiver.PowerShell.Cmdlets.AmazonS3.AddVBOAmazonS3Folder
 
Add-VBOAmazonS3CompatibleObjectStorageRepository : Cannot bind argument to parameter 'Folder' because it is null.
At line:20 char:75
+ ... -VBOAmazonS3CompatibleObjectStorageRepository -Folder $folder -Name $ ...
+                                                           ~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Add-VBOAmazonS3...orageRepository], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Veeam.Archiver.PowerShell.Cmdlets.ObjectStorage.AddVBOAmazonS3CompatibleOb 
   jectStorageRepository
 
Add-VBORepository : Cannot validate argument on parameter 'ObjectStorageRepository'. The argument is null. Provide a valid value for the argument, and t
hen try running the command again.
At line:24 char:103
+ ... me $vbobackupreponame -ObjectStorageRepository $objectStorage -Retent ...
+                                                    ~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Add-VBORepository], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Veeam.Archiver.PowerShell.Cmdlets.Repository.AddVBORepositoryWrapper
 
 

I only put in the variables.

[kavaa]

Okay, when I create the bucket first it works.
I can also do that with the AWSPowerShell tools, like I posted in my first post.

So your script does not create the bucket for us right?

[Mildur]

No, my script doesn’t create the bucket.
The bucket must be created before you run my script. My script uses only vbo powershell commands. Veeam cannot create a bucket for you.

So you can combine your AWSPowerShell commands with my script to automate the creation of a vbo365 backup repository.

[kavaa]

Yes, i got that working now!
Thanks!

[Mike Resseler]

Thanks for sharing folks!

[Mildur]

Your welcome

[kavaa]

Thanks for sharing folks!

Maybe a good idea to update your Knowledgebase, i think there will be more people that have the same issue.

[MarcM]

Great topic! I was already planning on looking into VBO + Powershell a bit more, so thanks for getting me started

Yesterday I used your code snippets and added some functionality to it. The script needs to be executed like this:

Code: Select all

PS C:\Users\Administrator.VEEAM> Get-Help .\create_repo.ps1 -examples

NAME
    C:\Users\Administrator.VEEAM\create_repo.ps1

SYNOPSIS
    This is a Powershell script that will:
    - Automatically create a S3 bucket in Wasabi and a folder in that bucket
    - Add your Wasabi access key and secret key to VBM
    - Add the bucket as Object Storage Repository in VBM
    - And create a VBM Backup repository with Cache disk and object storage
    - With an 1 Year Retention period, Snapshot Based


    -------------------------- EXAMPLE 1 --------------------------

    PS C:\>.\create_repo.ps1 s3-access-key s3-secret-key bucketname foldername-in-bucket vbo-server-ip-hostname-or-fqdn proxy-name

    .\create_repo.ps1 WPDPN8DODU04KSZ3SJ2V uOmftyucFGvXc3yYg5fWaQosmeBSJtbApOXwNJ2h vbotest veeam vbo.veeam.lab vbo

PS C:\Users\Administrator.VEEAM>

There are a couple of things that need to be done before executing the script.

• 1) Make sure the AWSPowershell module is installed and imported
  • Install-Module -Name AWSPowerShell -Force
  • Import-Module -Name AWSPowershell
  
  2) Edit the "USER INPUT STARTS HERE" part in the script to specify the following:
  

  Code: Select all

  ### USER INPUT STARTS HERE ###
  
  ## Wasabi Endpoint and region
  $EndpointUrl = "https://s3.eu-central-1.wasabisys.com"
  $Region = "eu-central-1"
  
  ## Repository names and cache path
  $objectstoragename = "Automated Object Storage Repo 1"
  $vbobackupreponame = "Automated Backup Repo 1"
  $cachepath = "C:\RepoCache" 		## make sure the folder is created on the proxy
  
  ## Repository Retention Settings
  $RetentionPeriod = "Year1"		## Year1 | Years2 | Years3 | Years5 | Years7 | Years10 | Years25 | KeepForever
  $RetentionType = "SnapshotBased"	## ItemLevel | SnapshotBased
  $RetentionFrequencyType = "Daily"	## Daily | Monthly
  $DailyType = "Everyday"			## Everyday | Workdays | Weekends | Monday | Tuesday | Wednesday | Thursday | Friday | Saturday | Sunday
  $DailyTime = "02:00:00"			## Default: 00:00:00
  
  ### USER INPUT ENDS HERE ###
  

I am looking forward to any additions to the script. For now it's only working with Wasabi, I've tested it with MinIO but I got errors that it could not establish trust relationship for the SSL/TLS secure channel...it probably has something to do with the Self-Signed cert I am using for MinIO.

The script can be downloaded here (for now): create_wasabi_repo_v1.ps1


Update: And this version of the script works with AWS and can be downloaded here (for now): create_aws_repo_v1.ps1

For the AWS script, there are some other Region settings that need to be changed (you can leave the Wasabi ones in place, they are not used in this version of the script):

Code: Select all

## AWS Region Settings
$RegionType = "Global"
$Region2 = "eu-west-2"

Eventually I will see if I can merge the two together and maybe even add Azure Blob to it, but for now...have fun with it

[Mildur]

Hi Marc

Thanks for sharing and putting together a script from our sample code I will send a link to this topic to my colleagues.

[MarcM]

@Mildur Tonight I had a bit of time to finish the script the way I wanted it to be. The script now supports Wasabi, AWS and Azure Blob.

Run "PS C:\>Get-Help .\create_vbm_repo_v1.ps1 -examples" to see how the script works, what it does and what limitations it has.

You can download it here: create_vbm_repo_v1.ps1


Thanks again for sharing the code and giving me a quick start...I didn't know Powershell at all, but learned a lot

[FireExit]

@MarcM your script links no longer work, could you share them anywhere else, please?
(Or did anyone else get a copy?)

[MarcM]

@FireExit My apologies, I've changed some things on my server and forgot to update the link. Here you go: https://veeam.marcmolleman.nl/scripts/c ... epo_v1.ps1

[FireExit]

@MarcM - thank you very much!
That gets me past adding an S3 folder, but I hadn't noticed that VBO and VBR are different, so I've made a new thread for a question about Wasabi repositories ( powershell-f26/add-wasabi-repository-to ... 89305.html )