PowerShell script exchange
Post Reply
donaldli
Service Provider
Posts: 24
Liked: 8 times
Joined: Mar 17, 2021 3:22 am
Full Name: Donald Li
Contact:

Veeam 12.3.1 cause automation failure

Post by donaldli » 1 person likes this post

Veeam Support Case #07660516

Upon upgrading to 12.3.1, we attempted to run a few tests, one of which involved deploying SOBRs that use IBM COS as S3Compatible capacity extents. We use Powershell APIs to configure these COS extents, specifically Add-VBRAmazonS3CompatibleRepository , which failed with an 'InvalidLocationConstraint' error like so:

Code: Select all

$cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $folder -Connection $connect -ProxyAppliance $helper -Name $vbr_name -ForceOwnershipChange -EnableBackupImmutability -ImmutabilityPeriod $immutability_period
Add-VBRAmazonS3CompatibleRepository : Unable to enable buckets auto-provisioning
S3 error: Container storage location with specified provisioning code not available
Code: InvalidLocationConstraint
Failed to create bucket
At line:1 char:13
+ $cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $fold ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-VBRAmazonS3CompatibleRepository], Exception
    + FullyQualifiedErrorId : System.Exception,Veeam.Backup.PowerShell.Cmdlets.AddVBRAmazonS3CompatibleRepository
This same command has worked for Veeam versions <12.3.1, and we confirmed it worked on a 12.3.0 Veeam install after we saw the failures. After a bit of searching, we reviewed the release notes and found this note:

Object Storage Enhancements

S3-Compatible Object Storage Compatibility Improvements:
The backup server will now ignore errors caused by S3-compatible object storage systems using unconventional storage class names. Additionally, automatic bucket creation is now the default for newly created S3-compatible object storage repositories.
so we went back to the Powershell API https://helpcenter.veeam.com/docs/backu ... ml?ver=120 , identified the EnableBucketAutoProvision parameter as likely being what is referenced in the release notes, and retried with that parameter explicitly set to 'false', which was then successful:

Code: Select all

 $cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $folder -Connection $connect -ProxyAppliance $helper -Name $vbr_name -ForceOwnershipChange -EnableBackupImmutability -ImmutabilityPeriod $immutability_period -EnableBucketAutoProvision:$false
As such, it seems very likely that this parameter was changed to default to 'true', and that this change broke our automation, so we'd like to understand more why this parameter was changed to default to 'true' / why this default setting seems to run into the InvalidLocationConstraint error
david.domask
Veeam Software
Posts: 2833
Liked: 647 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Veeam 12.3.1 cause automation failure

Post by david.domask »

Hi DonaldLi,

Interesting post, thank you very much for documenting it so clearly.

The message itself comes from IBM COS S3, but it's not clear for me if we're doing something wrong when trying to auto-provision buckets or when trying to address the bucket with the regions allowed.

Will review this internally and provided Support with some guidance, let's wait for the results.

For now, please continue with -EnableBucketAutoProvision:$false; this feature is meant for on-premises S3 deployments which typically are sensitive to load on individual buckets (high object count), and this feature mitigates that by spreading the load across multiple auto-generated buckets.
David Domask | Product Management: Principal Analyst
Dagr
Service Provider
Posts: 41
Liked: 5 times
Joined: Nov 30, 2020 12:48 pm
Full Name: Daniel Gram Knudsen
Contact:

Re: Veeam 12.3.1 cause automation failure

Post by Dagr »

Case #07749059
Can confirm that we have seen the same issue for a customer on version 12.3.1.1139, when trying to add IBM COS as S3Compatible repo (not in SOBR).

auto-generating buckets should not be a "mitigating" step. If teh issue is for on-premises S3 deployments, I would suggest either making a separate option for this, or add the option as a GUI- Check mark / possible parameter in PoSH.
david.domask
Veeam Software
Posts: 2833
Liked: 647 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Veeam 12.3.1 cause automation failure

Post by david.domask » 1 person likes this post

Hi Dagr,

The behavior itself is related to the way the bucket auto-creation feature works and the unique storage class names IBM uses, as well as how our current cmdlets work with the bucket auto-creation feature; in an upcoming release, there will be changes to the cmdlets to support IBM COS specifically via Powershell.

For now, when working with IBM COS via Powershell and the Add/Set-VBRAmazonS3CompactibleRepository, please use -EnableBucketAutoProvision:$false for the time being.
David Domask | Product Management: Principal Analyst
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests