Veeam 12.3.1 cause automation failure

[donaldli]

Veeam Support Case #07660516

Upon upgrading to 12.3.1, we attempted to run a few tests, one of which involved deploying SOBRs that use IBM COS as S3Compatible capacity extents. We use Powershell APIs to configure these COS extents, specifically Add-VBRAmazonS3CompatibleRepository , which failed with an 'InvalidLocationConstraint' error like so:

Code: Select all

$cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $folder -Connection $connect -ProxyAppliance $helper -Name $vbr_name -ForceOwnershipChange -EnableBackupImmutability -ImmutabilityPeriod $immutability_period
Add-VBRAmazonS3CompatibleRepository : Unable to enable buckets auto-provisioning
S3 error: Container storage location with specified provisioning code not available
Code: InvalidLocationConstraint
Failed to create bucket
At line:1 char:13
+ $cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $fold ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-VBRAmazonS3CompatibleRepository], Exception
    + FullyQualifiedErrorId : System.Exception,Veeam.Backup.PowerShell.Cmdlets.AddVBRAmazonS3CompatibleRepository

This same command has worked for Veeam versions <12.3.1, and we confirmed it worked on a 12.3.0 Veeam install after we saw the failures. After a bit of searching, we reviewed the release notes and found this note:

Object Storage Enhancements

S3-Compatible Object Storage Compatibility Improvements:
The backup server will now ignore errors caused by S3-compatible object storage systems using unconventional storage class names. Additionally, automatic bucket creation is now the default for newly created S3-compatible object storage repositories.
so we went back to the Powershell API https://helpcenter.veeam.com/docs/backu ... ml?ver=120 , identified the EnableBucketAutoProvision parameter as likely being what is referenced in the release notes, and retried with that parameter explicitly set to 'false', which was then successful:

Code: Select all

 $cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $folder -Connection $connect -ProxyAppliance $helper -Name $vbr_name -ForceOwnershipChange -EnableBackupImmutability -ImmutabilityPeriod $immutability_period -EnableBucketAutoProvision:$false

As such, it seems very likely that this parameter was changed to default to 'true', and that this change broke our automation, so we'd like to understand more why this parameter was changed to default to 'true' / why this default setting seems to run into the InvalidLocationConstraint error

[david.domask]

Hi DonaldLi,

Interesting post, thank you very much for documenting it so clearly.

The message itself comes from IBM COS S3, but it's not clear for me if we're doing something wrong when trying to auto-provision buckets or when trying to address the bucket with the regions allowed.

Will review this internally and provided Support with some guidance, let's wait for the results.

For now, please continue with -EnableBucketAutoProvision:$false; this feature is meant for on-premises S3 deployments which typically are sensitive to load on individual buckets (high object count), and this feature mitigates that by spreading the load across multiple auto-generated buckets.