Veeam 12.3.1 cause automation failure

[donaldli]

Veeam Support Case #07660516

Upon upgrading to 12.3.1, we attempted to run a few tests, one of which involved deploying SOBRs that use IBM COS as S3Compatible capacity extents. We use Powershell APIs to configure these COS extents, specifically Add-VBRAmazonS3CompatibleRepository , which failed with an 'InvalidLocationConstraint' error like so:

Code: Select all

$cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $folder -Connection $connect -ProxyAppliance $helper -Name $vbr_name -ForceOwnershipChange -EnableBackupImmutability -ImmutabilityPeriod $immutability_period
Add-VBRAmazonS3CompatibleRepository : Unable to enable buckets auto-provisioning
S3 error: Container storage location with specified provisioning code not available
Code: InvalidLocationConstraint
Failed to create bucket
At line:1 char:13
+ $cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $fold ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-VBRAmazonS3CompatibleRepository], Exception
    + FullyQualifiedErrorId : System.Exception,Veeam.Backup.PowerShell.Cmdlets.AddVBRAmazonS3CompatibleRepository

This same command has worked for Veeam versions <12.3.1, and we confirmed it worked on a 12.3.0 Veeam install after we saw the failures. After a bit of searching, we reviewed the release notes and found this note:

Object Storage Enhancements

S3-Compatible Object Storage Compatibility Improvements:
The backup server will now ignore errors caused by S3-compatible object storage systems using unconventional storage class names. Additionally, automatic bucket creation is now the default for newly created S3-compatible object storage repositories.
so we went back to the Powershell API https://helpcenter.veeam.com/docs/backu ... ml?ver=120 , identified the EnableBucketAutoProvision parameter as likely being what is referenced in the release notes, and retried with that parameter explicitly set to 'false', which was then successful:

Code: Select all

 $cos_repo = Add-VBRAmazonS3CompatibleRepository -AmazonS3Folder $folder -Connection $connect -ProxyAppliance $helper -Name $vbr_name -ForceOwnershipChange -EnableBackupImmutability -ImmutabilityPeriod $immutability_period -EnableBucketAutoProvision:$false

As such, it seems very likely that this parameter was changed to default to 'true', and that this change broke our automation, so we'd like to understand more why this parameter was changed to default to 'true' / why this default setting seems to run into the InvalidLocationConstraint error

[david.domask]

Hi DonaldLi,

Interesting post, thank you very much for documenting it so clearly.

The message itself comes from IBM COS S3, but it's not clear for me if we're doing something wrong when trying to auto-provision buckets or when trying to address the bucket with the regions allowed.

Will review this internally and provided Support with some guidance, let's wait for the results.

For now, please continue with -EnableBucketAutoProvision:$false; this feature is meant for on-premises S3 deployments which typically are sensitive to load on individual buckets (high object count), and this feature mitigates that by spreading the load across multiple auto-generated buckets.

[Dagr]

Case #07749059
Can confirm that we have seen the same issue for a customer on version 12.3.1.1139, when trying to add IBM COS as S3Compatible repo (not in SOBR).

auto-generating buckets should not be a "mitigating" step. If teh issue is for on-premises S3 deployments, I would suggest either making a separate option for this, or add the option as a GUI- Check mark / possible parameter in PoSH.

[david.domask]

Hi Dagr,

The behavior itself is related to the way the bucket auto-creation feature works and the unique storage class names IBM uses, as well as how our current cmdlets work with the bucket auto-creation feature; in an upcoming release, there will be changes to the cmdlets to support IBM COS specifically via Powershell.

For now, when working with IBM COS via Powershell and the Add/Set-VBRAmazonS3CompactibleRepository, please use -EnableBucketAutoProvision:$false for the time being.