Scaleway S3 Object Storage

[NightBird]

Hello all,
FYI
I have tried french cloud provider Scaleway S3 Object Storage with immutability.
It Works like a charm.

https://www.scaleway.com/en/object-storage/

[Gostev]

Thanks for reporting, I've added them to the sticky topic!

[NightBird]

Veeam 9.5 without immutability
https://www.scaleway.com/en/docs/how-to ... -scaleway/

S3 Object Storage API supported operations
https://www.scaleway.com/en/docs/s3-object-storage-api/

Why use GAFAM cloud providers when sovereign cloud provider could be used

[SchaeferTWS]

Hello all,
FYI
I have tried french cloud provider Scaleway S3 Object Storage with immutability.
It Works like a charm.

https://www.scaleway.com/en/object-storage/

Hey NightBird,

could you please give me a little hint how to get immutability working on Scaleway S3?
I tried to create a bucket by using --object-lock-enabled-for-bucket and now there is a lock enabled but Veeam does not like it.

10.05.2021 12:23:43 :: REST API error: 'S3 error: COMPLIANCE mode, override denied
Code: AccessDenied', error code: 403
Other:

If I create a bucket without the option Immutabiliy does not work at all. So somehow I am missing an important point?

Thanks!

[NightBird]

Do you have encryption enabled ? try to disable it, create a new bucket with object-lock => it should work.

[veremin]

In order to use immutability you need to enable the Object Lock and Versioning features on your bucket at the time it is created and then add it as object storage repository to backup server. The reversed order will not work (adding repository first and enabling object lock second). Thanks!

[SchaeferTWS]

Do you have encryption enabled ? try to disable it, create a new bucket with object-lock => it should work.

Thanks, seems to work.
But I don't like the idea that the files are stored unencrypted.

I tried to enable Encryption after the first initial upload and it seems to work. Hmmm...

[veremin]

I'm interested what kind of encryption is meant here. Encryption enabled in the Capacity Tier settings or the one enabled in the job settings? Thanks!

[NightBird]

Thanks, seems to work.
But I don't like the idea that the files are stored unencrypted.

I tried to enable Encryption after the first initial upload and it seems to work. Hmmm...

Oh interesting, you're talking about encryption on capacity tier side ?

I think there is a bug at veeam side
When you enable encryption on capicity tier it doesn't work :'(

[Gostev]

I highly doubt such a bug would remain unreported for over 2 years and 500PB of data moved into public clouds... I think storage-specific issue is much more likely here.

[NightBird]

How does work encryption on capacity tier (not on job side) ? some specific S3 API call ?

[Gostev]

No, we encrypt each object ourselves before sending them to object storage.

[Mildur]

S3 data transfer is always encrypted on https.
The veeam capacity encryption will encrypt the metadata (and blocks?) in the S3 Storage.

If you want to add the bucket to a fresh VBR Server, you will need access data for the bucket and the encryption password from veeam to read the metadata and restore your data.

[NightBird]

No, we encrypt each object ourselves before sending them to object storage.

Really strange this error, immutability work like a charm, everything work fine, but when you enable encryption on capacity tier... error...
This is why I asked if there is a specific S3 call.

My customer opened a support case #04757486

[veremin]

We will check the case internally to see what might be causing the error. Thank you for sharing the case number.

[SchaeferTWS]

@NightBird @veremin
If you create an S3 immutable storage (on Scaleway) and start with an unencrypted job you are able to sync.
After the first initial backup and scale-out processing you can enable the encryption of the backup files in the job details. Now it works. For every new job you have to do this again.

So...

• Create backup job without encryption
• Do first full backup and SOBR-Tiering
• Enable encryption in the backup job
• Works

For me it looks like the main encryption file for an job will be overwritten or something like this when you create an encrypted job. And this won't work due to immutability.

As I tried the same with MinIO this looks definitly like an issue with Scaleway here - but it does not make sense at all, I would say.

So hopefully Veeam can check.

[veremin]

My customer opened a support case #04757486

We have talked with your support engineer, it seems that we have a guess about potential cause and we will verify it during webex call next week.

So hopefully Veeam can check.

Thank you for sharing additional details; much appreciated.

[veremin]

The case has been escalated to R&D team. I will keep you posted on the results of our findings.

[veremin]

Turned out some S3 object retention API is not implemented correctly by storage vendor, so, you should probably open a case with them to confirm. Our support engineer will provide you with more information about that. Thanks!

[fouc]

Hi,
I also do tests and I also have problems.
I am interested in the following events to know the exact procedure for creating the bucket to be able to use them with VEEAM with the OBJECTLOCK option
Thanks

[SchaeferTWS]

Hi @fouc,

unfortunately Scaleway does not support the correct API implementation for using Veeam with objectlock.
Switch to Wasabi instead - it works nearly out of the box there! More information: https://wasabi-support.zendesk.com/hc/e ... ntegration

Be sure to modify the permissions of Wasabi like here:
post416181.html#p416181

Cheers,
Aaron

[veremin]

I am interested in the following events to know the exact procedure for creating the bucket to be able to use them with VEEAM with the OBJECTLOCK option

Currently you cannot archive this due to certain API implementation specifics. Opening a ticket with a vendor might raise additional awareness about this issue. Thanks!

[NightBird]

Scaleway is aware of the problem and will solve it in the coming weeks.

Reply from scaleway to my customer.

[flibouille]

It's a great ad, really! Except that it's not true. Everybody knows Scaleway and most of cloud compagnies works with VEEAM, because everybody is S3 compatible.

But when you use Veeam S3 storage, you often need immutability, and it's unfortunately doesn't work with Scaleway.

MODE: COMPLIANCE
veeam unable to use backup immutability the compliance mode is not supported

MODE:GOVERNANCE
veeam unable to use backup immutability the governance mode is not supported

[veremin]

According to NightBird's reply, they are aware of the issue that prevents customers from using immutability with their storage system. We've requested an update regarding this case internally, I will share more information, once I have it. Thanks!

[veremin]

It seems that currently the issue persists - the corresponding API that allows us to prolongate object lock on the selected entities does not work correctly. Hopefully, it will be fixed soon. Thanks!

[Eric from Paris]

Hello,
We plan to use Scaleway for our backup. Does anyone know if the bug indicated in this post is fixed or not ?
Thanks.