Using object storage as a backup target
Post Reply
NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Scaleway S3 Object Storage

Post by NightBird » 1 person likes this post

Hello all,
FYI
I have tried french cloud provider Scaleway S3 Object Storage with immutability.
It Works like a charm.

https://www.scaleway.com/en/object-storage/

Gostev
SVP, Product Management
Posts: 29104
Liked: 5358 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scaleway S3 Object Storage

Post by Gostev » 1 person likes this post

Thanks for reporting, I've added them to the sticky topic!

NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Scaleway S3 Object Storage

Post by NightBird » 1 person likes this post

Veeam 9.5 without immutability
https://www.scaleway.com/en/docs/how-to ... -scaleway/

S3 Object Storage API supported operations
https://www.scaleway.com/en/docs/s3-object-storage-api/

Why use GAFAM cloud providers :evil: when sovereign cloud provider could be used 8)

SchaeferTWS
Novice
Posts: 5
Liked: 2 times
Joined: May 10, 2021 10:35 am
Full Name: Sysadmin of the day
Location: Germany
Contact:

Re: Scaleway S3 Object Storage

Post by SchaeferTWS »

NightBird wrote: Apr 16, 2021 10:06 am Hello all,
FYI
I have tried french cloud provider Scaleway S3 Object Storage with immutability.
It Works like a charm.

https://www.scaleway.com/en/object-storage/
Hey NightBird,

could you please give me a little hint how to get immutability working on Scaleway S3?
I tried to create a bucket by using --object-lock-enabled-for-bucket and now there is a lock enabled but Veeam does not like it.

10.05.2021 12:23:43 :: REST API error: 'S3 error: COMPLIANCE mode, override denied
Code: AccessDenied', error code: 403
Other:


If I create a bucket without the option Immutabiliy does not work at all. So somehow I am missing an important point?

Thanks!

NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Scaleway S3 Object Storage

Post by NightBird » 1 person likes this post

Do you have encryption enabled ? try to disable it, create a new bucket with object-lock => it should work.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

In order to use immutability you need to enable the Object Lock and Versioning features on your bucket at the time it is created and then add it as object storage repository to backup server. The reversed order will not work (adding repository first and enabling object lock second). Thanks!

SchaeferTWS
Novice
Posts: 5
Liked: 2 times
Joined: May 10, 2021 10:35 am
Full Name: Sysadmin of the day
Location: Germany
Contact:

Re: Scaleway S3 Object Storage

Post by SchaeferTWS »

NightBird wrote: May 10, 2021 12:28 pm Do you have encryption enabled ? try to disable it, create a new bucket with object-lock => it should work.
Thanks, seems to work.
But I don't like the idea that the files are stored unencrypted. :?

I tried to enable Encryption after the first initial upload and it seems to work. Hmmm...

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

I'm interested what kind of encryption is meant here. Encryption enabled in the Capacity Tier settings or the one enabled in the job settings? Thanks!

NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Scaleway S3 Object Storage

Post by NightBird »

SchaeferTWS wrote: May 12, 2021 10:54 am Thanks, seems to work.
But I don't like the idea that the files are stored unencrypted. :?

I tried to enable Encryption after the first initial upload and it seems to work. Hmmm...
Oh interesting, you're talking about encryption on capacity tier side ?

I think there is a bug at veeam side
When you enable encryption on capicity tier it doesn't work :'(

Gostev
SVP, Product Management
Posts: 29104
Liked: 5358 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scaleway S3 Object Storage

Post by Gostev »

I highly doubt such a bug would remain unreported for over 2 years and 500PB of data moved into public clouds... I think storage-specific issue is much more likely here.

NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Scaleway S3 Object Storage

Post by NightBird »

How does work encryption on capacity tier (not on job side) ? some specific S3 API call ?

Gostev
SVP, Product Management
Posts: 29104
Liked: 5358 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scaleway S3 Object Storage

Post by Gostev »

No, we encrypt each object ourselves before sending them to object storage.

Mildur
Veeam Legend
Posts: 1694
Liked: 685 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: St. Gallen, Switzerland
Contact:

Re: Scaleway S3 Object Storage

Post by Mildur » 1 person likes this post

S3 data transfer is always encrypted on https.
The veeam capacity encryption will encrypt the metadata (and blocks?) in the S3 Storage.

If you want to add the bucket to a fresh VBR Server, you will need access data for the bucket and the encryption password from veeam to read the metadata and restore your data.
VMCE 2021 | Veeam Legends 2021
Working with Veeam since 2017 for a VCSP in Switzerland

NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Scaleway S3 Object Storage

Post by NightBird » 2 people like this post

Gostev wrote: May 13, 2021 5:30 pm No, we encrypt each object ourselves before sending them to object storage.
Really strange this error, immutability work like a charm, everything work fine, but when you enable encryption on capacity tier... error...
This is why I asked if there is a specific S3 call.

My customer opened a support case #04757486

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin » 1 person likes this post

We will check the case internally to see what might be causing the error. Thank you for sharing the case number.

SchaeferTWS
Novice
Posts: 5
Liked: 2 times
Joined: May 10, 2021 10:35 am
Full Name: Sysadmin of the day
Location: Germany
Contact:

Re: Scaleway S3 Object Storage

Post by SchaeferTWS » 1 person likes this post

@NightBird @veremin
If you create an S3 immutable storage (on Scaleway) and start with an unencrypted job you are able to sync.
After the first initial backup and scale-out processing you can enable the encryption of the backup files in the job details. Now it works. For every new job you have to do this again.

So...
  • Create backup job without encryption
  • Do first full backup and SOBR-Tiering
  • Enable encryption in the backup job
  • Works
For me it looks like the main encryption file for an job will be overwritten or something like this when you create an encrypted job. And this won't work due to immutability.

As I tried the same with MinIO this looks definitly like an issue with Scaleway here - but it does not make sense at all, I would say.

So hopefully Veeam can check.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

My customer opened a support case #04757486
We have talked with your support engineer, it seems that we have a guess about potential cause and we will verify it during webex call next week.
So hopefully Veeam can check.
Thank you for sharing additional details; much appreciated.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

The case has been escalated to R&D team. I will keep you posted on the results of our findings.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

Turned out some S3 object retention API is not implemented correctly by storage vendor, so, you should probably open a case with them to confirm. Our support engineer will provide you with more information about that. Thanks!

fouc
Lurker
Posts: 1
Liked: never
Joined: Apr 30, 2019 10:22 am
Full Name: Foucard
Contact:

Re: Scaleway S3 Object Storage

Post by fouc »

Hi,
I also do tests and I also have problems.
I am interested in the following events to know the exact procedure for creating the bucket to be able to use them with VEEAM with the OBJECTLOCK option
Thanks

SchaeferTWS
Novice
Posts: 5
Liked: 2 times
Joined: May 10, 2021 10:35 am
Full Name: Sysadmin of the day
Location: Germany
Contact:

Re: Scaleway S3 Object Storage

Post by SchaeferTWS » 1 person likes this post

Hi @fouc,

unfortunately Scaleway does not support the correct API implementation for using Veeam with objectlock.
Switch to Wasabi instead - it works nearly out of the box there! More information: https://wasabi-support.zendesk.com/hc/e ... ntegration

Be sure to modify the permissions of Wasabi like here:
post416181.html#p416181

Cheers,
Aaron

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

I am interested in the following events to know the exact procedure for creating the bucket to be able to use them with VEEAM with the OBJECTLOCK option
Currently you cannot archive this due to certain API implementation specifics. Opening a ticket with a vendor might raise additional awareness about this issue. Thanks!

NightBird
Service Provider
Posts: 231
Liked: 52 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Scaleway S3 Object Storage

Post by NightBird »

Scaleway is aware of the problem and will solve it in the coming weeks.

Reply from scaleway to my customer.

flibouille
Lurker
Posts: 1
Liked: never
Joined: Jul 09, 2021 8:27 am
Contact:

Re: Scaleway S3 Object Storage

Post by flibouille »

It's a great ad, really! Except that it's not true. Everybody knows Scaleway and most of cloud compagnies works with VEEAM, because everybody is S3 compatible.

But when you use Veeam S3 storage, you often need immutability, and it's unfortunately doesn't work with Scaleway.
MODE: COMPLIANCE
veeam unable to use backup immutability the compliance mode is not supported

MODE:GOVERNANCE
veeam unable to use backup immutability the governance mode is not supported

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

According to NightBird's reply, they are aware of the issue that prevents customers from using immutability with their storage system. We've requested an update regarding this case internally, I will share more information, once I have it. Thanks!

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scaleway S3 Object Storage

Post by veremin »

It seems that currently the issue persists - the corresponding API that allows us to prolongate object lock on the selected entities does not work correctly. Hopefully, it will be fixed soon. Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests