Create new Wasabi S3 object lock bucket with aws cli?

[pirx]

I want to try out Wasabi as alternative for our S3 buckets. Currently it's not possible to create a bucket in s3.eu-central-1.wasabisys.com region via GUI with a trial account (this is noted in the welcome mail).

https://wasabi-support.zendesk.com/hc/e ... th-Wasabi-

Do I have to do anything else than just issue the below command? Add anything? I remember that I've read something about object lock retention that has to be changed to work with Veeam but I don't find it anymore.

Code: Select all

$ aws s3api create-bucket --bucket veeam-wasabi-test --object-lock-enabled-for-bucket --endpoint-url=https://s3.eu-central-1.wasabisys.com

[chrisWasabi]

Hi Pirx,

This is all you need to do in order to create the Object Lock bucket!

You want to leave the default settings alone on the bucket. You can read in the Veeam Considerations and Limitations for Immutability doc.

"When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode."

[pirx]

The link gives me a 404. I just checked the object-lock-configuration.

{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}

In GUI I see all of the 3 options unchecked. I guess "None" is then active as I can't activate it anyway.

[veremin]

The link has been fixed. The provided command looks good, you can go ahead and try to create immutable object storage repository. Thanks!

[pirx]

I was able to create a S3 repository but I'm getting some strange errors now when I click through the properties again "Loading S3 compatible storage system configuration". I can reach the endpoint on port 443 from the configured proxy. Maybe a problem on Wasabi side?

I only whitelisted s3.eu-central-1.wasabisys.com on our firewall.


ComputerName : s3.eu-central-1.wasabisys.com
RemoteAddress : 130.117.252.18
RemotePort : 443
InterfaceAlias : PROD
SourceAddress : 10.11.35.22
PingSucceeded : False
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded : True

Code: Select all

> Test-NetConnection s3.eu-central-1.wasabisys.com  -Port 443
WARNING: Ping to s3.eu-central-1.wasabisys.com failed -- Status: TimedOut


ComputerName           : s3.eu-central-1.wasabisys.com
RemoteAddress          : 130.117.252.18
RemotePort             : 443
InterfaceAlias         : PROD
SourceAddress          : xxxxx
PingSucceeded          : False
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : True   <<<-----

[pirx]

Does anyone have an idea why Veeam tries to connect to 151.139.128.14 when I add a Wasabi S3 bucket? When this destination is not cleared on the firewall, the connection runs into a timeout.

Code: Select all

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting OwningProcess
------------                        --------- -------------                       ---------- -----       -------------- -------------
10.11.35.22                         53616     130.117.252.17                      443        Established Internet       12008
10.11.35.22                         53618     151.139.128.14                      80         SynSent                    12008

According to whois it belongs to

Code: Select all

ISP	StackPath LLC
Usage Type	Content Delivery Network
Domain Name	stackpath.com

Is this documented somewhere? Is this a Wasabi or a Veeam requirement? Connections to our existing AWS S3 buckets does not need access to this IP.

[pirx]

I've received feedback from Wasabi support, they don't know this IP 151.139.128.14 and suspect that it's a Veeam issue. I can access the Wasabi bucket without problems with aws cli from any gateway host and no connection to 151.139.128.14 is opened.

So why does Veeam need this IP? We have several firewall clearances for certificate checks, AWS S3 is working for a long time without an issue. What am I missing (created Veeam case 04928633). Any KB article I can check?

[Gostev]

I just entered "StackPath Wasabi" into Google and got some interesting hits, which tell me that Wasabi support should probably escalate the case within their organization to someone who is more in the know

[pirx]

Any particular result you wanna share? I also found some results, but they are mostly about how a customer can use Stackpath's CDN. But I did not configure anything that is described there.

[Gostev]

I just saw there's some play between the two companies, while Veeam don't have any play with StackPath for sure.

[pirx]

I still think this has to be something in Veeeam, as I can access the Wasabi bucket from any Veeam gateway with aws cli just fine. Only in Veeam it's not possible and the additional IP is connected.

[pirx]

Maybe this has nothing to do with Wasabi in particular, but VeeamAgent is the process that is trying to connect to this IP. If the issue is certificate verification I would assume that we have this problem with AWS S3 too. Which we not have.

Code: Select all

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting OwningProcess
------------                        --------- -------------                       ---------- -----       -------------- -------------
10.11.35.22                         63453     151.139.128.14                      80         SynSent                    21992



PS C:\windows\system32> Get-Process -Id 21992

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
    425      19     5996      15092       0,14  21992   0 VeeamAgent

[AlexHeylin]

Hi Pirx,

I'm using immutable Wasabi from VBR. I suggest two things:
1. If you can't do something from the trial account, make it a production one. It's cheap anyway and there's no commitment.
2. If you're having problems and you're SURE you're following the instructions then allow all traffic out from the firewall and try again.

It works fine, so any problems you're having are almost certainly with your config / the process you're following, and not VBR or Wasabi.

[pirx]

Well, I don't know what I could have done wrong by following https://wasabi-support.zendesk.com/hc/e ... th-Wasabi- and https://wasabi-support.zendesk.com/hc/e ... t-regions-

- without fw rule for 151.139.128.14 aws cli and s3browser are both able to connect to the bucket - IP 151.139.128.14 is not connected
- without fw rule for 151.139.128.14 Veeam is failing to add the bucket VeeamAgent connects to 151.139.128.14 port 80
- with fw rule access rule for 151.139.128.14 Veeam is able to connect to bucket

For me it's quite clear that this IP is used somewhere in Veeam and I've no idea why Veeam is not connecting this IP when using AWS S3 buckets. And maybe it's documented somewhere, but I fail to find the right place. Problem is that I don't want to whitelist a random IP that is not documented anywhere. Even if I did this, there could be others that have to be whitelisted too.

[pirx]

Code: Select all

Name:    ocsp.sectigo.com
Address:  151.139.128.14

This seems to be related to certificate for console.wasabisys.com. Funny that it works for aws cli and s3browser without fw clearance but Veeam needs it. Let's see if this does the trick.

[pirx]

With this IP added to the firewall rule Veeam can access the bucket and I was able to create the capacity extent. I'm still wondering if this IP should be documented somewhere either on Veeam side or Wasabi.

[veremin]

You can check where this IP is coming from by opening properties of Wasabi certificate, there you will see an address (CA endpoint) used to verify the certificate validity. Hope this clarifies your question. Thanks!

[pirx]

Where would I check the certificate in Veeam - because I got the error only I Veeam? I mean, now I know what this IP is about. But the IP does not resolve to ocsp.sectigo.com and when I check s3.eu-central-1.wasabisys.com in browser I get redirected to https://wasabi.com/ which has a different certificate from Let's encrypt.

I know https://www.veeam.com/kb3215 and I think it would be good to have a list of needed IP'S/URL's for S3 providers that are supported by Veeam.

[veremin]

You can open a certificate store, find a Wasabi certificate, open its properties, go to Details > Authority Information Access. There you will see ocsp.sectigo.com that gets resolved to the said IP. You are experiencing issues on a backup server, because the server tries to establish connection to this address to validate the certificate and fails because of firewall rules. Thanks!

[pirx]

Maybe a new KB or an addition to https://www.veeam.com/kb3215 would be a good idea?

[veremin]

But how you envision this KB? Would it contain the references to all S3 compatible storage systems, their certificates and addresses used to validate them? It does not look like an Veeam-specific issue that needs public description, as it relates more to general system configuration (certificate and their CA validation endpoints). Thanks!

[pirx]

Well, but there is one KB there for AWS. And the exact same error message I had with Wasabi. I still don't know why I only got an error in Veeam, as all my tests with aws cli and s3browser etc were successful without this IP whitelisted on the firewall. Just Veeam was complaining.

[veremin]

It was an issue on our side for which we even created a hotfix. Plus, it talks about specific public cloud with specific endpoints (which do not have tendency to change), while here we will need to document all S3 compatible devices we find along with their certificate specifics - not necessarily feasible.

Anyway, thanks for the feedback, it has been noted.