S3 On OpenStack
- 1 Overview
- 2 Prerequisites
- 3 Configuration with OpenStack CLI
- 4 Testing
- 4.1 Testing using S3CMD
- 4.1.1 Environment setup and installing s3cmd
- 4.1.2 Configure s3cmd
- 4.1.3 Creating a new bucket
- 4.1.4 Listing existing buckets
- 4.1.5 Uploading a file
- 4.1.6 Downloading a file
- 4.2 Testing using awscli
- 4.1 Testing using S3CMD
- 5 Example configuration of a Helm Application (Harbor)
- 6 References
Overview
S3 REST API is emulated in Swift, allowing you and your applications to interact with Swift object storage as a native S3 object store.
For full compatibility see S3/Swift REST API Comparison Matrix
Prerequisites
Contact cloud support at cloud-support@stfc.ac.uk with an estimate of how much storage you need for your object store.
Open https://openstack.stfc.ac.uk/project/containers/ on the project you’d like quota for. This will automatically create a user on the storage layer, helping us get quota allocated faster.
Configuration with OpenStack CLI
Generating EC2 credentials with OpenStack CLI
These credentials allow you to manage your object storage in OpenStack through Swifts emulation of the S3 REST API.
Creating credentials
To create a new EC2 credential
openstack ec2 credentials create
You should get an output like this, please keep these details safe and secure:
+------------+---------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+---------------------------------------------------------------------------------------------------------+
| access | 9a5fc02e2ed64a7cad249a8477d712457 |
| links | {'self': 'https://openstack.nubes.rl.ac.uk:5000/v3/users/...'} |
| project_id | fd12468zz0f24ed56721c68ef7a345 |
| secret | 3806354657e54aa3828285c5f5g68d8s |
| trust_id | None |
| user_id | 0aa830b1256d4d789cdcab81b9f32bc5 |
+------------+---------------------------------------------------------------------------------------------------------+
Listing current credentials
openstack ec2 credentials listDeleting credentials
OpenStack Object store region endpoint
The region end point to connect to your new OpenStack ‘S3’ object store.
Testing
Testing using S3CMD
You can use a CLI tool called S3CMD to test your credentials and access to your object store.
Environment setup and installing s3cmd
Create your virtual environment (optional)
Install s3cmd with pip
Configure s3cmd
Most of these options […] can be left as default by pressing the enter key, the main ones we need to configure are highlighted here.
Hopefully your credentials work fine, so now you’re ready to use them as you require!
Creating a new bucket
Listing existing buckets
Uploading a file
Downloading a file
Testing using awscli
Install
awscliusing either pip into a venv or Conda:
Configure the AWS Client following the prompts to add your access key and secret
Edit
~/.aws/configwith the following values:
Run
aws s3 lsto ensure the client can connectAWS Commands can now be used from the CLI, further details can be found on the AWS S3 Documentation
Example configuration of a Helm Application (Harbor)
Example configuration of S3 in Harbor HA to highlight that it works just the same as native S3.
References
https://openmetal.io/docs/manuals/openstack-admin/access-swift-s3-api
https://openmetal.io/docs/manuals/openstack-admin/swift-s3-api-access-with-s3cmd
https://docs.openstack.org/swift/latest/s3_compat.html