1 Overview
2 Prerequisites
3 Configuration with OpenStack CLI
- 3.1 Generating EC2 credentials with OpenStack CLI
  - 3.1.1 Creating credentials
  - 3.1.2 Listing current credentials
  - 3.1.3 Deleting credentials
- 3.2 OpenStack Object store region endpoint
4 Testing
- 4.1 Testing using S3CMD
  - 4.1.1 Environment setup and installing s3cmd
  - 4.1.2 Configure s3cmd
  - 4.1.3 Creating a new bucket
  - 4.1.4 Listing existing buckets
  - 4.1.5 Uploading a file
  - 4.1.6 Downloading a file
- 4.2 Testing using awscli
5 Example configuration of a Helm Application (Harbor)
6 References

Overview

S3 REST API is emulated in Swift, allowing you and your applications to interact with Swift object storage as a native S3 object store.

For full compatibility see S3/Swift REST API Comparison Matrix

Prerequisites

Contact cloud support at cloud-support@stfc.ac.uk with an estimate of how much storage you need for your object store.
Open https://openstack.stfc.ac.uk/project/containers/ on the project you’d like quota for. This will automatically create a user on the storage layer, helping us get quota allocated faster.

Configuration with OpenStack CLI

Generating EC2 credentials with OpenStack CLI

These credentials allow you to manage your object storage in OpenStack through Swifts emulation of the S3 REST API.

Creating credentials

To create a new EC2 credential

openstack ec2 credentials create

You should get an output like this, please keep these details safe and secure:

+------------+---------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                   |
+------------+---------------------------------------------------------------------------------------------------------+
| access     | 9a5fc02e2ed64a7cad249a8477d712457                                                                        |
| links      | {'self': 'https://openstack.nubes.rl.ac.uk:5000/v3/users/...'}                                          |
| project_id | fd12468zz0f24ed56721c68ef7a345                                                                        |
| secret     | 3806354657e54aa3828285c5f5g68d8s                                                                        |
| trust_id   | None                                                                                                    |
| user_id    | 0aa830b1256d4d789cdcab81b9f32bc5                                                                        |
+------------+---------------------------------------------------------------------------------------------------------+

Listing current credentials

openstack ec2 credentials list

Deleting credentials

openstack ec2 credentials delete <access_key>

OpenStack Object store region endpoint

The region end point to connect to your new OpenStack ‘S3’ object store.

s3.echo.stfc.ac.uk

Testing

Testing using S3CMD

You can use a CLI tool called S3CMD to test your credentials and access to your object store.

Environment setup and installing s3cmd

Create your virtual environment (optional)

virtualenv .venv
source .venv/bin/activate

Install s3cmd with pip

pip install s3cmd

Configure s3cmd

Most of these options […] can be left as default by pressing the enter key, the main ones we need to configure are highlighted here.

s3cmd --configure
[...]
Access Key: <access_key>
Secret Key: <secret_key>
[...]
S3 Endpoint: https://s3.echo.stfc.ac.uk
[...]
DNS-style bucket+hostname:port template for accessing a bucket: s3.echo.stfc.ac.uk
[...]
Use HTTPS protocol: yes
[...]
Test access with supplied credentials? y

Hopefully your credentials work fine, so now you’re ready to use them as you require!

Creating a new bucket

s3cmd mb s3://new-bucket
Bucket 's3://new-bucket/' created

Listing existing buckets

s3cmd ls
2023-11-01 11:13  s3://s3-demo
2023-11-01 11:15  s3://new-bucket

Uploading a file

s3cmd put hello-world.txt s3://s3-demo/hello-world.txt

Downloading a file

s3cmd get s3://s3-demo/hello-world.txt

Testing using awscli

Endpoint Support was fixed in >=1.29.0 and >=2.13.0. Please check your version with the --version flag if you are experiencing problems connecting

Install awscli using either pip into a venv or Conda:

sudo apt-get install awscli  # Ubuntu
pip install awscli  # Venv

Configure the AWS Client following the prompts to add your access key and secret

awscli configure

Edit ~/.aws/config with the following values:

[default]
endpoint_url =  https://s3.echo.stfc.ac.uk
region = RegionOne

Ensure or edit the credentials files with the appropriate values in ~/.aws/credetials

[default]
aws_access_key_id = abc123...
aws_secret_access_key = zyx0987...

Set the appropriate permission for both using chmod 600 ~/.aws/*
Run aws s3 ls to ensure the client can connect
AWS Commands can now be used from the CLI, further details can be found on the AWS S3 Documentation

Example configuration of a Helm Application (Harbor)

Example configuration of S3 in Harbor HA to highlight that it works just the same as native S3.

s3:
    bucket: harbor-bucket
    accessKey: <access_key>
    secretKey: <secret_key>
    regionendpoint: s3.echo.stfc.ac.uk
    encrypt: true
    secure: true