1 Overview
2 Prerequisites
3 Configuration with OpenStack CLI
- 3.1 Generating EC2 credentials with OpenStack CLI
  - 3.1.1 Creating credentials
  - 3.1.2 Listing current credentials
  - 3.1.3 Deleting credentials
- 3.2 OpenStack Object store region endpoint
4 Testing
- 4.1 Testing using S3CMD
  - 4.1.1 Environment setup and installing s3cmd
  - 4.1.2 Configure s3cmd
  - 4.1.3 Creating a new bucket
  - 4.1.4 Listing existing buckets
  - 4.1.5 Uploading a file
  - 4.1.6 Downloading a file
- 4.2 Testing using awscli
5 Example configuration of a Helm Application (Harbor)
6 References

Overview

S3 REST API is emulated in Swift, allowing you and your applications to interact with Swift object storage as a native S3 object store.

For full compatibility see S3/Swift REST API Comparison Matrix

Prerequisites

Contact cloud support at cloud-support@stfc.ac.uk with an estimate of how much storage you need for your object store.
Open https://openstack.stfc.ac.uk/project/containers/ on the project you’d like quota for. This will automatically create a user on the storage layer, helping us get quota allocated faster.

Configuration with OpenStack CLI

Generating EC2 credentials with OpenStack CLI

These credentials allow you to manage your object storage in OpenStack through Swifts emulation of the S3 REST API.

Creating credentials

To create a new EC2 credential

openstack ec2 credentials create

You should get an output like this, please keep these details safe and secure:

+------------+---------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                   |
+------------+---------------------------------------------------------------------------------------------------------+
| access     | 9a5fc02e2ed64a7cad249a8477d712457                                                                        |
| links      | {'self': 'https://openstack.nubes.rl.ac.uk:5000/v3/users/...'}                                          |
| project_id | fd12468zz0f24ed56721c68ef7a345                                                                        |
| secret     | 3806354657e54aa3828285c5f5g68d8s                                                                        |
| trust_id   | None                                                                                                    |
| user_id    | 0aa830b1256d4d789cdcab81b9f32bc5                                                                        |
+------------+---------------------------------------------------------------------------------------------------------+

Listing current credentials

openstack ec2 credentials list

Deleting credentials

OpenStack Object store region endpoint

The region end point to connect to your new OpenStack ‘S3’ object store.

Testing

Testing using S3CMD

You can use a CLI tool called S3CMD to test your credentials and access to your object store.

Environment setup and installing s3cmd

Create your virtual environment (optional)

Install s3cmd with pip

Configure s3cmd

Most of these options […] can be left as default by pressing the enter key, the main ones we need to configure are highlighted here.

Hopefully your credentials work fine, so now you’re ready to use them as you require!

Creating a new bucket

Listing existing buckets

Uploading a file

Downloading a file

Testing using awscli

Endpoint Support was fixed in >=1.29.0 and >=2.13.0. Please check your version with the --version flag if you are experiencing problems connecting

Install awscli using either pip into a venv or Conda:

Configure the AWS Client following the prompts to add your access key and secret

Edit ~/.aws/config with the following values:

Ensure or edit the credentials files with the appropriate values in ~/.aws/credetials

Set the appropriate permission for both using chmod 600 ~/.aws/*
Run aws s3 ls to ensure the client can connect
AWS Commands can now be used from the CLI, further details can be found on the AWS S3 Documentation