ClusterAPI Introduction

Cluster API is a platform agnostic way of managing physical resources (i.e. VMs) using Kubernetes.

This provides a number of advantages including:

• Familiarity to experienced K8s admins, as nodes act similarly to pods…etc.

• Tooling which directly supports Openstack, including resource creation and management.

• Support and fixes for newer Kubernetes versions without waiting for Openstack upgrades.

The Kubernetes image provided is based on upstream’s Ubuntu image, which has been forked to comply with UKRI security policy here.

Unlike Magnum, the Kubernetes version supported is completely decoupled from Openstack, with upstream typically supporting the N-1 major version (where N is the latest release).

Setup a new cluster using Cluster API

Deployment Considerations

Management Machine

For production workloads, it’s recommended to create a dedicated cluster management machine. This should only be accessible to Cluster Administrators and holds a copy of the kubeconfig. The kubeconfig cannot be recovered (unlike Magnum) after generation. Additionally, a known working dedicated machine can be used to quickly gain access and perform recovery or upgrades as required.

Account Security

A clouds.yaml file with the application credentials is also required. The credentials should not be unrestricted, as a compromised cluster would allow an attacker to create additional credentials. This file should be removed or restricted on shared machines to prevent unauthorized access.

Automatic Bootstrap

An automatic bootstrap script is recommended. This script will:

• Update your VM, install Kubectl, Helm and MicroK8s (to bootstrap)

• Verifies your application credentials, and adds the project ID

• Creates a file containing sensitive helm values (app cred, floating IP, etc) at /tmp/capi/secret-values.yaml required for setting up a CAPI cluster

• Creates a bootstrap cluster, install the OpenStack Provider and clusterctl

Requirements

• A Ubuntu VM

• Ensure a dedicated floating IP exists. If required, allocate an IP to the project from the External pool.

• Application Credentials

• Copy your application credentials, clouds.yaml into cloud-helm-charts/charts/stfc-cloud-openstack-cluster. Make sure the name is clouds.yaml

• Run the bootstrap script, provide your Floating IP as an argument

Configuring the management cluster

Yaml Files

The configuration is spread across multiple YAML files to make it easier to manage. These are as follows:

• values.yaml contains the default values for the cluster using the STFC Cloud service. These should not be changed.

• user-values.yaml contains some values that must be set by the user. There are also optional values that can be changed too for advanced users.

• flavors.yaml contains the Openstack flavors to use for worker nodes. Common flavors are provided and can be uncommented and changed as required. By default the cluster will use l3.nano workers by default if unspecified.

• clouds.yaml contains the Openstack application credentials. This file should be in the same directory as the other YAML files.

• secret-values.yaml.template contains templated yaml file which details what sensitive values are needed to create the cluster (application credential / Float IP). Copy this file and edit it, or use the bootstrap script to create this for you

Description of values

• Check that the Kubernetes version in machineImage: matches the kubernetesVersion:

• The flavors.yaml file contains the Openstack flavors to use for worker nodes. These can be changed as required but will use l3.nano by default if unspecified.

Deploying Cluster

This assumes you have completed the bootstrap steps and customised your cluster as described above, if required.

• Deploy your management cluster through Helm:

• When the deployment is complete clusterctl will report the cluster as Ready: True

• Progress can be monitored with the following command in a separate terminal:

Once this is deployed you can validate your cluster is up with:

Moving the control plane

At this point the control plane is still on the minikube cluster. This is not recommended for long-lived or production workloads. We can pivot the cluster to self-manage:

Moving to a self-managed cluster

• Install clusterctl into the new cluster and move the control plane

• Ensure the control plane is now running on the new cluster:

Using the new control plane by default

• Replace the existing kubeconfig with the new cluster’s kubeconfig

• Ensure that it does not say either minikube or microk8s (i.e. your local machine)

Check the cluster status

Creating workload cluster(s)

For production workloads we recommend a management cluster which then controls one or more child cluster(s). These child clusters would include prod, staging, developers areas…etc.

• Copy or clone the cloud values to a new directory. E.g. for a prod-cluster:

• Modify the user-values and flavors as required

• Change the floating IP to a new one, each cluster must have it’s own FIP and load balancer

• Deploy this cluster, it’s nodes are managed by the management cluster

• Distribute / store the kubeconfig as required for each cluster

  • kubectl can interact with multiple clusters from the same ~/.kube/config file, however we recommend single purpose management VMs for each cluster:

  • A production cluster management machine would have prod-cluster.kubeconfig stored in ~/.kube/config

  • A developer would take dev-cluster.kubeconfig and store it in their own ~/.kube/config.

  • This creates role-based access and prevents kubectl delete --all moments wiping out an entire tree of clusters.

Manual Deployment

Background

A Kubernetes cluster is required to create the cluster. To break this “chicken-egg” problem a minikube cluster is created to bootstrap the main cluster. It is not recommended to use this cluster for any production workloads.

Bootstrap Machine Prep

A Ubuntu machine is used to provide the minikube cluster. This should use the normal cloud Ubuntu image, not the stripped down CAPI image designed for nodes.

Versions

The latest validated versions can be found at cloud-capi-values/dependencies.json at master · stfc/cloud-capi-values where they map to

• azimuth-capo-addon-version : $addon-version

• capo-provider-version : $capo-provider-version

• cluster-ctl-version: $cluster-ctl-version

The following packages are required and can be installed and configured using the following commands:

• Docker is used to run a Kubernetes staging cluster locally

• Snap is used to install kubectl and Helm

Install minikube and start a cluster:

• Install clusterctl and the Openstack provider into your minikube cluster:

If you run into GitHub rate limiting you will have to generate a personal API token as described here. This only requires the Repo scope, and is set on the CLI as follows

These setup steps only have to be completed once per management machine.

Openstack Preparation

• Ensure a dedicated floating IP exists. If required, allocate an IP to the project from the External pool.

• Clone https://github.com/stfc/cloud-helm-charts.git

secret-values.yaml Prep

• Generate your application credentials: ​. It is recommended you use Horizon (the web interface) to download the clouds.yaml file.

The clouds.yaml file should have the following format:

Add the UUID of the project you want to create the cluster in. This is the project ID under the Openstack section which is omitted by default. This can be found here.

Your clouds.yaml should now look like:

• Copy the secret-values.yaml.template somewhere safe (other than the repo) to add secrets (to prevent accidentally committing and pushing your secrets! )

Fill out the copied yaml file with your floating IP and application credential

• Deploy the cluster requirements

• Follow the instructions in Cluster API Setup | Deploying Cluster