ClusterAPI Introduction

Cluster API is a platform agnostic way of managing physical resources (i.e. VMs) using Kubernetes.

This provides a number of advantages including:

• Familiarity to experienced K8s admins, as nodes act similarly to pods…etc.

• Tooling which directly supports Openstack, including resource creation and management.

• Support and fixes for newer Kubernetes versions without waiting for Openstack upgrades.

The Kubernetes image provided is based on upstream’s Ubuntu image, which has been forked to comply with UKRI security policy here.

Unlike Magnum, the Kubernetes version supported is completely decoupled from Openstack, with upstream typically supporting the N-1 major version (where N is the latest release).

Setup a new cluster using Cluster API

Deployment Considerations

Management Machine

For production workloads, it’s recommended to create a dedicated cluster management machine. This should only be accessible to Cluster Administrators and holds a copy of the kubeconfig. The kubeconfig cannot be recovered (unlike Magnum) after generation. Additionally, a known working dedicated machine can be used to quickly gain access and perform recovery or upgrades as required.

Account Security

A clouds.yaml file with the application credentials is also required. The credentials should not be unrestricted, as a compromised cluster would allow an attacker to create additional credentials. This file should be removed or restricted on shared machines to prevent unauthorized access.

Automatic Bootstrap

An automatic bootstrap script is recommended. This script will:

• Update your VM, install Kubectl, Helm and MicroK8s (to bootstrap)

• Verify your application credentials, and populate the project ID automatically

• Create a bootstrap cluster, install the OpenStack Provider and clusterctl

Requirements

• A Ubuntu VM

• Ensure a dedicated floating IP exists. If required, allocate an IP to the project from the External pool.

• Application Credentials

• Copy your application credentials, clouds.yaml into the management_cluster dir

• Add your allocated floating IP to user-values.yaml

• Run the bootstrap script

Configuring the management cluster

Yaml Files

The configuration is spread across multiple YAML files to make it easier to manage. These are as follows:

• values.yaml contains the default values for the cluster using the STFC Cloud service. These should not be changed.

• user-values.yaml contains some values that must be set by the user. There are also optional values that can be changed too for advanced users.

• flavors.yaml contains the Openstack flavors to use for worker nodes. Common flavors are provided and can be uncommented and changed as required. By default the cluster will use l3.nano workers by default if unspecified.

• clouds.yaml contains the Openstack application credentials. This file should be in the same directory as the other YAML files.

The cloud team will periodically update flavors.yaml, values.yaml, and user-values.yaml to reflect changes in the STFC Cloud service. These include new versions of Kubernetes or machine images, best practices, new flavors…etc. A user will pull these changes by running git pull in the cloud-capi-values directory in the future.

Description of values

• The Floating IP in user-values.yaml must be set. Optional values may also be changed as required.

• Check that the Kubernetes version in machineImage: matches the kubernetesVersion:

• The flavors.yaml file contains the Openstack flavors to use for worker nodes. These can be changed as required but will use l3.nano by default if unspecified.

Deploying Cluster

This assumes you have completed the bootstrap steps and customised your cluster as described above, if required.

• Deploy your management cluster through Helm:

• When the deployment is complete clusterctl will report the cluster as Ready: True

• Progress can be monitored with the following command in a separate terminal:

Once this is deployed you can validate your cluster is up with:

Moving the control plane

At this point the control plane is still on the minikube cluster. This is not recommended for long-lived or production workloads. We can pivot the cluster to self-manage:

Moving to a self-managed cluster

• Install clusterctl into the new cluster and move the control plane

• Ensure the control plane is now running on the new cluster:

Using the new control plane by default

• Replace the existing kubeconfig with the new cluster’s kubeconfig

• Ensure that it does not say either minikube or microk8s (i.e. your local machine)

Check the cluster status

Creating workload cluster(s)

For production workloads we recommend a management cluster which then controls one or more child cluster(s). These child clusters would include prod, staging, developers areas…etc.

• Copy or clone the cloud values to a new directory. E.g. for a prod-cluster:

• Modify the user-values and flavors as required

• Change the floating IP to a new one, each cluster must have it’s own FIP and load balancer

• Deploy this cluster, it’s nodes are managed by the management cluster

• Distribute / store the kubeconfig as required for each cluster

  • kubectl can interact with multiple clusters from the same ~/.kube/config file, however we recommend single purpose management VMs for each cluster:

  • A production cluster management machine would have prod-cluster.kubeconfig stored in ~/.kube/config

  • A developer would take dev-cluster.kubeconfig and store it in their own ~/.kube/config.

  • This creates role-based access and prevents kubectl delete --all moments wiping out an entire tree of clusters.

Manual Deployment

Background

A Kubernetes cluster is required to create the cluster. To break this “chicken-egg” problem a minikube cluster is created to bootstrap the main cluster. It is not recommended to use this cluster for any production workloads.

Bootstrap Machine Prep

A Ubuntu machine is used to provide the minikube cluster. This should use the normal cloud Ubuntu image, not the stripped down CAPI image designed for nodes.

The following packages are required and can be installed and configured using the following commands:

• Docker is used to run a Kubernetes staging cluster locally

• Snap is used to install kubectl and Helm

Install minikube and start a cluster:

• Install clusterctl and the Openstack provider into your minikube cluster:

If you run into GitHub rate limiting you will have to generate a personal API token as described here. This only requires the Repo scope, and is set on the CLI as follows

These setup steps only have to be completed once per management machine.

Openstack Preparation

• Ensure a dedicated floating IP exists. If required, allocate an IP to the project from the External pool.

• Clone https://github.com/stfc/cloud-capi-values then copy this to a directory called management_cluster

clouds.yaml Prep

• Generate your application credentials: ​. It is recommended you use Horizon (the web interface) to download the clouds.yaml file.

The clouds.yaml file should have the following format:

• Add the UUID of the project you want to create the cluster in. This is the project ID under the Openstack section which is omitted by default. This can be found here.

Your clouds.yaml should now look like:

• Place this file in the cloud-capi-values directory you cloned earlier.

• Deploy the cluster requirements