This is adapted from: https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html

For more complex upgrades, additional context or further considerations the upstream documentation is recommended.

The following steps will show you how to upgrade your CAPI cluster from start to finish. There are many components that make up a CAPI cluster, each of these will need to be upgraded in the following order:

1. Clusterctl: the command-line tool which is used to interact with Cluster API. We use this to perform upgrades on Cluster API and its dependencies
   
   

2. Cluster API and its dependencies: this provides a declarative set of APIs that are used to provision, delete and manage multiple kubernetes clusters https://github.com/kubernetes-sigs/cluster-api
   
   

3. Provisioned Clusters via Helm Charts: each CAPI Cluster that has already been provisioned must be upgrading to the latest supported helm chart version.

4. Kubernetes and OS Upgrades: the Kubernetes version and underlying packages at the OS level must be upgraded by updating your cluster to use the latest CAPI image released by the Cloud team

Clone our cloud-capi-values repo - http://github.com/stfc/cloud-capi-valuesand run source set-env.sh

this will load the latest supported dependency versions

Find out if you need to upgrade:

Check the versions of the underlying Cluster API components by running:

This command will show you the CURRENT VERSION and the version you can upgrade to if you need to.

You only need to perform upgrade for this if:

• cluster-api version < $CLUSTER_API

• infrastructure-openstack version < $CAPO_PROVIDER_VERSION

If it is equal to or greater you can ignore this upgrade

Check the kubernetes version of your cluster and what image the cluster is using.

You can find it on Openstack or by running:

compare this to the latest image release - check on openstack by running

The latest release will have the latest patch version.

We recommend always upgrading to the current kubernetes image once it becomes available.

Helm Chart Version: check the helm chart versions by running

You only need to perform upgrade for this if:

• cluster-api-addon-provider chart version < $ADDON_VERSION

• <cluster-name> (openstack-cluster chart release) chart version < $CAPI_HELM_CHART_VERSION

If it is equal to or greater you can ignore this upgrade

If you’re updating from an older cluster (>2 years running without upgrades) You might need to perform intermediary upgrade steps to catch up to the latest version.

Check this table and see what you can upgrade to. https://cluster-api.sigs.k8s.io/reference/versions#supported-versions-matrix-by-provider-or-component

This Upgrade Procedure docs assumes the reader/administrator is doing a full upgrade of all components. These can be upgraded independently with the caveat that the Infrastructure layer supports the version of Kubernetes planned: https://cluster-api.sigs.k8s.io/reference/versions

clusterctl is a command-line tool for managing Kubernetes clusters managed by Cluster API.

Find out more here: http://cluster-api.sigs.k8s.io/clusterctl/overview

Download the latest version which supports your cluster version.

In the case of multiple upgrades, download the latest clusterctl which supports your current Kubernetes cluster version including the management plane.

Copy the latest supported release $CLUSTER_API:

Once clusterctl is installed, you can upgrade various CAPI components including the CAPI Openstack Provider:

clusterctl upgrade plan

This will show you which versions you can upgrade to safely, and what command you should run to apply the changes using clusterctl upgrade apply --contract <contract-name>

Alternatively you can run:

1. Update the addon provider on all clusters first

2. Update the K-ORC dependency on any clusters managing infrastructure. (Check with kubectl get machines -A).

   1. This used to be shipped as an implicit requirement for CAPO, but as of CAPO 0.12+ administrators must to install or update the specified version manually:

3. The next step is to update each of your clusters. You can do so in any order, but it is recommended to follow this approach:

   1. Dev Worker cluster(s)

   2. Staging Worker cluster(s)

   3. Management cluster

   4. Production Worker cluster(s)

On your Management cluster run:

• Monitor the upgrade using clusterctl describe cluster <cluster_name> -n clusters

The operating system and associated packages can be updated independently, e.g. to apply security patches to the host OS.

The Cloud Team provides a lightweight VM image purpose built for CAPI clusters.

The image naming convention is as follows:

e.g. capi-ubuntu-2204-kube-v1.33.4

Images are updated for 2 reasons:

1. To make compatible with new Kubernetes versions

2. To patch underlying OS system-level packages to ensure that they come meet the combined UKRI and STFC Cloud security policies - see Terms Of Service

Steps

• Lookup the latest image build for Kubernetes, this can be found in images section of the web interface

• Alternatively, git pull to grab the latest version from the cloud-capi-values repo

  • Note: this will only work if you are 1 major version behind the latest. If you are >=2 versions behind you’ll need to manually edit the fields as described below:

• Edit the kubernetesVersion in user-values.yaml to match the image name

• Edit the machineImage in user-values.yaml to use the latest patch release

• Wait for the rollout of new infra to complete

  • The rollout can be monitored with kubectl get kcp -A and kubectl get md -A

  • Machine details can be found in kubectl get machines -A and kubectl get openstackmachines -A

• Repeat for each major upgrade step

  • You can only do a single major upgrade at a time, e.g. 1.29.12 to 1.30.4, then 1.31.6

To upgrade major versions you will need to follow the Cluster API Upgrade section first then Cluster API Upgrade for each hop.

You’ll need to upgrade VM images and Kubernetes version to the latest patch version available before doing any major upgrades:

I.e. if you’re on 1.29.12 upgrade to the latest 1.29.x, this ensures any bug-fixes are applied which could prevent later upgrades.

We aim to support Kubernetes versions and keep CAPI images for N -1 → all supported End of Life (EOL) versions from: https://kubernetes.io/releases/ OR the oldest CAPI image we have that gives an upgrade path - whichever is the older; where N is the latest Kubernetes version.

For example, as of the 29/01/2025, the latest Kubernetes version is v1.32 and therefore, we will provide CAPI images for Kubernetes versions v1.31 → v1.26 (oldest image we hold).

• Once we provide the latest N-1 image, we will keep the oldest image (v1.26) for a minimum of 4 weeks to let you upgrade older clusters and notify any users accordingly.

Open

• Check the machines and openstackmachines CRDs match the VMs in the web interface

  • kubectl get machines -A and kubectl get openstackmachines -A

  • Check the control plane node’s status kubectl describe machine <name> -n clusters

• Logs are available if nothing is happening / the process is stuck

  • OpenStack logs: kubectl logs deploy/capo-controller-manager -n capo-system -f

  • CAPI logs: kubectl logs deploy/capi-controller-manager -n capi-system -f

• Check the control plane status:

  • kubectl describe kcp/<name>-control-plane -n clusters

  • Check for events on the management cluster: kubectl get events -n clusters --watch

• Check you have access via kubectl

  • This could indicate an OpenStack networking configuration problem if you do not

  • Check the LBs and networks exist - if not check the CAPO logs on the management cluster

• Check etcd is healthy with kubectl get pods -n kube-system:

  • If they’re failing to start kubectl describe pod/etcd-<name> -n kube-system

  • If they’re running check they’re healthy with kubectl logs pod/etcd-<name> -n kube-system

  • In the event etcd is unhealthy contact the cloud team to assist with recovery

• Check the kubeapi pod is starting per machine

  • If they’re failing to start kubectl describe pod/kubeapi-<name> -n kube-system

  • If they’re running check they’re healthy with kubectl logs pod/kubeapi-<name> -n kube-system

  • In the event the Kubelet is failing to start or is unhealthy contact the cloud to assist with recovery

This affects users upgrading an existing cluster to the following minor versions (or beyond):

• v1.36.1

• v1.35.5

• v1.34.8

• v1.33.12

Caused by a security improvement in the minor versions above. The cause and a fix is described in these docs.

In short, the following explicit CRB must be applied to restore permissions. Users who have tailored their RBACs or cluster roles will need to adapt the following as appropriate: