STFC Docker Hub Mirror

Owned by Chris Green
Last updated: Aug 16, 2023
3 min read

Introduction

The STFC cloud team provides an internal Docker Hub mirror at https://dockerhub.stfc.ac.uk which is strongly recommended; users avoid rate limit issues, can pull large images faster over the local network and is free.

Docker Hub limit an IP to 100 pulls per 6 hours. Instances with floating IPs will have separate rate limits, whilst all other instances use a shared limit (as all pulls come from a single external IP).

Docker will automatically fall back to using Docker Hub directly if the mirror is ever unavailable, so there is minimal risk to applying these changes.

New Instances

By default all new STFC Cloud instances will use the mirror by default. To confirm the mirror is being used see https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211812505/STFC+Docker+Hub+Mirror#Checking-that-the-mirror-is-being-used.

Existing Instances

Existing instances will automatically update to pull in the required file described here https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211812505/STFC+Docker+Hub+Mirror#Restarting-Docker. A restart of the instance or docker service is required to start using the mirror.

Checking that the mirror is being used

Checking configuration exists

Users can quickly check if they have the mirror configuration present by running:

cat /etc/docker/daemon.json

An output similar to this indicates that the configuration is present:

{ "registry-mirrors": ["https://dockerhub.stfc.ac.uk"] }

To check whether the docker daemon is using this config you can run the following:

docker info

An output which includes the following near the bottom will indicate that the mirror is being used:

Monitoring Logs

For further peace of mind the logs can be checked. Docker will not produce any logs when pulling from the mirror successfully but will log an error if it fails to use direct connections:

  • Select an image which has not been pulled on the System before, for example Ubuntu or Debian latest.

  • In a separate terminal run a docker pull of your image:

  • If the mirror could be contacted there will be no output in the logs about it

  • If the mirror could not be used something similar to this will appear:

For more support see https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211812505/STFC+Docker+Hub+Mirror#Troubleshooting-Mirror-Connection.

Manually Configuring the Mirror

For most users this is not required; instances will automatically update the file as described in https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211812505/STFC+Docker+Hub+Mirror#Checking-that-the-mirror-is-being-used.

If you have internal machines that are outside of Openstack or separately managed or you need to apply the changes proactively the following steps can be followed:

Docker Daemon

If you are using Docker (SL7 / Ubuntu / CoreOS / K8s < 1.20) the following steps can be performed using sudo or root. By default most distributions do not pre-create this file:

Add or append the following JSON:

Then restart the service (see https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211812505/STFC+Docker+Hub+Mirror#Restarting-Docker).

Containerd

As of Kubernetes 1.20, a future release (TBA) will use Containerd by default. The STFC Core OS image already contains the mirror information on users behalf at /etc/containers/registries.conf

Further documentation on manually on setting up containerd will be included when upstream Kubernetes uses containerd by default and further internal testing is completed.

Restarting Docker

Restarting the Docker Daemon will also pause and resume any running containers. This may result in service interruption or lost work.

The Docker Daemon can be restarted to apply the changes. Any running containers will be either paused and resumed, or restarted during this process.

Depending on the operating system the service is either called docker or dockerd, with the former being more common. To restart simply run:

To verify the service has resumed successfully:

Troubleshooting Mirror Connection

The logs can be checked if you suspect the mirror is not being used (see https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211812505/STFC+Docker+Hub+Mirror#Monitoring-Logs).

If Docker is failing back some simple diagnostics steps can be performed and/or cloud support can be contacted. Including the output of these steps in your email will enable us to provide support faster: