Stuck Cinder volumes issue

Summary:

Many users may be experiencing some issues with Cinder volumes after the power blips on Jun 18, 2026 If you:

• have trouble mounting a volume

• have trouble writing/accessing an exiting mounted volume

• have trouble accessing a VM with a mounted volume

• have trouble deleting a volume

• have trouble creating volume snapshots

Please consider submitting a ticket to cloud-support@stfc.ac.uk

Why?

The power incident on Jun 18, 2026 has left stale locks on ceph RBD volumes that cinder uses. This has resulted in some volumes being stuck in Read-Only mode - preventing writes and making the volume inaccessible. The data on the volume is likely consistent and healthy and has a high chance of being recovered once we remove these stale locks

Depending on how the volume is used - this could affect the boot sequence of the VM - resulting in some VMs with cinder volumes becoming inaccessible.

The Read-Only also results in some cinder volumes are failing to delete or take snapshots

The Fix

To fix this issue, we need to perform a procedure on the ceph side to remove stale locks.

To recover the affected volumes, some backend Ceph maintenance is needed, and this work requires the affected VMs and K8s clusters to be powered off and the Cinder volumes to be detached while this takes place to prevent any data corruption.

If you believe you have Cinder volumes that are affected, please open a ticket to cloud-support@stfc.ac.uk

We will schedule a time with the Ceph team every week to complete this procedure on a batch of volumes at a time.

Please include:

• The volume Name and ID

• If you want us to recover these volumes - please either:

  • Shutoff your VMs/Clusters and detach volumes yourself (we’ll give you a time-frame we them off for when you submit the ticket)

  • Authorize us to shutoff your VMs/Clusters and detach your volumes before we begin the procedure

This will likely take a full day and we will inform you once the procedure is completed so you can restart your clusters/VMs

Please raise a ticket to cloud-support@stfc.ac.uk if you have any questions

Who is affected:

Any users of the Cinder Block Storage service with existing volumes

New cinder volumes - created after the power incident Jun 18, 2026 seem unaffected

Contact