Manila
- Chris Green
- Ramzi Jalili
- david.fairbrother@stfc.ac.uk
- Megan O'Flynn
- 1 Manila: Shared File Systems as a Service
- 1.1 Manila Commands
- 1.1.1 OpenStack CLI
- 1.1.2 Manila CLI
- 1.2 Usage
- 1.2.1 View Share Quota
- 1.2.2 List Share Types Available
- 1.2.3 Create a Share
- 1.2.4 Update a Share
- 1.2.5 Extend a Share
- 1.2.6 Shrink a Share
- 1.2.7 Add Access Rule through CLI
- 1.2.8 Remove Access Rule through CLI
- 1.2.9 Delete a Share
- 1.3 How to access a CephFS Share
- 1.1 Manila Commands
- 2 Using Shares on Kubernetes Clusters
Manila: Shared File Systems as a Service
OpenStack Manila provides services for managing shared file systems and is the third storage OpenStack component alongside Cinder and Swift. Manila provides a way to provision remote, sharable file systems (shares) which can be accessed by multiple instances simultaneously and file systems can have access rules assigned to control read-write and read-only access to the file system. On STFC Cloud, Manila provides a way to create Ceph file sytems (CephFS).
Manila Commands
In OpenStack Train, the Manila CLI is mainly used for handling shares. There are some limited commands in the OpenStack CLI for shares.
ย
To use Manila commands in OpenStack Train, the compatible version of the python package python-manilaclient needs to be installed. For OpenStack Train, this is v2.0.0. This can be installed using pip install python-manilaclient=2.0.0.
OpenStack CLI
In the OpenStack CLI the following commands are available:
openstack share --help
share create # create a new share
share delete # delete a share
share list # list all shares
share show # view details about a shareManila CLI
Manila provides a CLI for not only for managing shares, but also for viewing the share types available, managing access rules for a share, and viewing the quota for share in your project. Commands are run using manila <sub-command>. The next section will demonstrate how to do some of the Manila commands.
Usage
View Share Quota
You can view either the overall quota using manila quota-show, or the quota and the total quota used with manila absolute-limits.
$ manila quota-show
+-----------------------+----------------------------------+
| Property | Value |
+-----------------------+----------------------------------+
| gigabytes | 250 |
| id | PROJECT_ID |
| share_group_snapshots | 0 |
| share_groups | 5 |
| share_networks | 0 |
| shares | 10 |
| snapshot_gigabytes | 0 |
| snapshots | 0 |
+-----------------------+----------------------------------+
$ manila absolute-limits
+----------------------------+-------+
| Name | Value |
+----------------------------+-------+
| maxTotalShareGigabytes | 250 |
| maxTotalShareNetworks | 0 |
| maxTotalShareSnapshots | 0 |
| maxTotalShares | 10 |
| maxTotalSnapshotGigabytes | 250 |
| totalShareGigabytesUsed | 3 |
| totalShareNetworksUsed | 0 |
| totalShareSnapshotsUsed | 0 |
| totalSharesUsed | 3 |
| totalSnapshotGigabytesUsed | 0 |
+----------------------------+-------+
List Share Types Available
Share types available can be listed using manila share-type list or openstack share list. This will list the share types currently available and supported.
$ openstack share list
+--------------------------------------+-----------------------+------+-------------+-----------+-----------+-----------------+------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+-----------------------+------+-------------+-----------+-----------+-----------------+------+-------------------+
| a6b215a6-00c5-46a5-b1db-d86559097896 | test_share | 1 | CEPHFS | available | False | cephfs | | None |
| 7a1beb23-8dee-4709-9bcd-c947ae006653 | updated_demo_share | 1 | CEPHFS | available | False | cephfs | | nova |
+--------------------------------------+-----------------------+------+-------------+-----------+-----------+-----------------+------+-------------------+
Create a Share
Shares can be created by users using the manila create command. The required arguments are the share protocol and the size of the share in GiB.
Do not set a share to be public. This will make the share visible and accessible to everyone outside of the project.
$ manila create --help
usage: manila create [--snapshot-id <snapshot-id>] [--name <name>] [--metadata [<key=value> [<key=value> ...]]] [--share-network <network-info>] [--description <description>] [--share-type <share-type>]
[--public] [--availability-zone <availability-zone>] [--share-group <share-group>]
<share_protocol> <size>
Creates a new share (NFS, CIFS, CephFS, GlusterFS, HDFS or MAPRFS).
Positional arguments:
<share_protocol> Share protocol (NFS, CIFS, CephFS, GlusterFS, HDFS or MAPRFS).
<size> Share size in GiB.
Optional arguments:
--snapshot-id <snapshot-id>, --snapshot_id <snapshot-id>
Optional snapshot ID to create the share from. (Default=None)
--name <name> Optional share name. (Default=None)
--metadata [<key=value> [<key=value> ...]]
Metadata key=value pairs (Optional, Default=None).
--share-network <network-info>, --share_network <network-info>
Optional network info ID or name.
--description <description>
Optional share description. (Default=None)
--share-type <share-type>, --share_type <share-type>, --volume-type <share-type>, --volume_type <share-type>
Optional share type. Use of optional volume type is deprecated. (Default=None)
--public Level of visibility for share. Defines whether other tenants are able to see it or not. (Default=False)
--availability-zone <availability-zone>, --availability_zone <availability-zone>, --az <availability-zone>
Availability zone in which share should be created.
--share-group <share-group>, --share_group <share-group>, --group <share-group>
Optional share group name or ID in which to create the share (Experimental, Default=None).
So if we want to create a new CephFS share of size 1GiB, we can use the following command:
$ manila create --name demo_share --description "Demo creating a share" --share-type SHARE_TYPE_ID CephFS 1Which outputs:
+---------------------------------------+------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+------------------------------------------------------------------+
| status | creating |
| share_type_name | cephfs |
| description | Demo creating a share |
| availability_zone | None |
| share_network_id | None |
| share_group_id | None |
| revert_to_snapshot_support | False |
| access_rules_status | active |
| snapshot_id | None |
| create_share_from_snapshot_support | False |
| is_public | False |
| task_state | None |
| snapshot_support | False |
| id | 7a1beb23-8dee-4709-9bcd-c947ae006653 |
| size | 1 |
| source_share_group_snapshot_member_id | None |
| user_id | USER_ID |
| name | demo_share |
| share_type | SHARE_TYPE_ID |
| has_replicas | False |
| replication_type | None |
| created_at | 2022-10-19T15:03:28.000000 |
| share_proto | CEPHFS |
| mount_snapshot_support | False |
| project_id | PROJECT_ID |
| metadata | {} |
+---------------------------------------+------------------------------------------------------------------+
Update a Share
Once a share has been created in Manila, there are only three properties which can be updated: - Name of the share. - The description for the share. - Change the visibility of the share to public or private.
$ manila update --help
usage: manila update [--name <name>] [--description <description>] [--is-public <is_public>] <share>
Rename a share.
Positional arguments:
<share> Name or ID of the share to rename.
Optional arguments:
--name <name> New name for the share.
--description <description>
Optional share description. (Default=None)
--is-public <is_public>, --is_public <is_public>
Public share is visible for all tenants.
For example, we can update the name of a share from demo_share to updated_demo_share in the following way:
$ manila update --name updated_demo_share demo_shareThen we can see the updated share in the list of shares in the current project:
$ manila list
+--------------------------------------+-----------------------+------+-------------+-----------+-----------+-----------------+------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+-----------------------+------+-------------+-----------+-----------+-----------------+------+-------------------+
| 7a1beb23-8dee-4709-9bcd-c947ae006653 | updated_demo_share | 1 | CEPHFS | available | False | cephfs | | nova |
+--------------------------------------+-----------------------+------+-------------+-----------+-----------+-----------------+------+-------------------+
Extend a Share
The size of a share can be increased using the manila extend command.
$ manila extend --help
usage: manila extend <share> <new_size>
Increases the size of an existing share.
Positional arguments:
<share> Name or ID of share to extend.
<new_size> New size of share, in GiBs.
For example, if we want to extend a demo share from 1GiBs to 2GiBs, we can do the following:
$ manila extend updated_demo_share 2Viewing the details of the share we can see that the size of the share has been updated.
$ manila show updated_demo_share
+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| status | available |
| share_type_name | cephfs |
| description | Demo creating a share |
| availability_zone | nova |
| share_network_id | None |
| share_group_id | None |
| revert_to_snapshot_support | False |
| access_rules_status | active |
| snapshot_id | None |
| create_share_from_snapshot_support | False |
| is_public | False |
| task_state | None |
| snapshot_support | False |
| id | 7a1beb23-8dee-4709-9bcd-c947ae006653 |
| size | 2 |
| source_share_group_snapshot_member_id | None |
| user_id | USER_ID |
| name | updated_demo_share |
| share_type | SHARE_TYPE |
| has_replicas | False |
| replication_type | None |
| created_at | 2022-10-19T15:03:28.000000 |
| share_proto | CEPHFS |
| mount_snapshot_support | False |
| project_id | PROJECT_ID |
| metadata | {} |
| export_locations | |
| | path = EXPORT_PATH |
| | id = EXPORT_LOCATIONS_ID |
| | preferred = False |
+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
Shrink a Share
This can only be done through the command line only.
The size of a share can be reduced using the manila shrink command.
$ manila shrink --help
usage: manila shrink <share> <new_size>
Decreases the size of an existing share.
Positional arguments:
<share> Name or ID of share to shrink.
<new_size> New size of share, in GiBs.
Using the example in the previous section, we can reduce the size of a share from 2GiB to 1Gib using:
$ manila shrink updated_demo_share 1We can see using manila show updated_demo_share that the size of the share has been updated:
$ manila show updated_demo_share
+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| status | available |
| share_type_name | cephfs |
| description | Demo creating a share |
| availability_zone | nova |
| share_network_id | None |
| share_group_id | None |
| revert_to_snapshot_support | False |
| access_rules_status | active |
| snapshot_id | None |
| create_share_from_snapshot_support | False |
| is_public | False |
| task_state | None |
| snapshot_support | False |
| id | 7a1beb23-8dee-4709-9bcd-c947ae006653 |
| size | 1 |
| source_share_group_snapshot_member_id | None |
| user_id | USER_ID |
| name | updated_demo_share |
| share_type | SHARE_TYPE |
| has_replicas | False |
| replication_type | None |
| created_at | 2022-10-19T15:03:28.000000 |
| share_proto | CEPHFS |
| mount_snapshot_support | False |
| project_id | PROJECT_ID |
| metadata | {} |
| export_locations | |
| | path = EXPORT_PATH |
| | id = EXPORT_LOCATIONS_ID |
| | preferred = False |
+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
Add Access Rule through CLI
Access rules can be created for a share through the Web UI or on the command line. You can allow either read-write (RW) or read-only (RO) access to the share.
Only cephx access rules can be used for shares as only CephFS shares are currently supported. Any other type of access rule created will go into error state.
To create a new access rule for a share, e.g. a share named demo_share, we can use the manila access-allow command:
manila access-allow demo_share cephx aliceWith the arguments required described below:
$ manila access-allow --help
usage: manila access-allow [--access-level <access_level>] [--metadata [<key=value> [<key=value> ...]]] <share> <access_type> <access_to>
Allow access to a given share.
Positional arguments:
<share> Name or ID of the NAS share to modify.
<access_type> Access rule type (only "ip", "user"(user or group), "cert" or "cephx" are supported).
<access_to> Value that defines access.
Optional arguments:
--access-level <access_level>, --access_level <access_level>
Share access level ("rw" and "ro" access levels are supported). Defaults to rw.
--metadata [<key=value> [<key=value> ...]]
Space Separated list of key=value pairs of metadata items. OPTIONAL: Default=None.
ย
Here, access_to refers to the name given to the cephx access rule. This could be the name of a user to give share access to for example. The value given for access_to is used in later examples demonstrating how to mount a share.
Then we can view the access rules for the share using:
$ manila access-list demo_share
+--------------------------------------+-------------+----------------+--------------+--------+------------------------------------------+----------------------------+----------------------------+
| id | access_type | access_to | access_level | state | access_key | created_at | updated_at |
+--------------------------------------+-------------+----------------+--------------+--------+------------------------------------------+----------------------------+----------------------------+
| 56907a0c-024a-465e-8ebc-5a0b085ac87b | cephx | alice | rw | active | ACCESS_KEY | 2022-10-14T14:55:59.000000 | 2022-10-14T14:55:59.000000 |
+--------------------------------------+-------------+----------------+--------------+--------+------------------------------------------+----------------------------+----------------------------+Remove Access Rule through CLI
Access rules can be removed from a share using manila access-deny:
$ manila access-deny --help
usage: manila access-deny <share> <id>
Deny access to a share.
Positional arguments:
<share> Name or ID of the NAS share to modify.
<id> ID of the access rule to be deleted.Delete a Share
A share can be deleted by using the manila delete command:
$ manila delete --help
usage: manila delete [--share-group <share-group>] <share> [<share> ...]
Remove one or more shares.
Positional arguments:
<share> Name or ID of the share(s).
Optional arguments:
--share-group <share-group>, --share_group <share-group>, --group <share-group>
Optional share group name or ID which contains the share (Experimental, Default=None).How to access a CephFS Share
There are two methods that can be used to mount a share onto a VM:
ย
The kernel method is preferred as it uses significantly less CPU marshalling data through userspace.
The following examples assume that the following have been set up:
VM (examples use an Ubuntu VM)
CephFS Share and have the export path given in the share details copied to use
cephx share access rules
Monitor IP addresses
MON_HOST_IP_ADDRESSES can be found by doing manila show <share ID or name> under the export loction section
Access a share using the Kernel Client
The kernel client requires the ceph-common package to be installed. To access the filesystem on a share, we can use the mount command of the form:
mkdir /mnt/share # or any path you prefer
# breakdown of the command - all remaining lines are run as a single command without comments
mount -t ceph
# The output of the Monitor IP address from the service endpoint
{mon1 ip addr}:6789,{mon2 ip addr}:6789,{mon3 ip addr}:6789/{manilla-volume}
# name of the access rule and the secret associated to it
-o name={access-rule-name},secret={access-key}
# Existing path to attach the share to on the VM
/mnt/pathNote: This is run as a single command, it has been broken down for readability in the documentation
The directory will be owned by root by default. To allow another user to read and write add the appropriate user and group like so:
chown user:user /mnt/pathAccess a share using the FUSE Client
In order to use the FUSE client, make sure the version of the ceph-fuse package is at least octopus. The version of the package can be found using:
~$ ceph-fuse --version
ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
To use the FUSE client, we need to set up a keyring and a ceph.conf file. If we have named the cephx rule created with the name alice, then the keyring file will need to be named alice.keyring and needs to contain the following:
[client.alice]
key = CEPHX_KEYWhere CEPHX_KEY is the secret/access_key for the specific access rule, which can be found using manila access-list <share ID or name>
Then for the ceph.conf file, we need to add the IP addresses for the monitors from the ceph cluster. The IP address for the ceph monitors for the share can be found using manila show <share-id> under the export locations field.
[client]
client quota = True
mon host = MON_HOST_IP_ADDRESSESWhere MON_HOST_IP_ADDRESSES can be found by doing manila show <share ID or name> under the export loction section
Then the filesystem can be mounted to a test directory using the alice cephx access rule:
sudo ceph-fuse ~/mnt \
--id=alice \
--conf=/path/to/ceph.conf \
--keyring=/path/to/alice.keyring \
--client-mountpoint={mount-point}
Where the mount point is the path to the share on the ceph cluster. The share path can be found using manila show <share-id> under the export locations field, and it looks like /volumes/_nogroup/abc123-efd
Using Shares on Kubernetes Clusters
OpenStack Manila is currently in technical preview on STFC Cloud. If you have any feedback or suggestions, please sent it to cloud-support@stfc.ac.uk
This feature for using Shares with Kubernetes is also currently being tested by the Cloud Team.
To use shares on Kubernetes clusters we need the following csi installed and deployed onto the cluster:
The manila-csi is designed to be lightweight and only handles the interaction between Kubernetes and OpenStack to create, delete, and access shares. The ceph-csi handles the CephFS side of the filesystem on the Kubernetes cluster. See https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/manila-csi-plugin/developers-csi-manila.md#notes-on-design for more information on how manila-csi is designed.
Deploying onto the Cluster
Both the manila-csi and ceph-csi can be deployed to a Kubernetes cluster using a Helm chart.
ceph-csi
The ceph-csi can be deployed in the following way:
# Add the chart repository
helm repo add ceph-csi https://ceph.github.io/csi-charts
# Install the chart
helm install --create-namespace --namespace "ceph-csi-cephfs" "ceph-csi-cephfs" ceph-csi/ceph-csi-cephfs
You should then be able to see the following pods in the ceph-csi-cephfs namespace:
$ kubectl get pods -n ceph-csi-cephfs
NAME READY STATUS RESTARTS AGE
ceph-csi-cephfs-nodeplugin-6554g 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-6xcfc 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-ddt84 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-h97j5 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-hxj95 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-j65fp 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-ltt7q 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-mhwr8 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-wrjsk 3/3 Running 0 17d
ceph-csi-cephfs-nodeplugin-zfl8w 3/3 Running 0 17d
ceph-csi-cephfs-provisioner-765854df4c-csdft 5/5 Running 0 17d
ceph-csi-cephfs-provisioner-765854df4c-nzp77 5/5 Running 0 17d
ceph-csi-cephfs-provisioner-765854df4c-qznz4 5/5 Running 0 17d
manila-csi
Before deploying the Helm chart for manila-csi, we need to create a manila-csi-values.yaml file in order to deploy the csi to support CephFS shares:
# Enabled Manila share protocols
shareProtocols:
- protocolSelector: CEPHFS
fsGroupPolicy: None
fwdNodePluginEndpoint:
dir: /var/lib/kubelet/plugins/cephfs.csi.ceph.com
sockFile: csi.sock
Then we can deploy the chart:
# add helm repo
helm repo add cpo https://kubernetes.github.io/cloud-provider-openstack
helm repo update
# install helm chart with values.yaml file 'manila-csi-values.yaml'
helm install manila-csi --create-namespace --namespace manila-csi -f manila-csi-values.yaml cpo/openstack-manila-csiThen you should see pods similar to the following being created onto the cluster (for this example the chart has been deployed into the default namespace):
$ kubectl get pod -n manila-csi
NAME READY STATUS RESTARTS AGE
manila-csi-openstack-manila-csi-controllerplugin-0 4/4 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-6plbt 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-9dxrl 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-cjhvp 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-f72s9 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-ghfmx 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-h99sx 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-k5g9k 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-mlvpp 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-qpd9m 2/2 Running 0 14d
manila-csi-openstack-manila-csi-nodeplugin-slktx 2/2 Running 0 14d
Next step is to manually create the secret for the manila-csi to use in order to interact with OpenStack:
Managing Secrets using kubectl
Create an application credential for the associated account ( Application Credentials )
Copy this template:
# manila_csi_secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: csi-manila-secret
namespace: kube-system
stringData:
os-authURL: "https://openstack.stfc.ac.uk:5000/v3"
os-region: "RegionOne"
os-applicationCredentialID: "<credential_id>"
os-applicationCredentialSecret: "<secret>"Create the secret:
kubectl apply -f manila_csi_secret.yaml
Enabling Dynamic Shares using the storage class
Shares can also be created on demand and attached to specific pods.
First, a storage class needs to be created:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-manila-cephfs
provisioner: cephfs.manila.csi.openstack.org
allowVolumeExpansion: true
parameters:
type: cephfs
csi.storage.k8s.io/provisioner-secret-name: csi-manila-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
csi.storage.k8s.io/controller-expand-secret-name: csi-manila-secret
csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
csi.storage.k8s.io/node-stage-secret-name: csi-manila-secret
csi.storage.k8s.io/node-stage-secret-namespace: kube-system
csi.storage.k8s.io/node-publish-secret-name: csi-manila-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-systemThen the storage class can be created from this template using:
kubectl apply -f storageclass.yamlNext, a persistent volume claim needs to be created in order to define the size of the share and the access mode.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: new-cephfs-share-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: csi-manila-cephfsTo create the pvc:
kubectl apply -f dynamic_pvc.yamlThen, for example, if we want to have an nginx pod spun up with a new share attached we can use the following template:
apiVersion: v1
kind: Pod
metadata:
name: new-cephfs-share-pod
spec:
containers:
- name: web-server
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: mypvc
mountPath: /var/lib/www
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: new-cephfs-share-pvc
readOnly: falseThen we can create the pod and inspect it:
$ kubectl apply -f pod.yaml
$ kubectl describe pod new-cephfs-share-pod
Name: new-cephfs-share-pod
Namespace: default
Priority: 0
Service Account: default
Node: cloud-dev-md-nano-fxhmg/10.6.0.204
Start Time: Thu, 02 Feb 2023 14:21:56 +0000
Labels: <none>
Annotations: cni.projectcalico.org/containerID: d0741d0c1f5b8c4733c3a7c090dbaef9e6875508ab074c36cc671bfa51479494
cni.projectcalico.org/podIP: 192.168.232.10/32
cni.projectcalico.org/podIPs: 192.168.232.10/32
Status: Running
IP: 192.168.232.10
IPs:
IP: 192.168.232.10
Containers:
web-server:
Container ID: containerd://12a76ece022f9cbc45b6d9af3e2c0efa38dafb4734a52dcdbceaa4f63caf174e
Image: nginx
Image ID: docker.io/library/nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 02 Feb 2023 14:22:10 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/lib/www from mypvc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6l758 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
mypvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: new-cephfs-share-pvc
ReadOnly: false
kube-api-access-6l758:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
In the OpenStack project where the application credential was created for, we should now see a new share has been created with a cephx access rule.
Examples: Provisioning Manila Shares onto the Cluster
The following examples are following the examples from cloud-provider-openstack/examples/manila-csi-plugin at master ยท kubernetes/cloud-provider-openstack
Static (Manual) Share Provisioning
If a default storage class has been set up in the cluster, then storageClass: "" needs to be included in the pvc.yaml file. Otherwise, the cluster will default to attempting to make a pvc based on the clusterโs default storage class.
You will need the ID for the Share as well as the ID for the access rule the PV should use. See share access list for the getting the ID for the access rule.
Create a persistent volume claim (PVC) and persistent volume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: preprovisioned-cephfs-share
labels:
name: preprovisioned-cephfs-share
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
csi:
driver: cephfs.manila.csi.openstack.org
volumeHandle: preprovisioned-cephfs-share
nodeStageSecretRef:
name: csi-manila-secret
namespace: kube-system
nodePublishSecretRef:
name: csi-manila-secret
namespace: kube-system
volumeAttributes:
shareID: <share-id> # the share ID of the share you want to use as the persistent volume in the cluster
shareAccessID: <access-rule-id> # the ID of a cephx access rule created for the share.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: existing-cephfs-share-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: "" # to override any default storageclass in cluster
resources:
requests:
storage: 1Gi
selector:
matchExpressions:
- key: name
operator: In
values: ["preprovisioned-cephfs-share"]
Note
The ID of access rules for a specific share can be found using manila access-list <share-name/ID>. This will list all the access rules for a given share and include the ID for each rule. Access rules can be managed in the web UI or using the CLI. For more information on how to create a access rule using the CLI, see Manila | Add Access Rule through CLI
You should be able to create the persistent volume claim created successfully here:
kubectl apply -f pvc.yamlCheck if it was successful:
kubectl get pvcWhich should return:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
existing-cephfs-share-pvc Bound preprovisioned-cephfs-share 1Gi RWX 19hTo get the information on the existing persistent volume:
kubectl describe pvc existing-cephfs-share-pvcWhich should return:
Name: existing-cephfs-share-pvc
Namespace: default
StorageClass:
Status: Bound
Volume: preprovisioned-cephfs-share
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWX
VolumeMode: Filesystem
Used By: existing-cephfs-share-pod
Events: <none>List the persistent volumes:
kubectl get pvWhich returns:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
preprovisioned-cephfs-share 1Gi RWX Retain Bound default/existing-cephfs-share-pvc 19hGet information on the pre-provisioned share:
kubectl describe pv preprovisioned-cephfs-shareWhich returns:
Name: preprovisioned-cephfs-share
Labels: name=preprovisioned-cephfs-share
Annotations: pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Bound
Claim: default/existing-cephfs-share-pvc
Reclaim Policy: Retain
Access Modes: RWX
VolumeMode: Filesystem
Capacity: 1Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: cephfs.manila.csi.openstack.org
FSType:
VolumeHandle: preprovisioned-cephfs-share
ReadOnly: false
VolumeAttributes: shareAccessID=SHARE_ACCESS_RULE_ID
shareID=SHARE_ID
Events: <none>References:
https://docs.openstack.org/manila/train/user/create-and-manage-shares.html
https://docs.openstack.org/manila/train/admin/cephfs_driver.html