Deploying ElasticSearch monitoring for Rucio

This document will describe the process of setting up elasticsearch monitoring for Rucio. It will include all the steps from deploying the elasticsearch cluster, setting up the indexes, directing Hermes to forward messages to the index, and creating the dashboards for the data.

 Instructions

1. Deploy Elasticsearch
   Deploying Elasticsearch is going to be simplified by the new Rucio Helm chart, until then this link takes you to a GitLab Repo that has the instructions for deploying Elasticsearch

   1. Download the Repo

   2. Run the following in a location that has helm and access to your kubernetes cluster and it should deploy Elasticsearch to your cluster in the namespace elastic-system

      make elastic-deployment

2. Get access to Elasticsearch

   1. Recover the Elasticsearch password for the Elasticsearch deployment from the secret created during the deployment

      kubectl get secret elasticsearch-eck-elasticsearch-es-elastic-user kubectl get secret -n elastic-system elasticsearch-eck-elasticsearch-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'

   2. Ensure you have the correct credentials by running the following command in one terminal

      kubectl port-forward -n elastic-system svc/elasticsearch-eck-elasticsearch-es-worker 9200

   3. In another terminal run

   4. Should all be correct you will get a response that shows:

3. Create index in Elasticsearch
   Within the Repo from step 1 at overlays/Elastic/elasticsearch/rucio-events contains the index, this needs to be applied to Elasticsearch in the location you want the index to be created

4. Direct Rucio Hermes to Index
   Elasticsearch is now setup to receive messages from Rucio. It now needs to have Rucio Hermes daemon pointed at it. For this editing of the Rucio values will allow for such thing.

   1. You will need at least this in the config section of the daemon values

   2. You will also need to create a secret in the Rucio namespace to allow the injection of the Elasticsearch user secret for Hermes to be able to deposit the messages

   3. Add the secret to Hermes, an example is below

5. Connect or get connected Elasticsearch to Grafana
   Within Grafana now that Rucio is now communicating its logs to Elasticsearch it now needs to be able to be visualised using Grafana

   1. In your clusters Grafana deployment go to configuration / datasources

   2. Add data source

   3. Select Elasticsearch

   4. Put in the following details in the fields:

6. Create the dashboard

   1. Once the data source is created and verified you can create a dashboard using it

   2. Navigate to Dashboards / Import

   3. Paste in the contents of overlays/Elastic/elasticsearch/Dashboards/Rucio External into the import via panel json

 Related articles