Deploying ElasticSearch monitoring for Rucio

This document will describe the process of setting up elasticsearch monitoring for Rucio. It will include all the steps from deploying the elasticsearch cluster, setting up the indexes, directing Hermes to forward messages to the index, and creating the dashboards for the data.

 Instructions

1. Deploy Elasticsearch
   Deploying Elasticsearch is going to be simplified by the new Rucio Helm chart, until then this link takes you to a GitLab Repo that has the instructions for deploying Elasticsearch

   1. Download the Repo

   2. Run the following in a location that has helm and access to your kubernetes cluster and it should deploy Elasticsearch to your cluster in the namespace elastic-system

      make elastic-deployment

2. Get access to Elasticsearch

   1. Recover the Elasticsearch password for the Elasticsearch deployment from the secret created during the deployment

      kubectl get secret elasticsearch-eck-elasticsearch-es-elastic-user kubectl get secret -n elastic-system elasticsearch-eck-elasticsearch-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'

   2. Ensure you have the correct credentials by running the following command in one terminal

      kubectl port-forward -n elastic-system svc/elasticsearch-eck-elasticsearch-es-worker 9200

   3. In another terminal run

      curl -u "elastic:<Elasticsearch password>" -k "https://localhost:9200"

   4. Should all be correct you will get a response that shows:

      { "name" : "elasticsearch-eck-elasticsearch-es-worker-0", "cluster_name" : "elasticsearch-eck-elasticsearch", "cluster_uuid" : "wz8ZdQf1RjqHA68Ba8WzLA", "version" : { "number" : "8.11.0", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "d9ec3fa628c7b0ba3d25692e277ba26814820b20", "build_date" : "2023-11-04T10:04:57.184859352Z", "build_snapshot" : false, "lucene_version" : "9.8.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }

3. Create index in Elasticsearch
   Within the Repo from step 1 at overlays/Elastic/elasticsearch/rucio-events contains the index, this needs to be applied to Elasticsearch in the location you want the index to be created

   curl -u "elastic:<Elasticsearch password>" -XPUT "https://localhost:9200/<you Index>/?&pretty" -H "Content-Type: application/json" -d @rucio-events -k

4. Direct Rucio Hermes to Index
   Elasticsearch is now setup to receive messages from Rucio. It now needs to have Rucio Hermes daemon pointed at it. For this editing of the Rucio values will allow for such thing.

   1. You will need at least this in the config section of the daemon values

      hermes: services_list: "elastic" elastic_endpoint: "http://elasticsearch-eck-elasticsearch-es-worker.elastic-system.svc.cluster.local:9200/<your index>/_bulk"

   2. You will also need to create a secret in the Rucio namespace to allow the injection of the Elasticsearch user secret for Hermes to be able to deposit the messages

      kubectl create secret generic <daemons deploymentname>-elastic-secrets --namespace <Rucio namespace> --from-literal=USERNAME=elastic --from-literal=PASSWORD=<Elasticsearch password>

   3. Add the secret to Hermes, an example is below

      hermes: threads: 1 podAnnotations: {} bulk: 1000 resources: limits: memory: "600Mi" cpu: "210m" requests: memory: "300Mi" cpu: "140m" additionalEnvs: - name: RUCIO_CFG_DATABASE_DEFAULT valueFrom: secretKeyRef: name: rucio-database04 key: DEFAULT - name: RUCIO_CFG_HERMES_ELASTIC_USERNAME valueFrom: secretKeyRef: name: elastic-secrets key: USERNAME - name: RUCIO_CFG_HERMES_ELASTIC_PASSWORD valueFrom: secretKeyRef: name: elastic-secrets key: PASSWORD

5. Connect or get connected Elasticsearch to Grafana
   Within Grafana now that Rucio is now communicating its logs to Elasticsearch it now needs to be able to be visualised using Grafana

   1. In your clusters Grafana deployment go to configuration / datasources

   2. Add data source

   3. Select Elasticsearch

   4. Put in the following details in the fields:

      URL: http://elasticsearch-eck-elasticsearch-es-worker.elastic-system.svc.cluster.local:9200 Basic auth: True User: elastic Password: <Elasticsearch password> Index name: <your Index> Pattern: No pattern Time field name: create_at ElasticSearch version 8.0+ Max concurrent Shard Requests: 3 # If you have used the base repo settings

6. Create the dashboard

   1. Once the data source is created and verified you can create a dashboard using it

   2. Navigate to Dashboards / Import

   3. Paste in the contents of overlays/Elastic/elasticsearch/Dashboards/Rucio External into the import via panel json

 Related articles