Deploying OpenSearch monitoring for Rucio

This document will describe the process of setting up OpenSearch monitoring for Rucio. It will include all the steps from deploying the OpenSearch cluster, setting up the indexes, directing Hermes to forward messages to the index, and creating the dashboards for the data.

 Instructions

1. Deploy OpenSearch
   Deploying OpenSearch is going to be simplified by the new Rucio Helm chart, until then this link takes you to a GitLab Repo that has the instructions for deploying OpenSearch

   1. Download the Repo

   2. Configure the values for your own needs - default settings in the values are for 3 master nodes with 3Gi Storage, and 3 Worker nodes with 30Gi Storage

   3. Run the following in a location that has helm and access to your kubernetes cluster and it should deploy OpenSearch to your cluster in the namespace elastic-system

      make opensearch-deployment

2. Get access to OpenSearch

   1. Ensure you have the correct credentials by running the following command in one terminal

      kubectl port-forward -n opensearch-system svc/opensearch-cluster-worker 9200

   2. In another terminal run, the default username and password is admin:

      curl -u "<opensearch username>:<OpenSearch password>" -k "https://localhost:9200"

   3. Should all be correct you will get a response that shows:

3. Create index in OpenSearch
   Within the Repo from step 1 at overlays/Elastic/OpenSearch/rucio-events contains the index, this needs to be applied to OpenSearch in the location you want the index to be created

4. Direct Rucio Hermes to Index
   OpenSearch is now setup to receive messages from Rucio. It now needs to have Rucio Hermes daemon pointed at it. For this editing of the Rucio values will allow for such thing.

   1. You will need at least this in the config section of the daemon values

   2. You will also need to create a secret in the Rucio namespace to allow the injection of the OpenSearch user secret for Hermes to be able to deposit the messages

   3. Add the secret to Hermes, an example is below

5. Connect or get connected OpenSearch to Grafana
   Within Grafana now that Rucio is now communicating its logs to OpenSearch it now needs to be able to be visualised using Grafana

   1. In your clusters Grafana deployment go to configuration / datasources

   2. Add data source

   3. Select OpenSearch

   4. Put in the following details in the fields:

6. Create the dashboard

   1. Once the data source is created and verified you can create a dashboard using it

   2. Navigate to Dashboards / Import

   3. Paste in the contents of overlays/Elastic/OpenSearch/Dashboards/Rucio External into the import via panel json

 Related articles