Rucio Operations and WebUI Development

Owned by Timothy Noble
Last updated: Feb 20, 2023
3 min read

Project Summary:

Dates

September 2022 - March 2023

Participant

George Matthews

Objectives

  • Get Graduate to perform Rucio maintenance, check metrics, on board users

  • Improve Rucio - Work towards replacing the VM Rucio with K8S deployed Rucio

  • Ensure new WebUI is Multi-VO capable

 

Work Done

  • Knowledge of User creation

    • With this knowledge and use in M-VO Rucio found an issue

  • Used monitoring to keep Rucio running at a high percentage of the time

  • Onboarded self to a VO

  • Installed and configured ArgoCD for GitOps K8S workflows

  • Installed and configured Vault for cluster secret management

  • Wrote documentation for using vault with applications to ensure secrets are injected correctly

  • Converted functional test script to a K8S deployable cronjob

  • Investigated using Rucio daemons to automate the cronjob (possible but require Rucio 1.30 or later)

  • Got OLD webUI (mostly) working

  • Made WebUI 2.0 from the GSoC student Multi-VO capable

  • Found bug that caused DDoS attack from Rucio WebUI on Server, and put in a fix

  • Investigating and working on getting Multi-VO FTS delegation script

  • Learned new architecture for WebUI that meant a whole re-write

  • Deployed Multi-VO fix for new architecture WebUI

  •  

 

Original Project Proposal

Group

Data Services

Project name

Rucio Operation and WebUI development

Project LM

Timothy Noble

Project code/Task

STAK00024 02.03 (maybe specific EGI code to be set up)

Other resources

Rucio Developers at CERN, Data Services group.

Potential for travel to: CERN and/or Rucio-related workshops funding is available through the project.

Project Summary

Rucio is a distributed data management service that is utilised by ATLAS and CMS experiments at CERN to manage the huge volumes of data. At RAL we have a Rucio instance that is designed to facilitate data management for many smaller experiments. This Multi-Experiment Rucio requires additional development on top of the baseline development work so that features that are designed for Rucio (usually deployed for a single experiment) can function correctly for the Multi-Experiment Rucio deployment.

This project is to work on a newly developed WebUI for the Rucio software that will complement the current CLI, which will lower the barrier for entry to use Rucio. The ‘WebUI 2.0’ is currently being reworked by the Rucio developers at CERN, and RAL will ensure it is Multi-Experiment optimised.

As well as the development work on the WebUI, routine Rucio Service management jobs will also be undertaken, which will include ensuring the service is running normally, processing tickets, interacting with users, troubleshooting any operational issues, and updating the Rucio instance.

Project Outputs

·       Multi-Experiment capable Rucio WebUI (committed to project github)

·       Documentation on the code, setup and deployment of the Multi-Experiment Rucio WebUI

·       Deployment of the Multi-Experiment capable Rucio WebUI on the RAL instance of Rucio.

Skills and Expertise graduate will gain

·       Python and WebUI development work

 

·       Experience testing and deploying their code onto a production service.

 

·       Experience working as part of a large international collaboration.

 

 

·       Knowledge of the use and management of Rucio, as well as the software that integrates with Rucio (e.g. FTS)

 

·       Operational experience in maintaining a service

 

·       Development of WebUI elements

 

·       Knowledge of PostgreSQL databases.

 

 

Exit plan when Graduate moves to different project

Documentation, plans and details of any unfinished development are to be passed on to the Rucio Service manager. The WebUI element is an existing feature therefore there is no additional work beyond the current Rucio service management.