Project Summary:
Dates
September 2022 - March 2023
Participant
George Matthews
Objectives
Get Graduate to perform Rucio maintenance, check metrics, on board users
Improve Rucio - Work towards replacing the VM Rucio with K8S deployed Rucio
Ensure new WebUI is Multi-VO capable
Work Done
Knowledge of User creation
With this knowledge and use in M-VO Rucio found an issue
Used monitoring to keep Rucio running at a high percentage of the time
Onboarded self to a VO
Installed and configured ArgoCD for GitOps K8S workflows
Installed and configured Vault for cluster secret management
Wrote documentation for using vault with applications to ensure secrets are injected correctly
Converted functional test script to a K8S deployable cronjob
Investigated using Rucio daemons to automate the cronjob (possible but require Rucio 1.30 or later)
Got OLD webUI (mostly) working
Made WebUI 2.0 from the GSoC student Multi-VO capable
Found bug that caused DDoS attack from Rucio WebUI on Server, and put in a fix
Investigating and working on getting Multi-VO FTS delegation script
Learned new architecture for WebUI that meant a whole re-write
Deployed Multi-VO fix for new architecture WebUI
Original Project Proposal
Group | Data Services |
Project name | Rucio Operation and WebUI development |
Project LM | Timothy Noble |
Project code/Task | STAK00024 02.03 (maybe specific EGI code to be set up) |
Other resources | Rucio Developers at CERN, Data Services group. Potential for travel to: CERN and/or Rucio-related workshops funding is available through the project. |
Project Summary | Rucio is a distributed data management service that is utilised by ATLAS and CMS experiments at CERN to manage the huge volumes of data. At RAL we have a Rucio instance that is designed to facilitate data management for many smaller experiments. This Multi-Experiment Rucio requires additional development on top of the baseline development work so that features that are designed for Rucio (usually deployed for a single experiment) can function correctly for the Multi-Experiment Rucio deployment. This project is to work on a newly developed WebUI for the Rucio software that will complement the current CLI, which will lower the barrier for entry to use Rucio. The ‘WebUI 2.0’ is currently being reworked by the Rucio developers at CERN, and RAL will ensure it is Multi-Experiment optimised. As well as the development work on the WebUI, routine Rucio Service management jobs will also be undertaken, which will include ensuring the service is running normally, processing tickets, interacting with users, troubleshooting any operational issues, and updating the Rucio instance. |
Project Outputs | · Multi-Experiment capable Rucio WebUI (committed to project github) · Documentation on the code, setup and deployment of the Multi-Experiment Rucio WebUI · Deployment of the Multi-Experiment capable Rucio WebUI on the RAL instance of Rucio. |
Skills and Expertise graduate will gain | · Python and WebUI development work
· Experience testing and deploying their code onto a production service.
· Experience working as part of a large international collaboration.
· Knowledge of the use and management of Rucio, as well as the software that integrates with Rucio (e.g. FTS)
· Operational experience in maintaining a service
· Development of WebUI elements
· Knowledge of PostgreSQL databases.
|
Exit plan when Graduate moves to different project | Documentation, plans and details of any unfinished development are to be passed on to the Rucio Service manager. The WebUI element is an existing feature therefore there is no additional work beyond the current Rucio service management. |