...
Once the user has run their experiment, the data is stored on the internal DLC data store for 40 days. The metadata is then pushed to ICAT’s Oracle database, and the raw data is sent to the data centre for processing.
Data Ingestion
The process starts with the XML client (depicted below) transmitting file metadata to the File Aggregator. This undertakes the task of segmenting the incoming data into discrete chunks. Subsequently, it orchestrates requests to the Diamond file client, procuring these segmented portions. The metadata encapsulates pertinent details such as file chunk attributes, encompassing size and storage coordinates, and is stored within the Storage D component. This component is used to look up the location of the data for retrieval purposes. These segmented file fragments are then passed to a tape archive, facilitated by the CERN archival storage system. This archival strategy is employed for preserving data over extended durations, typically decades.
...
However, internal agreement on architecture has proven more complicated than anticipated. In order to progress with the project, the files will be stored in object storage in the interim, and the new, purpose-built application will be updated at a later date once a consensus has been agreed upon between the various departments. MinIO has been chosen as the S3 object store client. This is due to the fact it is open-source, free to use and part of the Cloud Native Computing Foundation.