Echo to Tape Software

Owned by Alex Kemp
Last updated: Jun 05, 2024
2 min read

The end location of the data should be the tape drive, so some software is needed to move it from Echo S3 to the tape drives.

Currently, all data goes through the file aggregator, storageD.

image-20240416-191552.png

This means that, if we were to transfer the data to tape without using storageD, The IDS wouldn’t know about the newly uploaded files and wouldn’t be able to download them.

A separate piece of software, in addition to the upload api, is needed to make the transfer.

The following 2 designs have been considered.

Architectural Design A

Here, the data will need to be downloaded to local storage so the StorageD client can pick it up. This has a high network overhead but has the advantage of being relatively easy to develop.

 

upload architecture-Page-1.png

Architectural Design B

This design explores the possibility of the storageD server downloading the data directly. This would save a download step. However, it will mean a change to the storageD code which may be problematic due to the state of the code.

Solution

After a chat with @Alan Kyffin, we decided to go with design A because it is simpler to develop and maintain.

  • the code isn’t in a git repo

  • isn't in great shape

  • is being looked after by another team

  • is currently being updated from python2 to 3

The code was put in a GitLab repo for this review.