Workflows strategy meeting

Owned by Alex Kemp, created
Last updated: Aug 06, 2024
1 min read

 Date

Aug 1, 2024

 Participants

  • @Alex Kemp

  • @Antony Wilson

  • @Kevin Phipps

  • @Louise Davies

 Goals

  • To figure out what it is we can offer to the various facilities given our capabilities

 Discussion topics

  • Things we are NOT doing:

    • developing workflows

    • running instances of ARGO

  • What we can do is:

    • store the provenance of proceed data from automated workflows

    • help set up a common repository to facilitate the reuse of common workflows

  • There seem to be 2 types of workflows:

    • facility ran workflows such as auto-reduction

    • user-defined workflows which happen on an ad-hoc basis

  • Agreed that we should tackle facility workflows first

  • We don't know how Diamond runs its workflows and where they sit in the ingestion process

  • Decided that there was no point in re-inventing the wheel in creating a custom repo for workflows, we should look into workflowHub

  • We should be able to store provenance data in ICAT & the schema may need to be changed.

  • The WHY (why are we doing this) is making sure facility workflow data is FAIR

  • Processed vs Raw data:

    • ISIS only stores raw data in the catalogue (ICAT)

    • Diamond stores raw and processed data but doesn’t distinguish between them in the catalogue (ICAT)

 Action items

@Alex Kemp to sort a meeting with Diamond to understand how they run workflows & what they are currently working on.
@Alex Kemp Investigate Workflow hub and assess roles and permissions

 Decisions

  1. At this stage, all we can offer is the ability to store provenance data in ICAT