Automate streamflow run process

Action items:

• create a project in rke2-1-dev-cineca for @vgeo
• @vgeo should familiarize with https://github.com/alpha-unito/streamflow , specifically https://github.com/alpha-unito/streamflow/tree/master/docs/source/cwl and all streamflow/cwl related material found in repo
• @vgeo hands-on/ test https://gitlab.ebrains.eu/ri/tech-hub/platform/workflows/streamflow-examples on k8s
• convert CWL workflow (<- this is what we ask from EBRAINS users) into streamflow.yml format (<- this is what steramflow expects) to run via streamflow in k8s
• use data-proxy for fetching the "input" files
• use data-proxy for uploading "output" files in object storage

Prior steps made by Nikos and Michalis:

• Although we have successfully tested StreamFlow Workflow Manager in streamflow-examples there is a need to automate the process of running CWL with streamflow like it is done by CWL-tes tool.
• For that purpose the streamflow-client repository has been created.

assigned to @vgeo

assigned to @npappas

assigned to @npappas

added 1 deleted label

added 1 deleted label and removed 1 deleted label

added Platform label and removed 1 deleted label

@ngeorgomanolis This is still WIP.

removed 1 deleted label

Do we have an issue for the multi-site aspect of workflows? E.g. orchestration (co-scheduling?), data transfer? Related: Do we also track aspects of workflow step reliability (automated retries, moving "steps" to a different site (maybe even "interactively") etc.) and other features for complex workflows (i.e. things that a plain "make" (build flow tool of your choice) cannot do)? Somewhat similar to what ganga supports (e.g. job manip and data transfer aspects).

I know that @noelp started collecting information in task 6.4 but I'm not sure if these things already hit the issue tracker. (In the last T6.4 meeting I suggested to create a cheatsheet that provides the collected information — maybe a feature matrix for the known tools.)

removed Platform label

moved from ri/tech-hub/platform/base#1 (moved)

changed the description

changed title from Automate streamflow run process. to Automate streamflow run process

Working with CWL and StreamFlow locally to familiarize myself with both.

So far, what has been tested and works as is, without requiring custom implementation, is the case where the CWL workflow is a containerized command-line tool (https://github.com/alpha-unito/streamflow/blob/master/docs/source/cwl/docker-requirement.rst). In such a case, it is executed successfully via StreamFlow.

I am currently working on getting it to run for a full CWL workflow consisting of multiple command-line tools. No results yet…

Thanks for sharing your progress. I'm curious how explicit WMS like StreamFlow and implicit WMS like Parsl compare in terms of usability, flexibility and reproducibility.

I'm currently working on porting the demonstrator_workflow.cwl to Parsl and Globus, i.e. I will reuse the CWL CommandLineTools, but rewrite the CWL Workflow itself as a Parsl script. I've already automated the creation of Globus Compute and Globus Transfer endpoints. This makes it easier to add new execution sites. Still, I need to figure out container management and log file collection to make the workflow execution visible to the user.

hello @noelp thank you for your updates as well. is this related to multisite HPC workflows? @vgeo is currenlty working on setting up a cwl workflow engine (or a compatible to CWL at least) and we suggested to start experimenting with Streamflow. in general, as a platform, we should provide to EBRAINS users a way to run CWL workflows (for now in the cloud / via k8s) - for sure we are interested about your work - and when we are in a mature stage, align with you for providing the possibility to run complex multisite HPC workflows

I have successfully run CWL workflows with multiple steps locally via Streamflow. This now needs to be tested in Kubernetes.

marked the checklist item create a project in rke2-1-dev-cineca for @vgeo as completed