Adding a workflow package to the ESD

• Adding a workflow package to the ESD
  • What is a Workflow Package?
  • How to Create a Workflow Package with Spack
    • Create the Workflow Package
    • Open a Merge Request

What is a Workflow Package?

A workflow package in the EBRAINS Software Distribution is a meta-package that encapsulates all the dependencies required to run a specific workflow. These workflows can include notebooks, scripts, CWL workflows, and more. By packaging them as workflow packages, we achieve several objectives:

• workflow packages provide a way to introduce essential tools that are not developed in EBRAINS but are crucial for the workflow/for EBRAINS users.
• They also ensure that EBRAINS workflows can be executed seamlessly on various platforms where the ESD is available, including the EBRAINS Lab and HPC systems.
• The workflows themselves are used as test for the interoperability of tools inside the ESD, ensuring that the tools that compose the workflow remain compatible after updates in the environment.

A workflow package includes:

• all software dependencies of the workflow, which may consist of both EBRAINS-developed and external tools
• well-defined tests to ensure the workflow runs correctly and performs as expected

How to Create a Workflow Package with Spack

Note: Instead of setting up a local Spack environment, it is recommended to just open a Merge Request to our repository, where automated CI will install and test the package on top of our existing environment.

Create the Workflow Package

In a fork/new branch of the ESD repository, navigate to the Spack repository (under packages/) and create a new package file for your workflow package. For EBRAINS workflow packages, we use a wf- prefix in the package name. For example, to create a package named wf-example, create a file at ebrains-spack-builds/packages/wf-example/package.py.

In your package file, define a new class for the workflow package. The class should inherit from Package. Include metadata such as the homepage, repository, and maintainers, and define the versions of the workflow. List all the dependencies required for your workflow using depends_on() statements. Then, include methods to handle the installation, testing, and validation of the workflow. Customize these methods based on your specific workflow requirements.

Dedicated documentation and instructions on how to create a Spack package can be retrieved from the official Spack documentation, that includes a very detailed packaging guide as well as a package creation tutorial. A very quick overview is also provided in the "Adding an EBRAINS tool to the ESD" page.

Here is a template for a new workflow package, wf-example:

from spack import *

class WfExample(Package):
    """Meta-package to collect all dependencies of the Example Workflow."""

    homepage = "https://example.com/wf-example"
    git = "https://github.com/EBRAINS/wf-example"
    maintainer = ["maintainer1", "maintainer2"]

    version("1.0.0", tag="v1.0.0")
    version("master", branch="master")

    depends_on("python@3.8:", type=("run", "test"))
    depends_on("py-jupyterlab", type=("run", "test"))
    depends_on("py-numpy", type=("run", "test"))
    depends_on("nest", type=("run", "test"))
    # Add more dependencies as needed

    # helper functions for running notebooks with nbconvert
    def install(self, spec, prefix):
        install_tree(".", join_path(prefix, "notebooks"))

    def _nbconvert(self, nb, nb_out):
        jupyter = Executable("jupyter")
        args = [
            "nbconvert",
            "--ExecutePreprocessor.kernel_name=python3",
            "--execute",
            "--to",
            "notebook",
            nb,
            "--output",
            nb_out
        ]
        try:
            jupyter(*args)
        except Exception as e:
            jupyter(*(args + ["--allow-errors"]))
            raise

    def _run_notebooks(self, output_dir):
        mkdirp(output_dir)
        self._nbconvert(join_path(self.prefix, "notebooks", "example-notebook.ipynb"), join_path(output_dir, "example-notebook.ipynb"))

    # installation-time tests
    @run_after("install")
    @on_package_attributes(run_tests=True)
    def installcheck(self):
        self._run_notebooks(join_path(self.stage.path, ".install_time_tests"))
        copy_tree(join_path(self.stage.path, ".install_time_tests"), join_path(self.prefix, '.build'))

    # standalone tests (executed periodically)
    def test_notebook(self):
        self._run_notebooks(join_path(self.test_suite.stage, self.spec.format("out-{name}-{version}-{hash:7}")))

Open a Merge Request

After defining your package, commit your changes and submit a Merge Request to our repository. The CI system will automatically attempt to install the package and run tests to ensure everything works as expected.

By following these steps and using the template provided, you can create workflow packages that include all necessary software dependencies and tests to ensure reproducibility and interoperability within the EBRAINS ecosystem.