Skip to content
Snippets Groups Projects
Select Git revision
  • noelp-master-patch-87404
  • master default protected
  • disable-view
  • experimental_rel
  • test_quiggeldy_service
  • update-arbor-0.10.0
  • image_build
  • spack_v0.22.1
  • ebrains-24-04
  • update-readme
  • create-module-file
  • add-nestml-tests
  • feat_add_py-norse
  • update-libneuroml
  • update-bluebrain-packages
  • feat_arbor_install_all_binaries
  • ebrains-23.09-jsc-site-config
  • spack-v0.20.0
  • ebrains-23-09-spack-v0.19.2
  • ebrains-23-09
20 results
Compare History
Eric Müller's avatar
fix(bazel): build on ju{suf,reca}
Eric Müller authored
01c24df8
Name Last commit Last update
.gitlab/issue_templates
etc
operations
packages
site-config
.gitignore
.gitlab-ci.yml
README.md
create_JupyterLab_kernel.sh
create_job.sh
install_spack_env.sh
repo.yaml
spack.yaml
README.md

ebrains-spack-builds

Build and distribute EBRAINS software tools with Spack.

Introduction

Building and deploying EBRAINS tools to make them available for direct use in the Collaboratory Lab environment is a challenging task, mainly due to the number of tools and the increasing complexity of resolving their dependency conflicts. To efficiently solve this problem, we have implemented a delivery strategy based on Spack, a package manager specifically designed to simplify the installation and management of scientific software (more information here).

Spack is highly customizable and flexible, allowing developers to create their own packages and specify their dependencies and custom build configurations. It supports a wide range of programming languages (Python, R, C, C++, Fortran etc), different hardware architectures and operating systems. All an EBRAINS developer needs to do is create a Spack package (simple description of the build logic of their software). Then Spack automatically handles downloading, building, and installing the entire EBRAINS environment, managing all their dependencies and other requirements.

Repository

This repository is where the EBRAINS Spack packages are maintained. It contains:

  • the EBRAINS Spack repository, that contains all available EBRAINS Spack packages (packages/)
  • the latest EBRAINS Spack environment (spack.yaml), a list of the specific versions of EBRAINS tools that are part of the software release
  • a script (install_spack_env.sh) that can be used to install the latest spack environment on any machine (either install from scratch or update an existing installation)
  • CI/CD pipelines configured for (a) testing the installation of new packages and (b) deploying the environment to the EBRAINS Collaboratory Lab.
  • (WIP) all the site-specific configurations that have been used to install the EBRAINS Spack environment in different systems, such as the EBRAINS Lab and the Fenix HPC systems.

Contributing

HBP's developers/component owners that want their EBRAINS tools to be included in the official EBRAINS releases and deployed for direct use from the Lab environment should follow those steps:

  • Fork this repository
  • Clone the forked repository to local machine (or use the Gitlab Web IDE)
  • Create a new Spack package for your tool. For instructions on how to do this, see the Spack documentation and the developer's section of our documentation. Add the new package to the repository, by creating a directory inside packages/ and copying the package recipe there. Also don't forget to add a spec to spack.yaml
  • Commit the new package and push to a new branch, e.g. add-packagename
  • Create a Merge Request to merge your changes into the master branch of the original repository
  • Check the results of the automated test build job and solve any issues
  • Once the tests pass, wait for TC to approve and merge. Once merged, the package will be automatically built and deployed to the integration environment of the Collaboratory Lab (lab-int).

Installing the EBRAINS official Software release

To install the EBRAINS Spack environment on a new machine, it is recommended to follow exactly the steps from install_spack_env.sh, that installs or updates and tests the spack environment defined in this repository. In general, you need to:

Clone this repository. You can use the ebrains-yy-mm branches to install the EBRAINS official releases, the experimental-rel branch for the experimental release, or the master branch for the latest state:

git clone --branch {branch-name} https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds.git

Clone Spack. We currently use version v0.21.1:

git clone --depth 1 -c advice.detachedHead=false -c feature.manyFiles=true --branch v0.20.0 https://github.com/spack/spack

Activate Spack:

. spack/share/spack/setup-env.sh

Add the project repository to your Spack environment:

spack repo add ebrains-spack-builds

Create the environment:

spack env create -d ebrains-spack-builds/

Define your site-specific configurations:

export SYSTEMNAME=<your-system-name>
mkdir ebrains-spack-builds/site-config/$SYSTEMNAME
# copy any site-specific .yaml files inside the new dir

Activate the environment and install:

spack env activate ebrains-spack-builds
spack install --fresh

Troubleshooting

If you encounter any issues with installing, testing, or using the EBRAINS Spack environment or a specific package, please open an issue using the provided templates, or contact the Technical Coordination team directly.