Skip to content

Build path parallelism and "distribution"

Steps to get from a structured software environment description (i.e. a list of spack package dependency incl. version and variant constraints) to a deployed software environment can be parallelized and distributed:

  • Separation into "connected" (cache interaction and downloading of spack package sources)
  • and "isolated" tasks (building, packaging (image construction))

Open questions:

  • Can we make use of spack env depfile to "dispatch" individual spack install (build task) steps?
    • job-local parallelism (→ -j)
    • Handling of per-package requirements — e.g., memory per core (i.e. memory_per_core * max_jobs < total_mem)
    • unicore?
  • Can we automatize the "build task category" dependency handling? (e.g., via some build or "workflow" tool like snakemake?)