Build path parallelism and "distribution"
Steps to get from a structured software environment description (i.e. a list of spack package dependency incl. version and variant constraints) to a deployed software environment can be parallelized and distributed:
- Separation into "connected" (cache interaction and downloading of spack package sources)
- and "isolated" tasks (building, packaging (image construction))
Open questions:
- Can we make use of
spack env depfileto "dispatch" individualspack install(build task) steps?- job-local parallelism (→
-j) - Handling of per-package requirements — e.g., memory per core (i.e.
memory_per_core * max_jobs < total_mem) - unicore?
- job-local parallelism (→
- Can we automatize the "build task category" dependency handling? (e.g., via some build or "workflow" tool like
snakemake?)