dev: bootstrap script and README update

README.md

 # Dedal
 
-This repository provides functionalities to easily ```managed spack environments``` and ```helpers for the container image build flow```.
+This repository provides functionalities to easily ```managed spack environments``` and
+```helpers for the container image build flow```.
 
 **Setting up the needed environment variables**
-    The ````<checkout path>\dedal\.env```` file contains the environment variables required for OCI registry used for caching.
-    Ensure that you edit the ````<checkout path>\dedal\.env```` file to match your environment.
-    The following provides an explanation of the various environment variables:
-
+The ````<checkout path>\dedal\.env```` file contains the environment variables required for OCI registry used for
+caching.
+Ensure that you edit the ````<checkout path>\dedal\.env```` file to match your environment.
+The following provides an explanation of the various environment variables:
 
        # OCI Registry Configuration Sample for concretization caches
        # =============================
@@ -41,13 +42,101 @@ This repository provides functionalities to easily ```managed spack environments
        # The password used for authentication with the Docker registry.
        BUILDCACHE_OCI_HOST="###ACCESS_TOKEN###"
 
-For both concretization and binary caches, the cache version can be changed via the attributes ```cache_version_concretize``` and ```cache_version_build```. 
+For both concretization and binary caches, the cache version can be changed via the attributes
+```cache_version_concretize``` and ```cache_version_build```.
 The default values are ```v1```.
 
 Before using this library, the following tool must be installed on Linux distribution:
+
 ````
     apt install -y bzip2 ca-certificates g++ gcc gfortran git gzip lsb-release patch python3 python3-pip tar unzip xz-utils zstd
 ````
+
 ````
     python3 -m pip install --upgrade pip setuptools wheel
 ````
+
+# Dedal library installation
+
+```sh
+  pip install dedal
+```
+
+# Dedal CLI Commands
+
+The following commands are available in this CLI tool. You can view detailed explanations by using the `--help` option
+with any command.
+
+### 1. `dedal install-spack`
+
+Install spack in the install_dir folder.
+
+**Options:**
+
+- `--spack_version <TEXT>` : Specifies the Spack version to be installed (default: v0.23.0).
+- `--bashrc_path <TEXT>` : Defines the path to .bashrc.
+
+### 2. `dedal set-config`
+
+Sets configuration parameters for the session.
+
+**Options:**
+
+- `--use_cache`                     Enables cashing
+- `--use_spack_global`              Uses spack installed globally on the os
+- `--env_name <TEXT>`                 Environment name
+- `--env_path <TEXT>`                 Environment path to download locally
+- `--env_git_path <TEXT>`             Git path to download the environment
+- `--install_dir <TEXT>`              Install directory for installing spack;
+  spack environments and repositories are
+  stored here
+- `--upstream_instance <TEXT>`        Upstream instance for spack environment
+- `--system_name <TEXT>`              System name; it is used inside the spack
+  environment
+- `--concretization_dir <TEXT>`       Directory where the concretization caching
+  (spack.lock) will be downloaded
+- `--buildcache_dir <TEXT>`           Directory where the binary caching is
+  downloaded for the spack packages
+- `--gpg_name <TEXT>`                 Gpg name
+- `--gpg_mail <TEXT>`                 Gpg mail contact address
+- `--cache_version_concretize <TEXT>`
+  Cache version for concretizaion data
+- `--cache_version_build <TEXT>`      Cache version for binary caches data
+
+### 3. `dedal show-config`
+
+Show the current configuration.
+
+### 4. `dedal clear-config`
+
+Clears stored configuration
+
+### 5. `dedal add-spack-repo`
+
+Adds a spack repository to the spack environments.
+
+**Options:**
+
+- `--repo_name <TEXT>`  Repository name  [required]
+- `--path <TEXT>`       Repository path to download locally  [required]
+- `--git_path <TEXT>`   Git path to download the repository  [required]
+
+### 6. `dedal setup-spack-env`
+
+Setups a spack environment according to the given configuration.
+
+### 7. `dedal concretize`
+
+Spack concretization step.
+
+### 9. `dedal install-packages`
+
+Installs spack packages present in the spack environment defined in configuration.
+
+**Options:**
+
+- `--jobs <INTEGER>`  Number of parallel jobs for spack installation
+
+# Dedal's UML diagram
+
+![screenshot](dedal/docs/resources/dedal_UML.png)
\ No newline at end of file

dedal/build_cache/BuildCacheManager.py

 import os
+import time
+
 import oras.client
 from pathlib import Path
 
@@ -46,7 +48,7 @@ class BuildCacheManager(BuildCacheManagerInterface):
                 rel_path = str(sub_path.relative_to(build_cache_path)).replace(str(sub_path.name), "")
                 target = f"{self._registry_host}/{self._registry_project}/{self.cache_version}:{str(sub_path.name)}"
                 try:
-                    self._logger.info(f"Pushing folder '{sub_path}' to ORAS target '{target}' ...")
+                    self._logger.info(f"Pushing file '{sub_path}' to ORAS target '{target}' ...")
                     self._client.push(
                         files=[str(sub_path)],
                         target=target,

dedal/cli/spack_manager_api.py

@@ -55,7 +55,7 @@ def cli(ctx: click.Context):
 def set_config(use_cache, env_name, env_path, env_git_path, install_dir, upstream_instance, system_name,
                concretization_dir,
                buildcache_dir, gpg_name, gpg_mail, use_spack_global, cache_version_concretize, cache_version_build):
-    """Set configuration parameters for tahe session."""
+    """Sets configuration parameters for the session."""
     spack_config_data = {
         'use_cache': use_cache,
         'env_name': env_name,
@@ -88,8 +88,8 @@ def show_config():
 
 
 @cli.command()
-@click.option('--spack_version', type=str, default='0.23.0', help='Spack version')
-@click.option('--bashrc_path', type=str, default="~/.bashrc", help='Path to .bashrc')
+@click.option('--spack_version', type=str, default='0.23.0', help='Specifies the Spack version to be installed (default: v0.23.0).')
+@click.option('--bashrc_path', type=str, default="~/.bashrc", help='Defines the path to .bashrc.')
 @click.pass_context
 def install_spack(ctx: click.Context, spack_version: str, bashrc_path: str):
     """Install spack in the install_dir folder"""
@@ -124,7 +124,7 @@ def setup_spack_env(ctx: click.Context):
 @cli.command()
 @click.pass_context
 def concretize(ctx: click.Context):
-    """Spack concretization step"""
+    """Spack concretization step."""
     ctx.obj.concretize_spack_env()
 
 
@@ -132,13 +132,13 @@ def concretize(ctx: click.Context):
 @click.option('--jobs', type=int, default=2, help='Number of parallel jobs for spack installation')
 @click.pass_context
 def install_packages(ctx: click.Context, jobs):
-    """Installs spack packages present in the spack environment defined in configuration"""
+    """Installs spack packages present in the spack environment defined in configuration."""
     ctx.obj.install_packages(jobs=jobs)
 
 
 @click.command()
 def clear_config():
-    """Clear stored configuration"""
+    """Clears stored configuration."""
     if os.path.exists(SESSION_CONFIG_PATH):
         os.remove(SESSION_CONFIG_PATH)
         click.echo('Configuration cleared!')

dedal/spack_factory/SpackOperationCreateCache.py

@@ -31,10 +31,10 @@ class SpackOperationCreateCache(SpackOperation):
     @check_spack_env
     def concretize_spack_env(self):
         super().concretize_spack_env(force=True)
-        dependency_path = self.spack_config.env.path / self.spack_config.env.env_name / 'spack.lock'
+        dependency_path = self.spack_config.env.path / self.spack_config.env.name / 'spack.lock'
         copy_file(dependency_path, self.spack_config.concretization_dir, logger=self.logger)
         self.cache_dependency.upload(self.spack_config.concretization_dir)
-        self.logger.info(f'Created new spack concretization for create cache: {self.spack_config.env.env_name}')
+        self.logger.info(f'Created new spack concretization for create cache: {self.spack_config.env.name}')
 
     @check_spack_env
     def install_packages(self, jobs: int = 2, debug=False):
@@ -44,8 +44,8 @@ class SpackOperationCreateCache(SpackOperation):
             self.create_gpg_keys()
         self.add_mirror('local_cache', self.spack_config.buildcache_dir, signed=signed, autopush=signed,
                         global_mirror=False)
-        self.logger.info(f'Added mirror for {self.spack_config.env.env_name}')
+        self.logger.info(f'Added mirror for {self.spack_config.env.name}')
         super().install_packages(jobs=jobs, signed=signed, debug=debug, fresh=True)
-        self.logger.info(f'Installed spack packages for {self.spack_config.env.env_name}')
+        self.logger.info(f'Installed spack packages for {self.spack_config.env.name}')
         self.build_cache.upload(self.spack_config.buildcache_dir)
-        self.logger.info(f'Pushed spack packages for {self.spack_config.env.env_name}')
+        self.logger.info(f'Pushed spack packages for {self.spack_config.env.name}')

dedal/spack_factory/SpackOperationUseCache.py

@@ -60,8 +60,8 @@ class SpackOperationUseCache(SpackOperation):
                                      stderr=subprocess.PIPE,
                                      text=True,
                                      logger=self.logger,
-                                     info_msg=f"Installing spack packages for {self.spack_config.env.env_name}",
-                                     exception_msg=f"Error installing spack packages for {self.spack_config.env.env_name}",
+                                     info_msg=f"Installing spack packages for {self.spack_config.env.name}",
+                                     exception_msg=f"Error installing spack packages for {self.spack_config.env.name}",
                                      exception=SpackInstallPackagesException)
         log_command(install_result, str(Path(os.getcwd()).resolve() / ".generate_cache.log"))
         return install_result

dedal/utils/bootstrap.sh

-# Minimal prerequisites for installing the esd_library
+# Minimal prerequisites for installing the dedal library
 # pip must be installed on the OS
 echo "Bootstrapping..."
+set -euo pipefail
+shopt -s inherit_errexit 2>/dev/null
+export DEBIAN_FRONTEND=noninteractive
 apt update
-apt install -y bzip2 ca-certificates g++ gcc gfortran git gzip lsb-release patch python3 python3-pip tar unzip xz-utils zstd
+apt install -o DPkg::Options::=--force-confold -y -q --reinstall \
+                  bzip2 ca-certificates g++ gcc make gfortran git gzip lsb-release \
+                  patch python3 python3-pip tar unzip xz-utils zstd gnupg2 vim curl rsync
 python3 -m pip install --upgrade pip setuptools wheel

dedal/utils/utils.py

@@ -97,6 +97,7 @@ def copy_to_tmp(file_path: Path) -> Path:
 def set_bashrc_variable(var_name: str, value: str, bashrc_path: str = os.path.expanduser("~/.bashrc"),
                         logger: logging = logging.getLogger(__name__)):
     """Update or add an environment variable in ~/.bashrc."""
+    value = value.replace("$", r"\$")
     with open(bashrc_path, "r") as file:
         lines = file.readlines()
     pattern = re.compile(rf'^\s*export\s+{var_name}=.*$')