Skip to content
Snippets Groups Projects

stand-alone (not built-time) tests

Merged stand-alone (not built-time) tests
stand-alone-tests into master
1 unresolved thread
Merged Eleni Mathioulaki requested to merge stand-alone-tests into master
1 unresolved thread

Merge request reports

Pipeline #25728 passed

Stage:

    Pipeline passed for 93c620e2 on stand-alone-tests

    Approval is optional

    Merged by Eleni MathioulakiEleni Mathioulaki 1 year ago (Oct 12, 2023 10:53pm UTC)

    Merge details

    • Changes merged into with efc64bbc.
    • Deleted the source branch.

    Pipeline #25799 waiting for manual action

    Stage:
      Stage:

        Pipeline waiting for manual action for efc64bbc on master

        Activity

        Filter activity
        • Approvals
        • Assignees & reviewers
        • Comments (from bots)
        • Comments (from users)
        • Commits & branches
        • Edits
        • Labels
        • Lock status
        • Mentions
        • Merge request status
        • Tracking
        • Eleni Mathioulaki marked this merge request as draft

          marked this merge request as draft

        • Eleni Mathioulaki
          Eleni Mathioulaki @elmath ·
          Author Owner

          Hey @emuller @terhorst

          First attempt for some scheduled tests, using Spack standalone tests:

          spack test run -x runs all methods named test_* from all top-level packages in the environment.

          Unfortunately the test dependencies are not loaded correctly, so some tests fail, probably a Spack bag - I need to look into it.

          For now those tests run:

          • in a scheduled CI job that tests the entire environment (I think weekly is fine?)
          • on every MR (only for any newly installed packages)
          • on every merge to master (for the entire environment)

          Once this works, we can use the same mechanism to test the Jupyterlab installation (with a scheduled OKD job from gitlab) and the HPC installations (by using unicore or jacamar or whatever else).

          I used your notebook tests as an example and changed the logic a bit to make this work, so please have a look :) and if you have any other ideas let me know

          Edited by Eleni Mathioulaki
        • Eleni Mathioulaki added 1 commit

          added 1 commit

          • 93c620e2 - only test wf packages for now

          Compare with previous version

        • Eleni Mathioulaki
          Eleni Mathioulaki @elmath ·
          Author Owner

          @emuller @terhorst test dependencies are still not loaded correctly when running spack test run, so tests for some packages fail when testing the entire env.

          But I think it makes sense to merge this just to have at least some stand-alone tests running e.g. weekly as an alternative to the installation-time tests - I started with your 2 wf packages.

          Then later we can decide which tests need to run when and debug the dependency issue.

          what do you think?

        • Dennis Terhorst
          Dennis Terhorst @terhorst started a thread on the diff
        packages/wf-multi-area-model/package.py
        27 depends_on("r-aod")
        28 depends_on("py-notebook", type="test")
        20 depends_on("py-nested-dict", type=("run", "test"))
        21 depends_on("py-dicthash", type=("run", "test"))
        22 depends_on("py-matplotlib", type=("run", "test"))
        23 depends_on("py-numpy", type=("run", "test"))
        24 depends_on("py-scipy", type=("run", "test"))
        25 depends_on("py-future", type=("run", "test"))
        26 depends_on("nest", type=("run", "test"))
        27 depends_on("r-aod", type=("run", "test"))
        28 depends_on("py-notebook", type=("run", "test"))
        29 29
        30 30 def install(self, spec, prefix):
        31 # sanity_check_prefix requires something in the install directory
        32 mkdirp(prefix + "/.spack_test_results")
        31 install_tree(".", join_path(prefix, "notebooks"))
        • Dennis Terhorst
          Dennis Terhorst @terhorst ·
          Maintainer

          Does this mean the notebooks are also installed into the software stack? – This would open some new possibilities also for tutorials and workshops!

        • Eleni Mathioulaki
          Eleni Mathioulaki @elmath ·
          Author Owner

          yes, if we want to use them as tests they need to be installed, right? :)

        • Dennis Terhorst
          Dennis Terhorst @terhorst ·
          Maintainer

          I suppose so (and am am very happy with that). The question would be if the sources are available during test time. For periodic tests either the notebooks are installed, or the repo needs to be re-cloned. I'm fine with having them installed. In our case the repos are really light-weight. In general, this might however require a conceptual discussion how wf- packages are set up. We might for example consider to require notebooks to be in a dedicated subfolder and install_tree("notebooks", join_path(prefix, "notebooks")) or so. For now I think we can leave it up to the package maintainers to make a sensible choice of what to install.

        • Eleni Mathioulaki
          Eleni Mathioulaki @elmath ·
          Author Owner

          Exactly, I think that's completely up to the package maintainers. For some workflows the entire repo needs to be copied, for others just one subfolder, for others maybe we can even build the notebooks here, whatever works best :)

        • Please register or sign in to reply
      • Dennis Terhorst
        Dennis Terhorst @terhorst ·
        Maintainer

        For me this looks like a good starting point. With the TEST_DEPLOYMENT variable this is controllable from the outside, so even post-release it can be turned off if necessary. It may not be the perfect solution for all workflows, as they might pull in quite some data when run, or require access to services, but I'd be interested to see how far this carries.

        :thumbsup: I'd vote to give it a try.

      • Dennis Terhorst approved this merge request

        approved this merge request

      • Eleni Mathioulaki marked this merge request as ready

        marked this merge request as ready

      • Eleni Mathioulaki merged

        merged

      • Eleni Mathioulaki mentioned in commit efc64bbc

        mentioned in commit efc64bbc

        • Please register or sign in to reply