Draft: feat(wf-unicore-examples): add initial version

This is a workflow package to test the interaction with UNICORE base HPC service.

For the GitLab based testing it might require a UNICORE service container as a stable reference to run against. For install-time tests the UNICORE install of the HPC sites could be used if the site-config exposes the corresponding base URL and quota to use.

added 1 commit

• 3b64dba9 - fix package syntax

Compare with previous version

To get this out of Draft state, I'd need some help. @noelp and @bschulle? Probably having the supercomputers-via-unicore package in my user namespace is also not the best of ideas.

Yes, I would suggest, as first steps:

• moving the repo to a different namespace
• consider making it public - would this be possible?

@bschulle, do you have a preferred server/namespace to relocate to? I would be fine migrating wherever you think it fits best.

Given that we have a LICENSE, it seems viable to become a public project, however there are some quirks and open questions that would make the project not pass e.g. our institute publication guidelines. For example the gitlab seems to detects it as BSD-3-clause, while there is a BSD-2-clause license file?! Who will be the official maintainer? …

hi Dennis

I think this is getting a little out of hand :-)

This is not "a workflow package to test integration" or something, it is a collection of EXAMPLES for how to use pyunicore in the Ebrains context (i.e. with an OAuth token).

It should not be in the "EBRAINS software distribution", since it is not software, but documentation.

I would close and delete this merge request.

PS wasn't the original idea just to have the examples somewhere on gitlab and outside of the wiki?

This is a good question and shows that we should be more explicit about our motivation and goals. UNICORE can handle user authentication, that's why we use it right? Is it the right tool to handle workflows? I don't know and I'm not sure if the grant agreement demands CWL support.

Maybe Dennis' idea was to start with the "documentation" and turn it into integration tests, that will be part of the "EBRAINS software distribution".

We could either use the unicore-docker-image, at least if we fix the slow container start, or use rootless.

I've got no experience with CI and which approach is the best.

https://gitlab.ebrains.eu/ri/tech-hub/platform/esd/yashchiki/-/blob/ebrains_testing/.gitlab-ci.yml

https://gitlab.jsc.fz-juelich.de/nest/ebrains-spack-builds/-/blob/ebrains-23-09-jsc/.gitlab-ci.yml

The UNICORE example notebooks could check ENV variables to set the URLs

hi @noelp - what do you mean by "slow container start"?

For me it freezes for about 5 minutes during startup after showing uid=0(root) gid=0(root) groups=0(root).

$ docker run -p 8080:8080 -ti ghcr.io/unicore-eu/unicore-testing-all:10.0.0
Creating user(s)...
Setting up Slurm...
uid=0(root) gid=0(root) groups=0(root)
Setting up UNICORE...
Configuring the installation for user unicore in directory /opt/unicore/unicore-servers, on machine localhost
...
Running interactive shell
root@6bdbd0211266:/opt/unicore#

Sadly email notifications on https://chat.ebrains.eu are currently broken.

hmm interesting, that is more or less instantaneous for me. In the integration tests, it takes less than a minute to start the services, including downloading the image and making sure the services are up.

for what it's worth, on Github I use the unicore-docker image for integration tests: https://github.com/HumanBrainProject/pyunicore/blob/dev/.github/workflows/integration-test-unicore10.yml

the code in "rootless" makes no sense to me at all in its current state

The idea behind "rootless" is that you don't need a container runtime like docker or apptainer to test UNICORE and Slurm. This could be useful in the future if HPC sites don't support UNICORE.

I understand the concept, but not the current implementation, or what it is trying to achieve ;-) but we should talk about this directly, not via comments

Here are some notes of today's meeting with Bernd https://notes.ebrains.eu/hzquuMFER-m2LuWDYXeRNg#