GitLab Runner gets stuck
The GitLab Runner randomly started getting stuck in different parts of the day, the result being a failed pipeline, since the runner has a timeout of 1 hour.
The runner gets stuck at a test which invokes spack concretize --force, but other tests which utilize this spack command succeed without any issues.
Activity
assigned to @elmath
added Base Infra label
@adrianciu could you please give an example of such a GitLab CI job? In general
spack concretizeoften has this issue with complex environments (it's probably not the GitLab runners, we actually often faced this on HPC), but I haven't encountered it with simple test environments so far.Edited by Eleni Mathioulaki@elmath the issue was caused by concretizing a complex environment (ebrains-spack-builds). However, during a previous run, the runner failed at the initialization step with a 500 error: https://gitlab.ebrains.eu/ri/tech-hub/platform/esd/dedal/-/jobs/171797. in this job, the tests appear to have failed due to a problem with the runner, as these tests had never failed before and successfully passed a few hours later in a new job.
Edited by Adrian CiuOK, thanks, I see.
The 500 error (when uploading the artifacts) was caused by the temporary unavailability of GitLab's underlying storage on JSC on March 31st (see https://status.jsc.fz-juelich.de/services/27#incident-1076), which did not allow the artifacts to be saved properly. It was an isolated, infrastructure-related incident not directly related to the runners, the concretizer issues or the specific pipeline, and it should not affect later runs of the job.
Edited by Eleni Mathioulaki
@elmath today, this GitLab runner failed: https://gitlab.ebrains.eu/ri/tech-hub/platform/esd/dedal/-/jobs/173297. The issue happened while installing some Linux tools, related to spack, via bash. After I run again this job, the issue did not persist.
Thanks for reporting this @adrianciu :)
It looks like the runner experienced a temporary network or DNS issue that prevented it from reaching the Ubuntu package servers. Since it didn't persist, this seems to be just a transient, short-lived connectivity issue.
