Installation issue: pynn-brainscales building environment

Summary

Summary	Spack package build failure
Package info	pynn-brainscales
System	Nord3 cluster (BSC), Intel CPU without GPUs

There are two issues with the package. The first one is that we've noticed that the system's environment is modified just before running the fetchers, using the following function: https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/-/blob/master/packages/pynn-brainscales/package.py#L56

This methodology can introduce potential issues. We have found that since it introduces all the environment's libraries into LD_LIBRARY_PATH, there can be cases where system binaries required for spack processes link to the wrong libraries. For example, our fetchers started failing with this package:

"symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b"

Since the concretization had a openssl package, I had to specifically filter it out inside the "package.py" file. I don't know if there is a spack solution for this situation.

The second issue is that even if we apply our workaround, the package has trouble recognizing boost libraries. See the "Error message" section for more details.

Steps to reproduce the issue

$ spack env activate <ebrains-enviornment>
$ spack install

I will provide the whole spec and spack.yaml of the environment for completeness.

Error message

This is the error message for our second issue, where we have trouble locating the boost libraries that are already installed by spack as a dependency:

Error message

======================= pynn-brainscales/wscript =======================
============================ logger/wscript ============================
Checking for 'g++' (C++ compiler)        : /.statelite/tmpfs/gpfs/apps/SPACK/spack/lib/spack/env/gcc/g++
Test text summary directory              : build/test_results
Test xml summary directory               : build/test_results
GoogleTest maximal runtime               : 30 seconds (default)
Checking for GoogleTest library          : yes
Checking for program 'dpkg-architecture' : not found
Checking boost includes                  : 1.79.0
Checking boost libs                      : lib system not found in /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/boost/1.79.0/gcc-10.3.0/fofqevu/lib
The configuration failed
NOTE: confcache is enabled, make sure you supply --disable-confcache to configure from scratch!

Contents of /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/boost/1.79.0/gcc-10.3.0/fofqevu/lib

$ ls /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/boost/1.79.0/gcc-10.3.0/fofqevu/lib
boost-python3.8                       libboost_graph_parallel.a           libboost_python38-mt.so.1.79.0
cmake                                 libboost_graph_parallel.so          libboost_python38.a
libboost_context-mt.a                 libboost_graph_parallel.so.1.79.0   libboost_python38.so
libboost_context-mt.so                libboost_mpi-mt.a                   libboost_python38.so.1.79.0
libboost_context-mt.so.1.79.0         libboost_mpi-mt.so                  libboost_regex-mt.a
libboost_context.a                    libboost_mpi-mt.so.1.79.0           libboost_regex-mt.so
libboost_context.so                   libboost_mpi.a                      libboost_regex-mt.so.1.79.0
libboost_context.so.1.79.0            libboost_mpi.so                     libboost_regex.a
libboost_coroutine-mt.a               libboost_mpi.so.1.79.0              libboost_regex.so
libboost_coroutine-mt.so              libboost_mpi_python38-mt.a          libboost_regex.so.1.79.0
libboost_coroutine-mt.so.1.79.0       libboost_mpi_python38-mt.so         libboost_serialization-mt.a
libboost_coroutine.a                  libboost_mpi_python38-mt.so.1.79.0  libboost_serialization-mt.so
libboost_coroutine.so                 libboost_mpi_python38.a             libboost_serialization-mt.so.1.79.0
libboost_coroutine.so.1.79.0          libboost_mpi_python38.so            libboost_serialization.a
libboost_graph-mt.a                   libboost_mpi_python38.so.1.79.0     libboost_serialization.so
libboost_graph-mt.so                  libboost_numpy38-mt.a               libboost_serialization.so.1.79.0
libboost_graph-mt.so.1.79.0           libboost_numpy38-mt.so              libboost_wserialization-mt.a
libboost_graph.a                      libboost_numpy38-mt.so.1.79.0       libboost_wserialization-mt.so
libboost_graph.so                     libboost_numpy38.a                  libboost_wserialization-mt.so.1.79.0
libboost_graph.so.1.79.0              libboost_numpy38.so                 libboost_wserialization.a
libboost_graph_parallel-mt.a          libboost_numpy38.so.1.79.0          libboost_wserialization.so
libboost_graph_parallel-mt.so         libboost_python38-mt.a              libboost_wserialization.so.1.79.0
libboost_graph_parallel-mt.so.1.79.0  libboost_python38-mt.so

Afer taking a look at the contents (and comparing it with our own manual installations of boost), it seems that some of the boost libraries are not being generated (for example, "libboost_system.so"). Digging around in the concretization of the environment, we have noticed that the package "pynn-brainscales" ("and also nest") has taken this boost dependency spec:

Boost spec

^boost@1.79.0%gcc@10.3.0~atomic~chrono~clanglibcpp~container+context~contract+coroutine~date_time~debug~exception~fiber~filesystem+graph~graph_parallel+icu~iostreams~json~locale~log~math+mpi+multithreaded~nowide+numpy~pic~program_options+python~random~regex~serialization+shared~signals~singlethreaded~stacktrace~system~taggedlayout~test~thread~timer~type_erasure~versionedlayout~wave context-impl=fcontext cxxstd=17 patches=a440f96,b8569d7 visibility=hidden arch=linux-rhel8-sandybridge

You can see that there are a lot of disabled components. I don't know why spack has decided to concretize boost this way, since there are no concrete specifications for boost in the environment that I am aware of (other than boost's minimum version). One way to prevent this issue would be to specify either in the spack.yaml or in the package itself the minimum boost concretization requirements for pynn-brainscales to build. At this moment I'm not sure which boost libraries are necessary for it.

Information on your system

Documentation of the system used (if necessary): https://www.bsc.es/supportkc/docs/Nord3v2/intro

Our spack.yaml:

spack.yaml

spack:
  config:
    url_fetch_method: curl
    verify_ssl: false
  specs:
  - py-wheel
  - git
  - openssl
  - openssh
  - curl
  - py-ipython
  - py-ipykernel
  - py-notebook
  - py-ipycanvas
  - clb-nb-utils@0.1.0
  - py-pip
  - py-numpy
  - py-scipy
  - py-pandas
  - py-seaborn
  - py-matplotlib
  - py-nameparser
  - py-lazyarray
  - nest@3.3 +python +gsl +mpi
  - arbor +python +mpi ^python@3:3.9
  - neuron +mpi
  - py-pynn@0.10.1 +mpi
  - py-brian2
  - tvb-data
  - tvb-library ^py-numba@0.54.0 ^binutils+ld+gold
  - tvb-storage ^binutils+ld+gold
  - py-pyaescrypt
  - py-formencode
  - tvb-framework ^binutils+ld+gold
  - pynn-brainscales@4.0-a1 ^log4cxx@0.10.0 ^googletest@1.11.0:+gmock
    #- pynn-brainscales@4.0-a1 ^log4cxx@0.10.0 ^googletest@1.11.0:+gmock ^util-linux
    #cflags="-I/apps/EUDEV/3.2.11/GCC/include"
  - py-neo
  - py-hdmf
  - py-pynwb
  - py-nixio
  - py-cerebrus
  - py-sciunit@0.2.5.1
  - py-quantities
  - py-quantities-scidash@0.12.4.3
  - py-ebrains-kg-core
  - py-pyld
  - py-fairgraph
  - py-hbp-archive@1.1.1
  - py-viziphant
  - py-hippounit@1.3.6
  - py-efel@4.0.4
  - py-hbp-neuromorphic-platform@0.10.1
  - py-lfpy@2.2.6
  - py-elephant
  - py-frites
  - py-snudda ^py-libsonata@0.1.10
  - biobb-common
  - biobb-gromacs
  - suite-sparse
  - apbs ^openblas
  - py-pdb2pqr
  - gcc@10.3.0
  - python@3.8.11%gcc@10.3.0
  repos:
  - /gpfs/apps/SPACK/spack4bsc/repos/ebrains_nord3v2
  compilers:
  - compiler:
      spec: gcc@10.3.0
      paths:
        cc: /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/gcc/10.3.0/gcc-8.4.1/pbd6c3a/bin/gcc
        cxx: /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/gcc/10.3.0/gcc-8.4.1/pbd6c3a/bin/g++
        f77: /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/gcc/10.3.0/gcc-8.4.1/pbd6c3a/bin/gfortran
        fc: /.statelite/tmpfs/gpfs/apps/SPACK/nord3v2/gcc/10.3.0/gcc-8.4.1/pbd6c3a/bin/gfortran
      flags: {}
      operating_system: rhel8
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  packages:
    all:
      target: [x86_64]
      compiler: [gcc@10.3.0]
    intel-mkl:
      buildable: false
      externals:
      - spec: intel-mkl@2018.4.0%intel@2021.4.0~ilp64+shared threads=none arch=linux-rhel8-sandybridge
        prefix: /gpfs/apps/MN4/INTEL/2018.4.057/compilers_and_libraries_2018.5.274/linux/mkl/
        prefix.lib: /gpfs/apps/MN4/INTEL/2018.4.057/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin
        prefix.bin: /gpfs/apps/MN4/INTEL/2018.4.057/compilers_and_libraries_2018.5.274/linux/mkl/bin
        prefix.inc: /gpfs/apps/MN4/INTEL/2018.4.057/compilers_and_libraries_2018.5.274/linux/mkl/include
        modules:
        - mkl/2018.4
    eudev:
      version: [3.2.11]
      buildable: false
      externals:
      - spec: eudev@3.2.11
        prefix: /gpfs/apps/NORD3/EUDEV/3.2.11/GCC/
        prefix.lib: /gpfs/apps/NORD3/EUDEV/3.2.11/GCC/lib
        prefix.bin: /gpfs/apps/NORD3/EUDEV/3.2.11/GCC/bin
        prefix.inc: /gpfs/apps/NORD3/EUDEV/3.2.11/GCC/include
        modules:
        - eudev/3.2.11
concretizer:
unify: when_possible

Our environment concretization: spec.txt

assigned to @emuller

The first one is that we've noticed that the system's environment is modified just before running the fetchers […]

Is it the "spack fetchers" or the fetching in our build tool? And I agree, we depend on system tools, and those should not use the LD_LIBRARY_PATH-provided libs — I'm currently not sure how to tackle this; the build tool needs to have the spack-provided libs in the env as it not only builds but also runs some binary executables (i.e. tests). We probably didn't encounter this until now, because we might have had luck in terms of ABI compatiblity between spack installed env and bare metal, or previous envs had static linkage, or previous envs provided all system tools via spack. The latter is probably also a possible option. I could also hack a bit and not perform env.set('LD_LIBRARY_PATH', ':'.join(library)), but rather set some 'WAF_CONFIGURE_LD_LIBRARY_PATH` and only do

env = os.environ
env.set('LD_LIBRARY_PATH', env.get('WAF_CONFIGURE_LD_LIBRARY_PATH'))

just before our configure step in line 133. The best solution would be to let waf know LD_LIBRARY_PATH entries only for test execution.

The second issue is that even if we apply our workaround, the package has trouble recognizing boost libraries. See the "Error message" section for more details.

Hey @rzarco, thanks for reporting this. My guess in the meeting was off… and you are right, we depend on many more boost libraries. We have depends_on(Boost.with_default_variants) in ebrains-spack-builds/packages/pynn-brainscales/package.py which should be defined in the boost package: https://github.com/spack/spack/blob/v0.18.1/var/spack/repos/builtin/packages/boost/package.py#L77 and for example also enable +system… I'm not sure what happened, but we could try to explicitly mention all the default variants from boost in the row below, i.e.

depends_on('boost@1.69.0: +atomic+chrono+coroutine+context+date_time+exception+filesystem+graph+icu+iostreams+locale+log+math+mpi+numpy+program_options+python+random+regex+serialization+signals+system+test+thread+timer+wave cxxstd=17', type=('build', 'link', 'run'))

— but I do not understand why the line above isn't influencing the concretizer…

Hi Eric,

Thanks for taking a look! Our error messages didn't specify which fetchers were failing, but I'm pretty sure that they were the spack ones. I tried fetching with curl and urllib, both failed with the same error but you could see that the binaries reporting the issue were different in each case.

And about the boost issue... I've just noticed that it is most likely my fault. I wasn't familiar with the depends_on(Boost.with_default_variants) that you mentioned, I usually check the package implementations using spack edit <package>. I compared the package that I have and the one in ebrains-spack-builds/packages/pynn-brainscales/package.py... and you are right.

My local package is version('4.0-a1', branch='waf') and it doesn't have the extra line defining the dependency of Boost's default variants. I will update the package and reconcretize again to see if it defines the proper boost requirements. I will let you know if it's still failing.

I'm sorry for wasting your time with this second "fake" issue, but at least I could see what was going on thanks to your comments.

No problem :) — I'm still working on fixing other build issues (and I test-drive getting rid of the override for LD_LIBRARY_PATH in !243 (merged)).

I believe this one should be fixed in the master branch now — can you check @rzarco?

Sorry to be here so late, but I can confirm that with the newer release, we haven't had to do any patching and it installs fine now (in our Intel cluster, we have to study what happens in ICC).

I.e. done → close?

closed