fix[bazel]: Use ld (and other tools) from binutils

Upstream bazel resorts to looking for tools in /usr/bin/ otherwise.

@andreasmueller reported a failing build of py-jaxlib on JSC:

     283    /usr/bin/ld: /p/software/jurecadc/stages/2023/software/GCCcore/11.3.0/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libgcc.a(_divdi3.o): unable to initialize decompress status for section .debug_info
  >> 284    /p/software/jurecadc/stages/2023/software/GCCcore/11.3.0/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libgcc.a: error adding symbols: File format not recognized

→ the build tool (bazel) seems to select the bare-metal linker, instead of the one from the loaded module → /p/software/jurecadc/stages/2023/software/binutils/2.38-GCCcore-11.3.0 (corresponding gcc is /p/software/jurecadc/stages/2023/software/GCCcore/11.3.0/bin).

My guess ⇒ that is always the case, but in other cases the linker is still capable of linking the libraries provided by gcc — here it seems to much older, so it fails → my proposed fix: make bazel behave like any other tool and try to find ld in the PATH (or via LD).

added 1 commit

• f9f37ba3 - fix[bazel]: Use ld from LD or search in path

Compare with previous version

just in case it helps:

I did a test installation on jusuf (with exactly the same site-config, only changed the system name) and py-jaxlib build failed (I'm not 100% sure it's the same thing as in jureca: I didn't see this error anywhere, but tbh I didn't look closely)

I tried setting GCC_HOST_COMPILER_PREFIX as suggested here and it worked:

    def setup_build_environment(self, env):
        env.prepend_path("GCC_HOST_COMPILER_PREFIX", "/path/to/binutils/bin")

Great find! Works on jureca too as it was the exact same issue.

Should I include a fix into my "jsc_jureca_sitespecific_changes" branch? I'll set it up in a way that won't interfere with installations on other systems.

Is there a reason not to do this in all systems? Since it works, I would say to just fix the py-jaxlib package in master directly.

(btw the reason I haven't already done that is that I'm not sure how to get the binutils prefix from the spec, and haven't had the time to look.. does any of you know how to do this? it's a dependency of the compiler, not the package, so can't do self.spec['binutils'])

Unfortunately, I haven't managed to get the compiler dependencies in python at all. What I did for pynn-brainscales was this, but that's not a bin path either.
In the setup_build_environment-step, there were not even all environment variables available.

My suggestion would be to set the variable for systems where it's needed, like this: https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/-/commit/5aa05f6e2aa81b94dd4fd93cdbcb304d0e7acc37

yep, sure, that would work, but I still think it's best to fix it in the package.. I'll try to find a way to do this later today - worst case scenario, I can always add a depends_on('binutils') :p

Interesting, I would have expected GCC_HOST_COMPILER_PREFIX to also have effects on non-binutils executables, i.e. GCC_HOST_COMPILER_PREFIX should be more of a merged set of paths, instead of just one… however, if it works, it works :) .

I.e. having an external package for binutils in case of, e.g., JSC — but this would not go well with the compilers.yaml as there is already a module loaded, I guess?

@elmath: I'm not sure if the depends_on would actually work, as all variables (like $PATH etc.) would already favor the correct path over /usr/bin/. Bazel is doing weird things there.

@emuller: I actually tried a colon-separated path (as done in $PATH) and it failed. It expects a single path.

@andreasmueller I think the suggestion would be to have the JSC binutils module in the "external" packages list, i.e. the depends_on on this would basically just load the same variables as the module itself. Ah, you mean you would still need the env var explicitly set → yes, at least building C++ with bazel is a mess — it is very hard to inject custom paths (and settings too; so it's also fun to build old stuff that fails with more modern compilers, but which would accept it with some -Wno-option…). However, you could then derive the content of GCC_HOST_COMPILER_PREFIX by looking at spec['binutils'].prefix.bin.

About GCC_HOST_COMPILER_PREFIX, see here - it should probably have a different name

About depends_on('binutils'), I just meant that if we add this we can use spec['binutils'].prefix.bin to set GCC_HOST_COMPILER_PREFIX in setup_build_environment()

spec['binutils'].prefix.bin

Commit/branch updated…

Ahh, now I get what you want to do. That would make sense!

I'll check if it works.

About GCC_HOST_COMPILER_PREFIX, see here - it should probably have a different name

…and it should be a list of search paths and there should be an actual lookup of the tool in the order of items in the list…

@emuller The changes should be in py-jaxlib, not bazel

(Sorry, I easily get into -mode when talking bazel ;)… (…especially when talking to Google people).)

no, it should be in dependents build env… sorry, have to fix…

updated :D (and afk for IRL meeting ;))

Hmm, but technically you are right… this is only needed for C++ builds via bazel — but it is still a "bazel problem" and changing all bazel-based packages seems ugly :/.

@emuller:
The changes you proposed work on jurecadc

amazing

added 1 commit

• d5064817 - fix[bazel]: Use ld (and other tools) from binutils

Compare with previous version

marked this merge request as ready

changed title from Draft: fix[bazel]: Use ld from LD or search in path to fix[bazel]: Use ld (and other tools) from binutils

changed the description

added 1 commit

• cfb2b120 - fix[bazel]: Use ld (and other tools) from binutils

Compare with previous version

merged

mentioned in commit 2374cdd6

OK, fixed in ebrains-23-06. thanks so much for looking into this! :D

mentioned in issue #52 (closed)