Crossdev bug, How to build 2 toolchains for glib AND newlib

[hackernotcracker]

I'm building a cross compiler ... but there is a problem with a bug (or three) in the crossdev script when accepting four part target <tuples> including a C library name.
If you don't need the gory details, but want an overview of my problem and crossdev bugs, scroll down to the second post of this thread.

------------------------------------------
My question, in very gory detail:

My cross compiler needs to link with newlib rather than glibc (sometimes) and glibc the rest of the time.
My build and host machine is an x86_64 platform, though I don't think that is the cause of the Gentoo scripts failing to compile the newlib library.
The processor I'm trying to build a toolchain for is the ultra low power arm family:
armv6m-softfloat-eabi-newlib and armv6m-softfloat-eabi-glib
This chipset is a microcontroller, typically used for bluetooth devices, but which can run a super tiny linux-like unix kernel.

Typing crossdev by itself on the command line will result in crossdev asking for a target tuple of ARCH-VENDOR-OS-LIBC , a four name target prefix convention.
But apparently the four name method is untested as it's very buggy.

for example, I tried:
crossdev armv6m-softfloat-eabi-newlib

But crossdev failed to even build the compiler, although it would build the compiler if I executed the command:

crossdev armv6m-softfloat-eabi

But the shorter name doesn't allow me to make two separate toolchains with different default C libraries.
(I'm a noob to Gentoo, so if I'm overlooking something, let me know.)

The reason for failure to compile the longer name appears to be that the gnu autoconfigure scripts don't recognize all versions of C libraries as vaild target names. (Even though C libraries aren't really part of gcc, itself; and should be separately validated. ) Gnu autoconfig chokes on an arbitrary library name is appended to a target, and returns error codes. So, the gnu configure scripts simply don't like gentoo's extended names. This problem, theoretically, can be worked around by changing the configuration switches passed to autoconfigure by ebuild/eclass scripts; Changing configuration switches does work for binutils, but there's a bug in gcc autotools that makes compiling with a four name prefix impossible. ( I suspect this is why the gentoo scripts were never fully tested before releasing, as the problem is very hard to work around. )

I debugged gcc autoconfigure, created a user patch for gcc in gentoo, and then modified ebuild gentoo scripts to pass two swtiches; eg: an abbreviated --target name, and a full length --program-prefix to properly name the binaries with the C library name and pass the --target test. Using two configure switches does fix the bug in gcc's autoconfig, and allows building both gcc and binutils binaries with a four name prefix. So I was quite ecstatic when I got a good compiler from this fix.

The bug in gcc autoconfigure is documented, here, by me;
https://stackoverflow.com/questions/50850313/autotools-gccs-makefile-in-has-a-bug-7-3-0-and-earlier-how-to-re-program-au
I intend to talk with the gcc developers about it later, and see if they will fix the autoconfigure bug permanently.

The patch I came up with is largely gcc version independent, because the autoconfigure scripts in question are hand written ones.
Only the line numbers occasionally change in the "configure" scripts. ( an updated patch for gcc-8.2.0 is here: https://forums.gentoo.org/viewtopic-p-8256364.html#8256364 see the Wednesday Aug29, 2018 post. )
However, the compiler is not usable (for some reason) to compile most of Gentoo's packages.

Part of the problem "seems" to be that several inconsistent (undocumented) and redundant places exist for the binaries of cross compilers and C libraries.
The rest of the problem is that the toolchain naming convention and assumptions change in a lot of places, making computing path names difficult to impossible.
I am unsure of the philosophy of the gentoo crossdev programmers when reading ebuild source code.
How can I get in touch with the programmers responsible for making crossdev?

What is the relationship of /usr/libexec/gcc/arm* vs. /usr/bin/arm-* vs. /usr/arm-* etc... there are so many places where cross compiler binaries and links show up!
Edit: ( I found out that /usr/arm-* is the target's root staging directory. So that directory, I understand now. I still don't understand the relationship of the other two.)

Secondly, based on my experience:
A C compiler is traditionally only linked with one default C library ; I assume GCC isn't optimized to compile for a "use case" of one gcc binary but multiple C libraries. eg: because gcc documentation suggests that gcc partially decides how to compile and optimize itself based on which C library is used. I'd rather just not take a chance on conflicts. For now, I'd like to make two completely binary independent compiler toolchains. Although, maybe Gentoo developers will want to go in another direction ... and I'd like to collaborate and not waste effort. I almost have everything working, BUT... today I discovered that there are two newlib libraries; one newlib library comes with gcc's source code, and another library is a standalone package under Gentoo.

Even though crossdev properly recognizes that I WANT to build a newlib library .. it ignores the one bundled with gcc, and is incapable of compiling the one which gentoo supplies.
This is true regardless of how I've tried to get crossdev to make a toolchain.

I am not sure what the intention of the authors of crossdev is... which library really needs to be built, and under what conditions?

With my patches, Gentoo does build armv6m-softfloat-eabi-newlib-gcc , armv6m-softfloat-eabi-newlib-as, and all other toools (Only stage -s0 and -s1, but -s3 fails. ) . eg: Crossdev can't make a crt0 library with Gentoo's standalone newlib ; but it does make a crt library with gcc's newlib,

I'd appreciate some advice on how to proceed.
In order to reproduce my situation, the rest of this post is dedicated to my patches which enabled me to compile gcc with newlib in the prefix.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



First; I pre-set the toolchain use flags before crossdev is even run, to minimize what is being built.
Once I get newlib to compile, I'll probably change the use flags to more reasonable settings, but this simplifies debugging Gentoo scripts.

FILE: /etc/portage/packages.use/cross-armv6m-softfloat-eabi-newlib

[hackernotcracker]

A shorter version of the opening question, asking it in a generic way:

Gentoo's crossdev script is supposed to create an overlay for various cross compiler toolchains, automatically.
There appear to be possible conflicts in the way the packages are installed, that can potentially corrupt the integrity of a cross-compiler toolchain.
These conflicts are what I'm trying to understand (I'm a gentoo noob in 2018, but a programming vetran on other linux systems. I learn really fast. )
I barely understand slots on Gentoo, but crossdev seems to be setting them up wrong .... ??

IF crossdev is run with no arguments, it prints a help message saying "Target (-t) takes a tuple ARCH-VENDOR-OS-LIBC"
This target <tuple> is the default argument when -t is not specified, so a naiive attempt is to do:

crossdev armv6b-softfloat-eabi-newlib # This doesn't work.
crossdev armv6b-softfloat-eabi # This does work, and defaults to newlib; but doesn't add the name for newlib to the toolchain...

Crossdev then creates an overlay in a local portage repository which uses the supplied name as a directory/container. ( on my system it's: /var/db/repos/crossdev/cross-armv6m-softfloat-eabi )

Inside the overlay directory, crossdev will create links to the original portage repository for C compiler source code:
For crossdev -s1 armv6m-softfloat-eabi , the following overlay directory was made (notice newlib is not appended to the tuple name, for this to work without my patches! )

[mu0n]

While I don't have an answer to your problem, I think the problem I would like to solve is related to yours...

Clearly, each crossdev maps to a specific PORTAGE_CONFIGROOT and represents, effectively, a toolchain AND a (potential) installation of Gentoo under at the location of PORTAGE_CONFIGROOT. However each crossdev target only maps to one PORTAGE_CONFIGROOT --which is limiting when one needs to make multiple, differently configured roots for the same type of target. I have a need for multiple aarch64-unknown-linux-gnu and for multiple x86_64-pc-linux_gnu PORTAGE_CONFIGROOTs (i.e., with different make.conf and make.profile settings) even though the toolchains themselves are fundamentally the same.

So I thought maybe I could use the VENDOR portion of the 4-tuple target to do this? Due to poor parsing of the target name and/or due to various packages/tools various assumptions of what the targets "should" be, one discovers when cross-emerging (e.g., some @system packages) a few packages fail and they fail because they how they interpret the target string.

For example, with one cross-emerge, a package (for a x86_64-*-linux-gnu target) had "aarch64" being used and that clearly failed to build.

To me, the "fix" for cross compiling (which is messy no matter how it is done) involves:

A) The target names must be precise, well-defined and must ALWAYS be properly parsed. No short forms as these are typically ambiguous for parsing and require humans to manually fix/hack a "solution" in. If something is not recognized, e.g., in the VENDOR field, there should be a fallback value for it, e.g., "unknown", so things can continue on using that architecture (+ a message to that effect output somewhere). No target processing should result in an x86_64 target being treated as an aarch64 one. When one doesn't care about a field within the target name, then empty, e.g., have no text such as x86_64--linux-gnu (notice the --).

B) With respect to crossdev the target really needs to be a 5-tuple in general: ARCH-VENDOR-OS-LIBC-SYSTEM_NAME where "SYSTEM_NAME" is a name that maps to that target's build root. Such should depend on the 4-tuple ARCH-VENDOR-OS-LIBC target toolchain which already works (and there is no need to keep building additional toolchains when one already has one!). This would allow one to have multiple installations (by default Gentoo but they need not be) under any given ARCH-VENDOR-OS-LIBC without touching the VENDOR field, etc. because that is a non-Gentoo specific mess.

In the interim, I am considering investigating if patching target parsing bugs is feasible (ugh!), or, if I should wrap/rework crossdev to do its work using a 5-tuple. I am thinking the latter would be more time-efficient and fruitful.

Any thoughts on this?

[mu0n]

I missed mentioning this...

To handle the original poster's problem the 5-tuple would need a separate toolchain OR for the 4-tuple system to be able to have multiple toolchains. The latter is more complicated than the former so, perhaps, the 5-tuple should, for now, have its own toolchain system at the expense of some disk space and initial compilation time.

[NeddySeagoon]

hackernotcracker,

Post a bug with your patches at bugs.gentoo.org. The Gentoo toolchain team are more likely to see a bug than a forum post.

Join #gentoo-toolchain on freenode to talk to them. You will need a registered freenode nick. Be patient, there is only a few of them.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.