libtool failure during package emerge [SOLVED]

[thammer]

I am trying to get an updated "base" of Gentoo working with our product code. Our build system is primarily "native" compilation in a chroot jail representing the x86 32-bit embedded target file system (however, we also cross-compile the kernel and drivers for a 64-bit version of our system).
I am only just starting to understand Gentoo Portage package management and am somewhat overwhelmed by it.

We have not updated in a while- we were on a base of stage3-i686-20180314T214502Z.tar.bz2 with very few (if any) updates to any packages.

I started with stage3-i686-20200527T000231Z.tar.xz as my new base and added our kernel source build (4.14.34-gentoo (update to latest 4.14 gentoo version will (hopefully) wait until everything else is updated...)).
I then worked on building our company software source, adding packages as needed. One key item during this phase was downgrading OpenSSL from 1.1.1g to 1.0.2u (several of our source packages will need to be re-worked for the API changes introduce in 1.1.0 but resources are not available yet to address that).

I next started adding the remaining packages (updated versions) included/used in our current system to support full functionality. Most of the packages have built without issue, but a number (10 that I have identified so far), are failing with libtool issues.

Code: Select all

>>> Failed to emerge dev-libs/libgamin-0.1.10-r5, Log file:
>>> Failed to emerge dev-libs/libev-4.31, Log file:
>>> Failed to emerge app-text/aspell-0.60.8, Log file:
>>> Failed to emerge sys-apps/dbus-1.12.16, Log file:
>>> Failed to emerge dev-libs/libsodium-1.0.18, Log file:
>>> Failed to emerge net-misc/ntp-4.2.8_p14-r2, Log file:
>>> Failed to emerge net-misc/quagga-1.2.4, Log file:
>>> Failed to emerge net-nds/openldap-2.4.50, Log file:
>>> Failed to emerge net-mail/mailutils-3.9, Log file:
>>> Failed to emerge media-libs/gd-2.3.0, Log file:

All of them fail with the initial messages similar to this example:

Code: Select all

make[3]: Entering directory '/var/tmp/portage/net-mail/mailutils-3.9/work/mailutils-3.9/libmailutils/auth'
/bin/sh ../../libtool  --tag=CC   --mode=compile i686-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../..  -I../.. -I../../include  -I../../include -I/libmailutils -DSYSCONFDIR=\"/etc\"  -O2 -march=i686 -pipe -fno-strict-aliasing -Wall -Wdeclaration-after-statement -c -o auth.lo auth.c
../../libtool: line 955: X--tag=CC: command not found
../../libtool: line 988: libtool: ignoring unknown tag : command not found
../../libtool: line 955: X--mode=compile: command not found
../../libtool: line 1122: *** Warning: inferring the mode of operation is deprecated.: command not found
../../libtool: line 1123: *** Future versions of Libtool will require --mode=MODE be specified.: command not found
../../libtool: line 1266: Xi686-pc-linux-gnu-gcc: command not found

and continue from there with a number of additional "X-" items not found (either commands or files/directories). In one case (openldap) tag=CC is not used, in another (aspell) mode=link, but otherwise they appear to start out the same.
Given their similarities, I am hopeful that there is a common root cause in my setup somewhere that once addressed will allow all of these to build properly.

In my searching, I have not found a lot that I understand enough to help me; and nothing recently that indicates these are generally encountered issues.


Continuing with the mailutils example:

Code: Select all

gentooSS / # emerge -pqv '=net-mail/mailutils-3.9::gentoo'
[ebuild  N    ] net-mail/mailutils-3.9  USE="berkdb clients gdbm ipv6 nls pam (split-usr) ssl tcpd threads -bidi -emacs -guile -kerberos -kyotocabinet -ldap -mysql -postgres -python -sasl -servers -static-libs -tokyocabinet" PYTHON_SINGLE_TARGET="python3_7 -python3_6 (-python3_8)" 

Code: Select all

gentooSS / # emerge --info '=net-mail/mailutils-3.9::gentoo'
Portage 2.3.99 (python 3.7.7-final-0, default/linux/x86/17.0, gcc-9.3.0, glibc-2.30-r8, 4.9.76-gentoo-r1 i686)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.9.76-gentoo-r1-i686-Intel-R-_Core-TM-_i7-8665U_CPU_@_1.90GHz-with-gentoo-2.6
KiB Mem:     3624588 total,    326720 free
KiB Swap:    2097148 total,   2091632 free
Timestamp of repository gentoo: Fri, 29 May 2020 00:45:01 +0000
Head commit of repository gentoo: 9e5f0b894af4ad7780998a137656d0835b73213e
sh bash 5.0_p17
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
app-shells/bash:          5.0_p17::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.18::gentoo, 3.7.7-r2::gentoo, 3.8.2-r2::gentoo
dev-util/cmake:           3.16.5::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.33.1-r1::gentoo
sys-devel/gcc:            9.3.0::gentoo
sys-devel/gcc-config:     2.2.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.30-r8::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24
    sync-rsync-extra-opts: 
    sync-rsync-verify-jobs: 1

crossdev
    location: /var/db/repos/localrepo-crossdev
    masters: gentoo
    priority: 10

ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="@FREE"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=i686 -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=i686 -pipe"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=i686 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="C.UTF8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl berkdb bzip2 cli crypt dri fortran gdbm iconv ipv6 libtirpc ncurses nls nptl openmp pam pcre readline seccomp split-usr ssl tcpd unicode x86 xattr zlib" ABI_X86="32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, MAKEOPTS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Any guidance that can be provided will be greatly appreciated!

[mike155]

1. I can't see anything wrong in your make.conf.
2. When I run 'emerge --oneshot mailutils', the same libtool command gets executed:

   Code: Select all

   making all in auth
   make[3]: Entering directory '/var/tmp/portage/net-mail/mailutils-3.9/work/mailutils-3.9/libmailutils/auth'
   /bin/sh ../../libtool  --tag=CC   --mode=compile i686-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../..  -I../.. -I../../include  -I../../include -I/libmailutils -DSYSCONFDIR=\"/etc\"  -march=native -O2 -pipe -fno-strict-aliasing -Wall -Wdeclaration-after-statement -c -o auth.lo auth.c

   So there's nothing wrong with the command itself.
3. ../../libtool (realpath = /var/tmp/portage/net-mail/mailutils-3.9/work/mailutils-3.9/libtool ) is a shell script. It seems to fail in line 955 on your machine.
   
   Please look at this line. Maybe you can find out why the script fails. It could be something like

   Code: Select all

   CC="X"     # instead of CC="/usr/bin/gcc "
   $CC--tag=CC

   which would result in the error message you posted.

[thammer]

1. I can't see anything wrong in your make.conf.
2. ../../libtool (realpath = /var/tmp/portage/net-mail/mailutils-3.9/work/mailutils-3.9/libtool ) is a shell script. It seems to fail in line 955 on your machine.
   
   Please look at this line. Maybe you can find out why the script fails. It could be something like

   Code: Select all

   CC="X"     # instead of CC="/usr/bin/gcc "
   $CC--tag=CC

   which would result in the error message you posted.

The ../../libtool script in each of these failures is generated automatically. I have not checked the others, but in this mailutils example the line of the first failure is parsing the command line options:

Code: Select all

  case $arg in
  -*=*) optarg=`$echo "X$arg" | $Xsed -e 's/[-_a-zA-Z0-9]*=//'` ;;
  *) optarg= ;;
  esac

Line 955 is the first case (-*=*)) and it is processing the first argument from the libtool call (--tag=CC).

I am not seeing $echo defined (in this file anyway; $ECHO is defined...). $Xsed is defined:

Code: Select all

# Sed that helps us avoid accidentally triggering echo(1) options like -n.
Xsed="$SED -e 1s/^X//"

but I am still not grokking what this is supposed to be doing, let alone why it is failing.

[Ionen]

Strange, my own libtool file for mailutils doesn't have that optarg line nor any lowercase '$echo', could you have old outdated libtool / macros hiding somewhere? /usr/local?

Your normal libtool does look up to date so it shouldn't be that unless the installation is broken.

[thammer]

...
could you have old outdated libtool / macros hiding somewhere? /usr/local?

Your normal libtool does look up to date so it shouldn't be that unless the installation is broken.

I am hard-pressed to see how I could have anything "old" lying around. As described above, I started with a Gentoo stage3 tarball and have been unable to identify anything other than libtool scripts created in specific package build directory trees.

I am feeling pretty well completely blocked at this point with no ideas how to resolve or even investigate further...

[GDH-gentoo]

...
could you have old outdated libtool / macros hiding somewhere? /usr/local?

Your normal libtool does look up to date so it shouldn't be that unless the installation is broken.

I am hard-pressed to see how I could have anything "old" lying around.

Gentoo ebuilds that call eautoreconf in their prepare phase might indirectly call libtoolize as a result, and if that happens, some files from the Libtool package are copied to the build directory. In particular, the ebuild for net-mail/mailutils-3.9 happens to do that. One of the files that is copied and ends up being part of the generated 'libtool' script is /usr/share/libtool/build-aux/ltmain.sh. That is the one you should check. You said you use a chroot jail. In that case, you should check the file with that pathname in the chroot jail. The one that belongs to Libtool 2.4.6 contains:

Code: Select all

PROGRAM=libtool
PACKAGE=libtool
VERSION=2.4.6
package_revision=2.4.6
# ...
# Set a version string for this script.
scriptversion=2015-10-04.22; # UTC

My version, just like Ionen's, doesn't contain anything resembling the script snippets you posted.

You know that an ebuild calls eautoreconf and libtoolize when you see something like this in the build log:

Code: Select all

>>> Preparing source in /var/tmp/portage/net-mail/mailutils-3.9/work/mailutils-3.9 ...
...
 * Running eautoreconf in '/var/tmp/portage/net-mail/mailutils-3.9/work/mailutils-3.9' ...
...
 * Running libtoolize --install --copy --force --automake ...

[thammer]

The one that belongs to Libtool 2.4.6 contains:

Code: Select all

PROGRAM=libtool
PACKAGE=libtool
VERSION=2.4.6
package_revision=2.4.6
# ...
# Set a version string for this script.
scriptversion=2015-10-04.22; # UTC

Thank you, thank you, thank you, for that pointer!

Sure enough, /usr/share/libtool/build-aux/ltmain.sh file was "wrong"- it was version 1.5.26!

Looking at all of the copies of ltmain.sh that are in my target, I see that the pam_tacplus that we build from source has the same version (there are a couple other items we build from source that have other versions as well). I am guessing that maybe our build process allows this file to "leak" out and impact these other builds.

I replaced the bad file with the one from stage3 tarball and all of these emerge builds succeeded.

This may also be the cause of our other libtool issue that randomly occurs building our PHP extensions.

I will have to spend a little more time investigating how ltmain.sh gets replaced, but I can work on that at a lower priority.

Once again the community comes through!

[thammer]

Looking at all of the copies of ltmain.sh that are in my target, I see that the pam_tacplus that we build from source has the same version (there are a couple other items we build from source that have other versions as well). I am guessing that maybe our build process allows this file to "leak" out and impact these other builds.

I replaced the bad file with the one from stage3 tarball and all of these emerge builds succeeded.

This may also be the cause of our other libtool issue that randomly occurs building our PHP extensions.

Apparently, my joy was to be short-lived. After working on some of the other issues associated with the update, I came to the knowledge that I needed to add APCu support to my PHP installation. Unfortunately, the emerge of dev-php/pecl-apcu is also failing with the libtool version mismatch and this time I cannot locate a version of the ltmain.sh file that might be the cause.

Code: Select all

gentooSS / # find / -name ltmain.sh -print | xargs grep VERSION=
/var/tmp/portage/dev-php/pecl-apcu-5.1.18/work/php7.4/build/ltmain.sh:VERSION=2.4.6
/usr/share/libtool/build-aux/ltmain.sh:VERSION=2.4.6
/usr/lib/php7.4/lib/build/ltmain.sh:VERSION=2.4.6
/usr/src/pam_tacplus-1.4.1/config/ltmain.sh:VERSION=2.4.6
/usr/src/net-snmp-5.6.2.1/ltmain.sh:VERSION=2.2.6
/usr/src/spectracom/phpext/lafayette/build/ltmain.sh:VERSION=2.4.6
/usr/src/spectracom/phpext/tsync/build/ltmain.sh:VERSION=2.4.6
/usr/src/spectracom/gmock/gtest/build-aux/ltmain.sh:VERSION="2.4.2 Debian-2.4.2-1ubuntu1"
/usr/src/spectracom/gmock/build-aux/ltmain.sh:VERSION="2.4.2 Debian-2.4.2-1ubuntu1"
/usr/src/spectracom/ThirdParty/gpstk/ltmain.sh:VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"

The pam-tacplus version I mentioned above seems to have been an outcome of the bad version in /usr/share/libtool/build-aux/ because rebuilding after I fixed things last week resulted in it having the correct version.

As far as I can tell, the macro_{version|revision}= assignments that should be in the generated libtool, do not come from ltmain.sh (the only occurrences of those variables in ltmain.sh are in the test and error message(s)).

Code: Select all

gentooSS / # grep macro_ /var/tmp/portage/dev-php/pecl-apcu-5.1.18/work/php7.4/build/ltmain.sh
    if test "$package_revision" != "$macro_revision"; then
      if test "$VERSION" != "$macro_version"; then
        if test -z "$macro_version"; then
$progname: definition of this LT_INIT comes from $PACKAGE $macro_version.
$progname: but the definition of this LT_INIT comes from revision $macro_revision.
  generated_by_libtool_version='$macro_version'

So, now I am once again stymied as to how libtool is being genereated without these definitions!

Code: Select all

gentooSS / # grep macro_ /usr/bin/libtool
macro_version=2.4.6
macro_revision=2.4.6
    if test "$package_revision" != "$macro_revision"; then
      if test "$VERSION" != "$macro_version"; then
        if test -z "$macro_version"; then
$progname: definition of this LT_INIT comes from $PACKAGE $macro_version.
$progname: but the definition of this LT_INIT comes from revision $macro_revision.
  generated_by_libtool_version='$macro_version'


gentooSS / # grep macro_ /var/tmp/portage/dev-php/pecl-apcu-5.1.18/work/php7.4/libtool
    if test "$package_revision" != "$macro_revision"; then
      if test "$VERSION" != "$macro_version"; then
        if test -z "$macro_version"; then
$progname: definition of this LT_INIT comes from $PACKAGE $macro_version.
$progname: but the definition of this LT_INIT comes from revision $macro_revision.
  generated_by_libtool_version='$macro_version'

And I am turning to the community for help again...

[GDH-gentoo]

As far as I can tell, the macro_{version|revision}= assignments that should be in the generated libtool, do not come from ltmain.sh

No, they likely come from an Autoconf macro defined in /usr/share/aclocal/ltversion.m4. The one that corresponds to Libtool 2.4.6 contains:

Code: Select all

# serial 4179 ltversion.m4
# ...
AC_DEFUN([LTVERSION_VERSION],
[macro_version='2.4.6'
macro_revision='2.4.6'
_LT_DECL(, macro_version, 0, [Which release of libtool.m4 was used?])
_LT_DECL(, macro_revision, 0)
])