Google protobuf Python libraries import fails [SOLVED]

Message

Troopo · Post by **Troopo** » Sat Dec 12, 2020 11:14 pm

Hi,

I'm trying to build a project that uses protobuf and python-protobuf as dependencies (https://github.com/thestr4ng3r/chiaki), i have both packages emerged but the compiler is still not happy about the import.

Here is the error:

Code: Select all

        *************************************************************
         *** Could not import the Google protobuf Python libraries ***
         *** Try installing package 'python-protobuf' or similar.  ***
         *************************************************************
    
Traceback (most recent call last):
  File "/nanopb/generator/nanopb_generator.py", line 22, in <module>
    import protobuf.text_format as text_format
ModuleNotFoundError: No module named 'protobuf'
make[2]: *** [lib/protobuf/CMakeFiles/chiaki-pb.dir/build.make:82: lib/protobuf/takion.pb.c] Error 1
make[1]: *** [CMakeFiles/Makefile2:382: lib/protobuf/CMakeFiles/chiaki-pb.dir/all] Error 2
make: *** [Makefile:183: all] Error 2

My assumption is it's either:
Package names are mixed up i.e. it's looking for google.protobuf while the gentoo version is not named google.
or
I've seen some places that mention "pip install googlle" to install the google meta package, not sure what is the gentoo equivalent of that.

No idea how to tackle this problem further, any ideas?

Naib · Post by **Naib** » Sat Dec 12, 2020 11:26 pm

have you tried: emerge -av dev-python/protobuf-python

Post by Hu » Sat Dec 12, 2020 11:42 pm

OP wrote that they are emerged. However, I wonder if they were installed for the active Python version. OP: please post the output of eselect python list ; emerge --info dev-libs/protobuf dev-python/protobuf-python and emerge --pretend --verbose for the package that is failing for you.

Troopo · Post by **Troopo** » Sun Dec 13, 2020 8:09 am

Naib wrote:have you tried: emerge -av dev-python/protobuf-python

protobuf-python and protobuf are emerged

Hu wrote:OP wrote that they are emerged. However, I wonder if they were installed for the active Python version. OP: please post the output of eselect python list ; emerge --info dev-libs/protobuf dev-python/protobuf-python and emerge --pretend --verbose for the package that is failing for you.

They are, here is the output:

Code: Select all

Available Python interpreters, in order of preference:
  [1]   python3.7
  [2]   python3.8
  [3]   python3.6 (uninstalled)
  [4]   python2.7
  [5]   python3.9 (fallback)
Portage 3.0.9 (python 3.8.6-final-0, default/linux/amd64/17.1/desktop/plasma/systemd, gcc-9.3.0, glibc-2.32-r2, 5.4.80-gentoo-r1-x86_64 x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-5.4.80-gentoo-r1-x86_64-x86_64-Intel-R-_Core-TM-_i7-2600K_CPU_@_3.40GHz-with-glibc2.2.5
KiB Mem:    16361644 total,   8222928 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Tue, 08 Dec 2020 06:30:01 +0000
Head commit of repository gentoo: e14d3c85a8ff068849f209b25351af96f995df79
sh bash 5.0_p18
ld GNU ld (Gentoo 2.34 p6) 2.34.0
distcc 3.3.3 x86_64-pc-linux-gnu [disabled]
app-shells/bash:          5.0_p18::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18-r4::gentoo, 3.7.9::gentoo, 3.8.6::gentoo, 3.9.0::gentoo
dev-util/cmake:           3.17.4-r1::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/sandbox:         2.20::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.16.2-r1::gentoo
sys-devel/binutils:       2.34-r2::gentoo
sys-devel/gcc:            9.3.0-r1::gentoo
sys-devel/gcc-config:     2.3.2-r1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.32-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-extra-opts: 
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-jobs: 1

Tron
    location: /usr/local/portage
    masters: gentoo

brother-overlay
    location: /var/lib/layman/brother-overlay
    masters: gentoo
    priority: 50

jOverlay
    location: /var/lib/layman/jOverlay
    masters: gentoo
    priority: 50

jorgicio
    location: /var/lib/layman/jorgicio
    masters: gentoo
    priority: 50

nest
    location: /var/lib/layman/nest
    masters: gentoo
    priority: 50

reagentoo
    location: /var/lib/layman/reagentoo
    masters: gentoo
    priority: 50

ricerlay
    location: /var/lib/layman/ricerlay
    masters: gentoo
    priority: 50

stha09
    location: /var/lib/layman/stha09
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/tmp"
USE="X a52 aac acl acpi activities alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli crypt cups dbus declarative dri dts dvd dvdr emboss encode exif ffmpeg flac fortran gdbm gif gpm gstreamer gtk gui iconv icu ipv6 jpeg kde kipi kwallet lcms libglvnd libnotify libtirpc mad minizip mng mp3 mp4 mpeg mtp multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt5 readline sdl seccomp semantic-desktop spell split-usr sqlite ssl startup-notification svg systemd tcpd tiff tk truetype udev udisks unicode upower usb vorbis widgets wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_8" PYTHON_TARGETS="python2_7 python3_8" RUBY_TARGETS="ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

dev-libs/protobuf-3.13.0::gentoo was built with the following:
USE="zlib -emacs -examples -static-libs -test" ABI_X86="(64) -32 (-x32)"
FEATURES="ipc-sandbox qa-unresolved-soname-deps assume-digests ebuild-locks news sfperms binpkg-dostrip binpkg-docompress userpriv distlocks unmerge-logs fixlafiles pid-sandbox multilib-strict binpkg-logs network-sandbox parallel-fetch preserve-libs protect-owned usersync config-protect-if-modified sandbox usersandbox xattr userfetch merge-sync strict unmerge-orphans unknown-features-warn"


dev-python/protobuf-python-3.13.0-r1::gentoo was built with the following:
USE="" ABI_X86="(64)" PYTHON_TARGETS="python3_8 -python3_6 -python3_7 -python3_9"

As for the failing package, it's not a package but rather a straight build from github sources

Post by **fedeliallalinea** » Sun Dec 13, 2020 8:27 am

I suspect that you have wrong python set in eslect python list, set python3.8 as default.

Troopo · Post by **Troopo** » Sun Dec 13, 2020 11:54 am

fedeliallalinea wrote:I suspect that you have wrong python set in eslect python list, set python3.8 as default.

Just tried that, still the same

wjb · Post by **wjb** » Mon Dec 14, 2020 12:18 am

The python package is "google.protobuf". Try running:

Code: Select all

$ python
>>> import google.protobuf

If this fails, protobuf is not installed for your default python.

check the python version that was listed when you started python
compare with the python version(s) protobuf was installed for... "eix protobuf-python" or "equery f protobuf-python"

Troopo · Post by **Troopo** » Mon Dec 14, 2020 3:22 pm

Troopo wrote:
fedeliallalinea wrote:I suspect that you have wrong python set in eslect python list, set python3.8 as default.
Just tried that, still the same

Actually you were right i should have tested the import not the build

wjb wrote:The python package is "google.protobuf". Try running:
Code: Select all
$ python
>>> import google.protobuf
If this fails, protobuf is not installed for your default python.
check the python version that was listed when you started python

compare with the python version(s) protobuf was installed for... "eix protobuf-python" or "equery f protobuf-python"

Thanks! no idea why i didn't think of testing it like that
It seems python 3.7 really didn't have the module while 3.8 did however the issue was that somehow the python file used by the dependency of the project had the import as "import protonbuf" instead of "import google.protonbuf" now i say somehow because i couldn't recreate the situation, i've looked at the commits on both projects and re-ran with both python 3.7 and 3.8 and the import was always "google.protonbuf" so i have no idea which version did it use and how...

Anyway to sum this up, changing the import in the python file to the proper one along with the right python version that has the ebuild fixed the issue and every future build afterwards (somehow)

Post by Hu » Sat Dec 19, 2020 12:48 am

Troopo wrote:It seems python 3.7 really didn't have the module while 3.8 did

The emerge output I requested, which you provided, shows this to be the case:

Troopo wrote:

Code: Select all

dev-python/protobuf-python-3.13.0-r1::gentoo was built with the following:
USE="" ABI_X86="(64)" PYTHON_TARGETS="python3_8 -python3_6 -python3_7 -python3_9"

Your PYTHON_TARGETS tells us that, for this package, only python3.8 support was built. You could have had support for python3.6, 3.7, or 3.9, but you did not include those, probably because you used the profile default and the defaults do not enable those other targets.