Needless rebuilds due to python use flag removal [Solved]

Message

proteusx · Post by **proteusx** » Tue Dec 31, 2019 3:35 am

Today portage wants to rebuild more than 300 packages because of the removal of
the python-3.5 use flags. Too much wasted CO2. Greta Thunberg will be in tears.

First emerge packages that really need to be updated

Code: Select all

 emerge -Dua world

Backup the package database in case something goes wrong.

Code: Select all

cp -ar  /var/db/pkg /tmp/pkg-bak

List all rebuilds required by use changes to a file.

Code: Select all

emerge -DupN world > emerge.list

For each package in the list, this script deletes the python_bla_bla_bla_python3_5 entries from IUSE, IUSE_EFFECTIVE and USE in /var/db/pkg/

Code: Select all

#! /bin/sh
P1=python_targets_python3_5
P2=python_single_target_python3_5
P3=python_targets_pypy
P4=python_single_target_pypy

for d in $(awk -F ']' '{gsub(/ /, "", $2); print $2}' emerge.list)
do
  echo $d
  for f in $(ls -1 /var/db/pkg/$d/*USE*)
  do
    sed -i "s/-\?$P1 //; s/-\?$P2 //; s/-\?$P3 //; s/-\?$P4 //" $f
  done
done

And enjoy the result.

Code: Select all

 emerge -DuNa world

Now only 5 packages needed rebuild.
And since all is well we can delete the package database backup.

Post by **Ionen** » Tue Dec 31, 2019 3:39 am

The trick is simply to use -U (--changed-use) instead of -N, --newuse is silly and mostly only useful as a source of information, not something you should actually go through.

--changed-use tries to determine if the change is actually going to change something, that python3.5 support is removed when you weren't using it doesn't fall under that.

wrc1944 · Post by **wrc1944** » Wed Jan 01, 2020 5:05 pm

Ionen,
my original emerge -upDNv @world list from 85 packages which ended with this ( I'm not showing the whole list here), showing 56 rebuilds:

Code: Select all

Total: 85 packages (27 upgrades, 2 in new slots, 56 reinstalls), Size of downloads: 299,645 KiB

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

app-text/poppler:0

  (app-text/poppler-0.84.0:0/94::gentoo, ebuild scheduled for merge) USE="cairo cxx introspection jpeg jpeg2k lcms png qt5 tiff utils -cjk -curl -debug -doc -nss" ABI_X86="(64)" pulled in by
    (no parents that aren't satisfied by other packages in this slot)

  (app-text/poppler-0.82.0:0/92::gentoo, installed) USE="cairo cxx introspection jpeg jpeg2k lcms png qt5 tiff utils -cjk -curl -debug -doc -nss" ABI_X86="(64)" pulled in by
    >=app-text/poppler-0.32:0/92=[cxx,jpeg,lcms,tiff,utils] required by (net-print/cups-filters-1.26.0:0/0::gentoo, installed) USE="dbus foomatic jpeg ldap pdf png postscript tiff -ipp_autosetup -pclm -perl -static-libs -test -zeroconf" ABI_X86="(64)"

AND:
The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by dev-python/pycairo-1.18.2::gentoo
# required by dev-python/pygobject-3.32.1::gentoo[cairo]
# required by app-admin/system-config-printer-1.5.12::gentoo
# required by kde-apps/print-manager-19.12.0::gentoo[gtk]
# required by kde-apps/kdeutils-meta-19.12.0::gentoo[cups]
# required by @selected
# required by @world (argument)

I then tried the "U" instead of "N" option, and it did cut the list to 32 packages, and emerged fine:

Code: Select all

emerge -uDvU -pv @world --keep-going

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] dev-lang/python-exec-2.4.6-r1:2::gentoo  PYTHON_TARGETS="(pypy3) (python2_7) (python3_6) (python3_7) (python3_8) (-jython2_7%*) (-pypy%*) (-python3_5%*)" 0 KiB
[ebuild     U  ] media-libs/giflib-5.2.1-r1:0/7::gentoo [5.2.1:0/7::gentoo] USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U  ] sys-devel/binutils-config-5.2::gentoo [5.1-r1::gentoo] 0 KiB
[ebuild     U  ] dev-libs/vala-common-0.46.5::gentoo [0.44.9::gentoo] 3,297 KiB
[ebuild     U  ] sys-libs/libcap-2.29-r3::gentoo [2.28-r1::gentoo] USE="pam (split-usr) -static-libs" ABI_X86="(64) -32 (-x32)" 96 KiB
[ebuild     U  ] app-office/libreoffice-l10n-6.3.4.2::gentoo [6.3.3.2::gentoo] USE="-offlinehelp" L10N="-af -am -ar -as -ast -be -bg -bn -bn-IN -bo -br -brx -bs -ca -ca-valencia -cs -cy -da -de -dgo -dz -el -en -en-GB -en-ZA -eo -es -et -eu -fa -fi -fr -ga -gd -gl -gu -gug -he -hi -hr -hu -id -is -it -ja -ka -kk -km -kmr-Latn -kn -ko -kok -ks -lb -lo -lt -lv -mai -mk -ml -mn -mni -mr -my -nb -ne -nl -nn -nr -nso -oc -om -or -pa -pl -pt -pt-BR -ro -ru -rw -sa -sat -sd -si -sid -sk -sl -sq -sr -sr-Latn -ss -st -sv -sw-TZ -ta -te -tg -th -tn -tr -ts -tt -ug -uk -uz -ve -vi -xh -zh-CN -zh-TW -zu" 0 KiB
[ebuild  NS    ] sys-kernel/gentoo-sources-5.4.7:5.4.7::gentoo [5.2.16:5.2.16::gentoo, 5.4.2:5.4.2::gentoo, 5.4.3:5.4.3::gentoo, 5.4.4:5.4.4::gentoo, 5.4.5:5.4.5::gentoo, 5.4.6:5.4.6::gentoo] USE="experimental -build -symlink" 280 KiB
[ebuild     U  ] sys-devel/libtool-2.4.6-r6:2::gentoo [2.4.6-r5:2::gentoo] USE="-vanilla" 0 KiB
[ebuild   R    ] dev-libs/libnatspec-0.3.0::gentoo  USE="-doc -python" PYTHON_SINGLE_TARGET="python2_7*" PYTHON_TARGETS="python2_7" 0 KiB
[ebuild     U  ] dev-libs/fribidi-1.0.8::gentoo [1.0.7::gentoo] USE="-static-libs" ABI_X86="(64) -32 (-x32)" 2,029 KiB
[ebuild     U  ] dev-libs/re2-0.2019.12.01:0/gentoo-2019-01-01::gentoo [0.2019.09.01:0/gentoo-2019-01-01::gentoo] USE="icu" ABI_X86="(64) -32 (-x32)" 383 KiB
[ebuild     U  ] dev-util/meson-0.52.1::gentoo [0.52.0::gentoo] USE="-test" PYTHON_TARGETS="python3_6 -python3_7 (-python3_5%)" 1,473 KiB
[ebuild   R    ] net-wireless/bluez-5.52:0/3::gentoo  USE="cups mesh obex readline udev -btpclient -debug -deprecated -doc -experimental -extra-tools -midi (-selinux) -systemd -test -test-programs -user-session" ABI_X86="(64) -32 (-x32)" PYTHON_SINGLE_TARGET="python2_7*" PYTHON_TARGETS="python2_7" 0 KiB
[ebuild     U  ] dev-lang/python-2.7.17-r1:2.7::gentoo [2.7.17:2.7::gentoo] USE="bluetooth gdbm ipv6 ncurses readline sqlite ssl (threads) (wide-unicode) xml (-berkdb) -build -examples -hardened -libressl -tk -wininst (-doc%)" 15 KiB
[ebuild     U  ] dev-python/pygments-2.5.2::gentoo [2.4.2::gentoo] USE="-doc -test" PYTHON_TARGETS="python2_7 python3_6 -pypy3 -python3_7 -python3_8 (-pypy%) (-python3_5%)" 19,790 KiB
[ebuild     U  ] dev-python/mako-1.1.0::gentoo [1.0.7::gentoo] USE="-doc -test" PYTHON_TARGETS="python2_7 python3_6 -pypy3 -python3_7 -python3_8% (-pypy%) (-python3_5%)" 454 KiB
[ebuild     U  ] app-text/liblangtag-0.6.3::gentoo [0.6.2::gentoo] USE="-debug -doc -introspection -static-libs -test" 738 KiB
[ebuild  NS    ] dev-lang/vala-0.46.5:0.46::gentoo [0.44.9:0.44::gentoo] USE="-test -valadoc" 0 KiB
[ebuild     U  ] app-crypt/rhash-1.3.9::gentoo [1.3.8::gentoo] USE="nls ssl -debug -libressl -static-libs" ABI_X86="(64) -32 (-x32)" 394 KiB
[ebuild     U  ] app-editors/nano-4.7::gentoo [4.6::gentoo] USE="magic ncurses nls spell (split-usr) unicode -debug -justify -minimal -slang -static" 2,909 KiB
[ebuild     U  ] dev-util/gtk-update-icon-cache-3.24.13::gentoo [3.24.11::gentoo] 89 KiB
[ebuild     U  ] dev-libs/double-conversion-3.1.5:0/3::gentoo [3.1.4-r1:0/3::gentoo] USE="(-static-libs%) (-test%)" 6,860 KiB
[ebuild     U  ] dev-db/mariadb-connector-c-3.1.6:0/3::gentoo [3.1.5:0/3::gentoo] USE="curl ssl -gnutls -kerberos -libressl -static-libs -test" ABI_X86="(64) -32 (-x32)" 726 KiB
[ebuild     U  ] dev-qt/qtgui-5.14.0-r1:5/5.14::gentoo [5.14.0:5/5.14::gentoo] USE="dbus egl gif jpeg libinput png udev xcb -accessibility -debug -eglfs -evdev (-gles2) -ibus -test -tslib -tuio -vnc -wayland" 0 KiB
[ebuild  r  U  ] app-text/poppler-0.84.0:0/94::gentoo [0.82.0:0/92::gentoo] USE="cairo cxx introspection jpeg jpeg2k lcms png qt5 tiff utils -cjk -curl -debug -doc -nss" 1,525 KiB
[ebuild     U  ] x11-libs/gtk+-3.24.13:3::gentoo [3.24.11:3::gentoo] USE="X cups introspection (-aqua) -broadway -cloudprint -colord -examples -gtk-doc -test -vim-syntax -wayland -xinerama" ABI_X86="(64) -32 (-x32)" 22,719 KiB
[ebuild     U  ] app-crypt/gcr-3.34.0-r1:0/1::gentoo [3.34.0:0/1::gentoo] USE="gtk introspection vala -debug -test" 0 KiB
[ebuild  rR    ] net-print/cups-filters-1.26.0::gentoo  USE="dbus foomatic jpeg ldap pdf png postscript tiff -ipp_autosetup -pclm -perl -static-libs -test -zeroconf" 0 KiB
[ebuild     U  ] x11-terms/xfce4-terminal-0.8.9.1::gentoo [0.8.8::gentoo] USE="-utempter" 922 KiB
[ebuild     U  ] app-portage/portage-utils-0.83::gentoo [0.82::gentoo] USE="nls openmp qmanifest qtegrity -libressl -static" 1,730 KiB
[ebuild     U  ] app-office/libreoffice-6.3.4.2::gentoo [6.3.3.2::gentoo] USE="bluetooth branding cups dbus gtk kde ldap mariadb -accessibility -coinmp -debug -eds -firebird -googledrive -gstreamer -gtk2 -java -odk -pdfimport -postgres -test" LIBREOFFICE_EXTENSIONS="-nlpsolver -scripting-beanshell -scripting-javascript -wiki-publisher" PYTHON_SINGLE_TARGET="python3_6 -python3_7 -python3_8% (-python2_7%) (-python3_5%)" PYTHON_TARGETS="python3_6 -python3_7 -python3_8% (-python2_7%*) (-python3_5%)" 233,227 KiB
[ebuild     U  ] kde-frameworks/plasma-5.65.0-r1:5/5.65::gentoo [5.65.0:5/5.65::gentoo] USE="X -debug -doc (-gles2) -test -wayland" 0 KiB

Total: 32 packages (26 upgrades, 2 in new slots, 4 reinstalls), Size of downloads: 299,645 KiB

The following packages are causing rebuilds:

  (app-text/poppler-0.84.0:0/94::gentoo, ebuild scheduled for merge) causes rebuilds for:
    (net-print/cups-filters-1.26.0:0/0::gentoo, ebuild scheduled for merge)

>=dev-python/setuptools-42.0.2 python_targets_python3_7

A subsequent emerge -upDNv @world listed at the end this:

Code: Select all

Total: 55 packages (2 upgrades, 53 reinstalls), Size of downloads: 0 KiB

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by dev-python/pycairo-1.18.2::gentoo
# required by app-admin/system-config-printer-1.5.12::gentoo
# required by kde-apps/print-manager-19.12.0::gentoo[gtk]
# required by kde-apps/kdeutils-meta-19.12.0::gentoo[cups]
# required by @selected
# required by @world (argument)
>=dev-python/setuptools-42.0.2 python_targets_python3_7
# required by dev-python/setuptools-42.0.2::gentoo
# required by dev-python/markupsafe-1.1.1::gentoo
# required by dev-python/mako-1.1.0::gentoo
# required by media-libs/mesa-19.3.1::gentoo
# required by app-office/libreoffice-6.3.4.2::gentoo[gtk]
# required by @selected
# required by @world (argument)
>=dev-python/certifi-2019.11.28 python_targets_python3_7

The 53 reinstalls were removinf pypy and python3_5 support, and eselect python now lists:

Code: Select all

gentoo-main ~ # eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.8 (fallback)
  [2]   python3.7 (fallback)
  [3]   python3.6 (fallback)
  [4]   python2.7 (fallback)

Should I go ahead and make the two package.use entries, and run the 55 package emerge @ world rebuilds, and does this mean that on ~arch systems more and more of these 3.7 requirements will be showing up? Also, do we still set 3.6 as default, and why would eselect show all verions as fallbacks?

Post by **asturm** » Wed Jan 01, 2020 5:14 pm

Once again, eselect has nothing to do with what is used for emerge. It is probably not correctly set right now. Choose one implementation, and only the other ones should remain as fallback.

Check for stray python_targets* entries in your package.use. Use -N if you want to have useless rebuilds or if you want to globally switch on a new target, it is completely up to you.

Post by **Ionen** » Wed Jan 01, 2020 5:17 pm

Do you actually need to use 3.7 and 3.8 targets? Generally you're better off _not_ setting PYTHON_TARGETS and especially not PYTHON_SINGLE_TARGET anywhere (neither package.use and especially not make.conf), likewise for the USE form, and let gentoo decide what you need.

Currently no packages _need_ 3.7/3.8 in ::gentoo, unless you set it somewhere and then dependencies start wanting it as well.

And yeah, eselect is more personal preference (like a default for your local script at ~/bin or whatever). Packages will use appropriate versions as needed.

wrc1944 · Post by **wrc1944** » Wed Jan 01, 2020 5:33 pm

Thanks guys,
All I have in package.use is:

Code: Select all

dev-python/pycairo python_targets_python3_7
dev-util/gtk-doc python_single_target_python2_7
app-text/asciidoc python_single_target_python2_7

which are from recent emerge requests on @world updates.
And emerge -upDNv @world says

Code: Select all

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by dev-python/pycairo-1.18.2::gentoo
# required by app-admin/system-config-printer-1.5.12::gentoo
# required by kde-apps/print-manager-19.12.0::gentoo[gtk]
# required by kde-apps/kdeutils-meta-19.12.0::gentoo[cups]
# required by @selected
# required by @world (argument)
>=dev-python/setuptools-42.0.2 python_targets_python3_7
# required by dev-python/setuptools-42.0.2::gentoo
# required by dev-python/markupsafe-1.1.1::gentoo
# required by dev-python/mako-1.1.0::gentoo
# required by media-libs/mesa-19.3.1::gentoo
# required by app-office/libreoffice-6.3.4.2::gentoo[gtk]
# required by @selected
# required by @world (argument)
>=dev-python/certifi-2019.11.28 python_targets_python3_7

[/code]

Post by **asturm** » Wed Jan 01, 2020 5:35 pm

wrc1944 wrote: All I have in package.use is:
Code: Select all
dev-python/pycairo python_targets_python3_7
which are from recent emerge requests on @world updates.
And emerge -upDNv @world says...

That is bogus and the reason for emerge asking for more package.use changes. Don't do that.

And the other two entries wouldn't be necessary either if you didn't have the make.conf entries set back then, I suspect.

wrc1944 · Post by **wrc1944** » Wed Jan 01, 2020 5:47 pm

Thanks much, asturm.
Checked, and found targets 2.7 and 3.6

Code: Select all

emerge setuptools dev-python/certifi -pv

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] dev-python/setuptools-42.0.2::gentoo  USE="-test" PYTHON_TARGETS="python2_7 python3_6 -pypy3 -python3_7 -python3_8 (-pypy%) (-python3_5%)" 0 KiB
[ebuild   R    ] dev-python/certifi-2019.11.28::gentoo  PYTHON_TARGETS="python2_7 python3_6 -pypy3 -python3_7 -python3_8 (-pypy%) (-python3_5%)" 0 KiB

Total: 2 packages (2 reinstalls), Size of downloads: 0 KiB

removed those three entries in package.use, and emerge -upDNv @world no longer wants

Code: Select all

>=dev-python/setuptools-42.0.2 python_targets_python3_7
>=dev-python/certifi-2019.11.28 python_targets_python3_7

figueroa · Post by **figueroa** » Thu Jan 02, 2020 3:15 am

asturm wrote:Check for stray python_targets* entries in your package.use. Use -N if you want to have useless rebuilds or if you want to globally switch on a new target, it is completely up to you.

Why doesn't the Gentoo Handbook contain this advice? It reads:

Code: Select all

If the USE settings of the system have been altered, it is recommended to add --newuse as well. Portage will then verify if the change requires the installation of new packages or recompilation of existing ones:
root #emerge --update --deep --with-bdeps=y --newuse @world

Is wheter to use -N or -U controversial among developers? What's the practical disadvantage of using -U instead of -N?

Post by **asturm** » Thu Jan 02, 2020 9:09 am

You're just reading it wrong, here's the important part in bold:

figueroa wrote:If the USE settings of the system have been altered

figueroa · Post by **figueroa** » Thu Jan 02, 2020 4:42 pm

asturm wrote:You're just reading it wrong, here's the important part in bold:
figueroa wrote:If the USE settings of the system have been altered

I'm not saying you are wrong, though I've got 15 year habit of using newuse on my many Gentoo installations. That didn't start by accident.

I don't think the handbook ever mentions changed-use. Where does a user find an authoritative recommendation on how to optimally keep an installation up-to-date? The forum is wonderful, but I consider the handbook(s) authoritative.

The manual for emerge implies the advantage of changed-use. I've seen the recommendation to use changed-use instead of newuse before in the forum. However, most guides instruct users to use -N (newuse) as part of routine maintenance.

Post by **Ionen** » Thu Jan 02, 2020 5:05 pm

figueroa wrote:I don't think the handbook ever mentions changed-use. Where does a user find an authoritative recommendation on how to optimally keep an installation up-to-date? The forum is wonderful, but I consider the handbook(s) authoritative.

For what it's worth, the gentoo cheat sheet does mention it..kinda? I feel like it should be saying --changed-use instead of --quiet here

Avoid unnecessary rebuilds when USE flags only get added to or dropped from the repository and use the --quiet flag for quicker execution:
Code: Select all
root #emerge --update --changed-use --deep --quiet @world

But yeah, it would probably be good for the handbook to update this part, honestly I find the scenarios where --newuse is useful to be very rare. It's more informational, like to see new/removed flags, than something you should actually do when you didn't change any of those flags.. which changed-use would pickup if you did (without also picking up all those removed python3_5 support for nothing). It shouldn't even be mentioned in the handbook in my opinion and straight up replaced with --changed-use.