I have a couple of "identically" configured devices.
"Identical" means
- almost same `/etc/portage` content with the exception of different MAKEOPTS and binrepo uri
- same world set
Whatever device I am using at the time becomes the "binary package server", and I typically update it once a day (so dependency resolution time is not that bad).
For the other devices, I try to update them once a week (and dependency resolution can be quite slow, but still barely bearable).
Sometimes, I don't have access to a device and don't update it for a while. This is what happened recently with my fastest device (an i9-13900K, which wasn't updated for ~2.5 months) and dependency resolution took >30 mins only for it to fail and me attempt a fix, then wait another >30 min for it to fail again, etc...
I think the main culprits were Haskell and Rust. For Haskell, I just deselected the packages that depended on Haskell temporarily, and, for Rust, I had to increase the backtrack (I have no idea why it thought I needed ~amd64 on Rust and LLVM).
Anyway, the specifics are not the point of this post, and I am not going to question Portage's dependency resolution time (I am sure there are a lot of other posts about it).
My question is: since I already have an "identical" updated system with known working packages which already passed Portage's dependency checking, can I not just use it as a "source of truth" and have the other systems just try to match it? Note that it is not enough for just the versions to match: there is also the issue of rebuilds which are sometimes required for ABI reasons (Boost, Qt, Haskell, kernel modules, etc.).
Now that I am writing this, I am getting an idea of how I could do this. I think it would be enough to check the contents of `/var/db/pkg` on both systems (say 'A' is the updated system and 'B' is the system we want to update).
I think it would be enough to have a simple and dumb check on the full directory names. If A has a package that B does not have, then add it to an "install list". If B has a package that A does not have, then add it to an "uninstall list". If both A and B have the same package, check the modified time to know if it should be added to "install" list (to catch rebuilds due to ABI). Note that since we are doing a simple and dumb check on the full directory name (not considering version of packages), then the "uninstall list" is kind of useless at this stage.
Then you install the packages from the "install list" with something like the `--oneshot --nodeps` options, which should hopefully be fast.
Now, you re-do the `/var/db/pkg` check to get a more proper "uninstall list", which you can use with `--unmerge`.
One problem: since `/var/db/pkg` of A and B are on two different systems, I think you would first need to serialize its content (probably just filenames and `mtime`). Or I guess you could just `tar` one of them to transfer to other system and use that for comparisons..
What do you think of my solution? If I have spare time soon, I will code it up and post it here, so would be good to know if there are obvious mistakes in the logic.
Are there other solutions that already exist?
Slow Dependency Resolution: Using Another System's Pkg List?
Message
For this check, you can tryktoupt wrote:Sometimes, I don't have access to a device and don't update it for a while. This is what happened recently with my fastest device (an i9-13900K, which wasn't updated for ~2.5 months) and dependency resolution took >30 mins only for it to fail and me attempt a fix, then wait another >30 min for it to fail again, etc...
Code: Select all
emerge -pve @world --backtrack=0Code: Select all
emerge -avuDU @worldI did try that (although with `--backtrack=1`), but it was not helpful at all.Josef.95 wrote:For this check, you can tryktoupt wrote:Sometimes, I don't have access to a device and don't update it for a while. This is what happened recently with my fastest device (an i9-13900K, which wasn't updated for ~2.5 months) and dependency resolution took >30 mins only for it to fail and me attempt a fix, then wait another >30 min for it to fail again, etc...This should be much faster, and shows a good helpful output. Is the output ok, then run your normalCode: Select all
emerge -pve @world --backtrack=0update.Code: Select all
emerge -avuDU @world
I can't recall the exact details now, so I won't insist on you believing me.
The long resolution time in that case had something to do with rust subslots. I think it was going back saying that rust-1.86.0-r2 needed rust-1.86.0-r2 which needed rust-1.85.1-r1 and so on (and `--backtrack=0` would not help).
So what I did was 'emerge -1a --with-bdeps=n =dev-lang/rust-1.86.0-r2`, which successfully merged (this was the version already on my updated machine and I already had a binary package built). But, when I tried to update world, I still got the same issue.
I think with lowering the backtrack it was telling me to enable ~amd64 on rust and llvm to install rust 1.87 and llvm 20 (why do I need that?!). I then tried setting the backtrack to 30 and then portage was finally happy (and it did not need to do anything with rust or llvm, so I have no idea why it kept complaining about them...).
This is not normal. Is this device slow in other usages as well? If not, you have something really messy in /etc/portage.ktoupt wrote:This is what happened recently with my fastest device (an i9-13900K, which wasn't updated for ~2.5 months) and dependency resolution took >30 mins
Also, have you considered using the official binhost?
You can look here for ideas for the rest of your questions: https://wiki.gentoo.org/wiki/Binary_package_guide
One thing I imagine is store the packages in a shared location so that all computers can store and use packages from there.
EDIT:
Definitely a mess in /etc/portage or world pollution.ktoupt wrote: I then tried setting the backtrack to 30 and then portage was finally happy (and it did not need to do anything with rust or llvm, so I have no idea why it kept complaining about them...).
Disabling backtrack (setting it to 0) is intended to make portage throw the first error it encounters, rather than backtracking for another solution. Useful for finding problematic packages. If portage backtracks, the solutions are always more complicated.ktoupt wrote: I did try that (although with `--backtrack=1`), but it was not helpful at all.
Best Regards,
Georgi
No, its fast in every other way.logrusx wrote:This is not normal. Is this device slow in other usages as well? If not, you have something really messy in /etc/portage.ktoupt wrote:This is what happened recently with my fastest device (an i9-13900K, which wasn't updated for ~2.5 months) and dependency resolution took >30 mins
I don't think this is abnormal. If you don't update a Gentoo system for a while, then dependency resolution takes a long time, especially if you have Haskell and/or Rust.
If you keep a Gentoo system updated regularly, then a resolution time of >30 min would be very abnormal. Actually, anything more than a minute would be abnormal if you update every day.
I am using musl, and I don't think the official binhost builds for musl. (Plus I have a bunch of patches...). Anyway, the issue is not with building packages or binary packages specifically.logrusx wrote:Also, have you considered using the official binhost?
I think you misunderstood my question. I already have a "binary package host" set up.logrusx wrote:You can look here for ideas for the rest of your questions: https://wiki.gentoo.org/wiki/Binary_package_guide
One thing I imagine is store the packages in a shared location so that all computers can store and use packages from there.
I am just trying to find a way to use the list of packages installed from one updated system on another to skip all this dependency checking stuff. The updated system already has everything figured out.
I really think it is just due to not updating in a while and having Haskell and Rust packages (especially with Rust subslots, which I think are known to cause Portage to slow down--subslots that is)logrusx wrote: EDIT:
Definitely a mess in /etc/portage or world pollution.ktoupt wrote: I then tried setting the backtrack to 30 and then portage was finally happy (and it did not need to do anything with rust or llvm, so I have no idea why it kept complaining about them...).
Disabling backtrack (setting it to 0) is intended to make portage throw the first error it encounters, rather than backtracking for another solution. Useful for finding problematic packages. If portage backtracks, the solutions are always more complicated.ktoupt wrote: I did try that (although with `--backtrack=1`), but it was not helpful at all.
Best Regards,
Georgi
Thanks,
ktoupt
Example with a 5 Ghz CPU: Works for me :)
Code: Select all
emerge -pve @world --backtrack=0
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 5.39 s (backtrack: 0/0).
.....
.....
Total: 1069 packages (1069 reinstalls), Size of downloads: 1.591 KiBI assume your system is not ~2.5 months out of date with Haskell and Rust issues that Portage needs to sort out.Josef.95 wrote:Example with a 5 Ghz CPU:Works for me :)Code: Select all
emerge -pve @world --backtrack=0 These are the packages that would be merged, in order: Calculating dependencies... done! Dependency resolution took 5.39 s (backtrack: 0/0). ..... ..... Total: 1069 packages (1069 reinstalls), Size of downloads: 1.591 KiB
I agree, it works after updating (and resolution is fast if you update everyday).
The issue is with updating systems that have not been updated in a while. In those cases it takes a LONG time. So, skipping that dependency resolution and using a list of packages from another system which already did those updates daily and incrementally without suffering from long resolution times, would be greatly beneficial.
- NeddySeagoon
- Administrator
- Posts: 56085
- Joined: Sat Jul 05, 2003 9:37 am
- Location: 56N 3W
ktoupt,
Dependency resolution does not just calculate what needs to be updated.
It also calculates the update order so you don't end up with a broken system part way thorough the update.
How will you install packages in the right order if you don't do dependency resolution?
Dependency resolution does not just calculate what needs to be updated.
It also calculates the update order so you don't end up with a broken system part way thorough the update.
How will you install packages in the right order if you don't do dependency resolution?
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
That's a good point I hadn't considered. I guess since I will just be installing binary packages, I can just hope that it would just work out (and not use my system during the update)... But yeah, definitely risky. If I ever do implement the idea at the end of my comment, I will make sure to take a btrfs snapshot before trying it.NeddySeagoon wrote:ktoupt,
Dependency resolution does not just calculate what needs to be updated.
It also calculates the update order so you don't end up with a broken system part way thorough the update.
How will you install packages in the right order if you don't do dependency resolution?
I really don't know what to say. For me, I've always had slow dependency resolution times if I haven't updated in a while. Although, this recent >30 min one is probably a new record. Do you have Haskell packages installed and Rust on that system. Also, now that I think about it, there was also the Python 3.12 -> Python 3.13 recently which the ~2.5 month old system had to go through (but I didn't need to do anything manually for that).Hu wrote:I have one system that, for operational reasons, routinely goes long periods between updates. Its dependency resolution is less than a minute. I don't recall the exact timing, but it's fast enough that I have not been motivated to deal with improving it.
Also, for context:
Code: Select all
Packages installed: 2053
Packages in world: 211
Packages in system: 52
Required packages: 2053
Number removed: 0
- NeddySeagoon
- Administrator
- Posts: 56085
- Joined: Sat Jul 05, 2003 9:37 am
- Location: 56N 3W
ktoupt,
Please post your /var/lib/portage/world so we can check for world pollution.
That can cause all sorts of problems until one day portage says "There are no ebuilds to satisfy ... " because the ebuild you need has been removed from the repo.
It also leads to portage keeping things that should have been --depcleaned
Please post your /var/lib/portage/world so we can check for world pollution.
That can cause all sorts of problems until one day portage says "There are no ebuilds to satisfy ... " because the ebuild you need has been removed from the repo.
It also leads to portage keeping things that should have been --depcleaned
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Sure. First, here are the repos I have enabled (other than the main Gentoo repo): guru, haskell, and librewolf.NeddySeagoon wrote:ktoupt,
Please post your /var/lib/portage/world so we can check for world pollution.
That can cause all sorts of problems until one day portage says "There are no ebuilds to satisfy ... " because the ebuild you need has been removed from the repo.
It also leads to portage keeping things that should have been --depcleaned
There is also the cross-x86_64-w64-mingw32 and cross-i686-w64-mingw32 repo. I also have a custom local repo with some stuff.
I was not sure what is meant by world pollution, but I guess ebuilds that are no longer in the repo. The only one I am aware of is media-gfx/transfig, which will be removed soon (or has been recently removed). I will deal with that this weekend hopefully...
Anyway, here is the world set:
Code: Select all
app-admin/doas
app-admin/entr
app-admin/keepassxc
app-admin/metalog
app-admin/stow
app-arch/p7zip
app-arch/sharutils
app-backup/restic
app-containers/podman
app-crypt/efitools
app-crypt/sbsigntools
app-crypt/signify
app-crypt/swtpm
app-editors/emacs
app-emulation/dxvk
app-emulation/qemu
app-emulation/ruffle
app-emulation/virt-viewer
app-eselect/eselect-repository
app-i18n/fcitx-anthy
app-i18n/fcitx-configtool
app-i18n/fcitx-gtk
app-i18n/fcitx-qt
app-i18n/translate-shell
app-i18n/wlanthy
app-misc/anki
app-misc/brightnessctl
app-misc/dragon
app-misc/nnn
app-misc/resolve-march-native
app-misc/rlwrap
app-misc/tmux
app-mobilephone/dfu-util
app-mobilephone/scrcpy
app-office/calcurse
app-office/libreoffice
app-portage/cpuid2cpuflags
app-portage/gentoolkit
app-shells/bash-completion
app-shells/fzf
app-text/OCRmyPDF
app-text/languagetool
app-text/pandoc-cli
app-text/qpdf
app-text/texlive
app-text/xournalpp
app-text/zathura-meta
cross-i686-w64-mingw32/binutils
cross-i686-w64-mingw32/gcc
cross-i686-w64-mingw32/mingw64-runtime
cross-x86_64-w64-mingw32/binutils
cross-x86_64-w64-mingw32/gcc
cross-x86_64-w64-mingw32/mingw64-runtime
dev-debug/gdb
dev-debug/strace
dev-java/ant
dev-lang/tcc
dev-lang/typescript
dev-libs/bemenu
dev-libs/cyrus-sasl-xoauth2
dev-python/affine
dev-python/matplotlib
dev-python/rasterio
dev-python/scikit-image
dev-python/scipy
dev-python/shapely
dev-python/tqdm
dev-scheme/mit-scheme
dev-util/android-tools
dev-util/cloc
dev-util/ctags
dev-util/debugedit
dev-util/pkgdev
dev-util/shellcheck
dev-vcs/git
dev-vcs/git-lfs
games-misc/fortune-mod-bofh-excuses
games-misc/fortune-mod-chucknorris
games-misc/fortune-mod-kernelcookies
games-misc/fortune-mod-norbert-tretkowski
games-misc/fortune-mod-osfortune
games-misc/fortune-mod-taow
games-misc/fortune-mod-zx-error
games-util/game-device-udev-rules
games-util/mangohud
gui-apps/foot
gui-apps/grimshot
gui-apps/kanshi
gui-apps/swaybg
gui-apps/swayidle
gui-apps/swaylock
gui-apps/wl-mirror
gui-apps/wlsunset
gui-libs/xdg-desktop-portal-wlr
gui-wm/sway
mail-client/aerc
mail-mta/msmtp
media-fonts/fonts-meta
media-fonts/freefont
media-fonts/terminus-font
media-gfx/blender
media-gfx/gimp
media-gfx/img2pdf
media-gfx/imv
media-gfx/inkscape
media-gfx/scrot
media-gfx/transfig
media-gfx/ueberzugpp
media-gfx/waifu2x-ncnn-vulkan
media-gfx/xsane
media-gfx/zbar
media-libs/exiftool
media-libs/libva-intel-driver
media-libs/libva-intel-media-driver
media-sound/alsa-utils
media-sound/easyeffects
media-sound/pulsemixer
media-video/ffmpegthumbnailer
media-video/input-overlay
media-video/mpv
media-video/obs-studio
media-video/v4l2loopback
net-analyzer/nmap
net-analyzer/openbsd-netcat
net-analyzer/ssh-audit
net-analyzer/traceroute
net-analyzer/wireshark
net-dialup/picocom
net-dns/dnscrypt-proxy
net-dns/dnsmasq
net-im/chatterino
net-irc/weechat
net-libs/ldns
net-mail/isync
net-misc/dhcpcd
net-misc/iperf
net-misc/networkmanager
net-misc/openntpd
net-misc/yt-dlp
net-news/newsboat
net-p2p/syncthing
net-p2p/transmission
net-print/cups-meta
net-print/hplip
net-proxy/shadowsocks-libev
net-vpn/tor
net-vpn/wireguard-tools
net-wireless/iwd
sci-electronics/fujprog
sci-electronics/ghdl-yosys-plugin
sci-electronics/gtkwave
sci-electronics/iverilog
sci-electronics/nextpnr
sci-libs/composable-kernel
sci-libs/libqalculate
sci-libs/openblas
sci-mathematics/octave
sci-visualization/gnuplot
sys-apps/flatpak
sys-apps/man-pages
sys-apps/man-pages-posix
sys-apps/smartmontools
sys-apps/xdg-desktop-portal-gtk
sys-auth/oath-toolkit
sys-block/io-scheduler-udev-rules
sys-boot/efibootmgr
sys-boot/woeusb-ng
sys-devel/crossdev
sys-devel/gf
sys-firmware/intel-microcode
sys-firmware/sof-firmware
sys-fs/btrfs-progs
sys-fs/cryptsetup
sys-fs/dosfstools
sys-fs/exfat-utils
sys-fs/ncdu
sys-kernel/dracut
sys-kernel/linux-firmware
sys-power/acpid
sys-process/cronie
sys-process/htop
sys-process/lsof
virtual/wine
www-client/librewolf
www-client/lynx
x11-apps/mesa-progs
x11-apps/xhost
x11-apps/xinit
x11-apps/xinput
x11-apps/xrdb
x11-apps/xset
x11-base/xorg-server
x11-misc/arandr
x11-misc/autorandr
x11-misc/dmenu
x11-misc/dunst
x11-misc/hsetroot
x11-misc/i3status
x11-misc/numlockx
x11-misc/picom
x11-misc/redshift
x11-misc/slock
x11-misc/sxhkd
x11-misc/xclip
x11-misc/xss-lock
x11-terms/alacritty
x11-terms/ghostty
x11-terms/st
x11-terms/xterm
x11-themes/gnome-icon-theme-extras
x11-wm/i3Actually, I think if you make the package checking more intelligent by checking versions and by sorting by modified time, this would be less of an issue.ktoupt wrote:That's a good point I hadn't considered. I guess since I will just be installing binary packages, I can just hope that it would just work out (and not use my system during the update)... But yeah, definitely risky. If I ever do implement the idea at the end of my comment, I will make sure to take a btrfs snapshot before trying it.NeddySeagoon wrote:ktoupt,
Dependency resolution does not just calculate what needs to be updated.
It also calculates the update order so you don't end up with a broken system part way thorough the update.
How will you install packages in the right order if you don't do dependency resolution?
- NeddySeagoon
- Administrator
- Posts: 56085
- Joined: Sat Jul 05, 2003 9:37 am
- Location: 56N 3W
ktoupt,
That looks 'mostly harmless' :)
There are a few suspects
These all appear to be libraries, in which case they will be depended on by whatever needs them.
If the dependuncy is missing from the ebuild, thats a bug.
Test as follows ...
to have portage tell what depends on each package.
If its non-zero output,
That will leave the package to be managed by portage and when it's no longer required, it will be --depcleaned.
If portage says that there are no dependences, keep the package in world if you need it for other reasons.
e.g. something installed outside of portage needs it.
That looks 'mostly harmless' :)
There are a few suspects
Code: Select all
dev-libs/bemenu
dev-libs/cyrus-sasl-xoauth2
dev-python/matplotlib
gui-libs/xdg-desktop-portal-wlr
media-libs/exiftool
media-libs/libva-intel-driver
media-libs/libva-intel-media-driver
net-libs/ldns
net-proxy/shadowsocks-libev
sci-libs/composable-kernel
sci-libs/libqalculate
sci-libs/openblas
If the dependuncy is missing from the ebuild, thats a bug.
Test as follows ...
Code: Select all
emerge -cpv <category>/<package>If its non-zero output,
Code: Select all
emerge --deselect <category>/<package>If portage says that there are no dependences, keep the package in world if you need it for other reasons.
e.g. something installed outside of portage needs it.
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
You're misunderstanding the suggestion. With shared location you don't need a binhost. It is your binhost.ktoupt wrote: I think you misunderstood my question. I already have a "binary package host" set up.
You can't skip "dependency checking stuff". This is what keeps your system consistent.ktoupt wrote:I am just trying to find a way to use the list of packages installed from one updated system on another to skip all this dependency checking stuff. The updated system already has everything figured out.
You think what you like but 30 minutes on a 13th generation Intel is way too much. The most I've reached is ~5 minutes back when I had two binhosts enabled. Processing graphs is of M*N complexity where M and N are the number of vertices and edges. Adding a repo increases complexity in a non-linear fashion.ktoupt wrote:I really think it is just due to not updating in a while and having Haskell and Rust packages (especially with Rust subslots, which I think are known to cause Portage to slow down--subslots that is)
Post your emerge --info
p.s. I've always wondered why people complicate their lives with musl et.c. It doesn't give anything in exchange for the trouble.
Best Regards,
Georgi
I use all those for reasons...NeddySeagoon wrote:ktoupt,
That looks 'mostly harmless' :)
There are a few suspectsThese all appear to be libraries, in which case they will be depended on by whatever needs them.Code: Select all
dev-libs/bemenu dev-libs/cyrus-sasl-xoauth2 dev-python/matplotlib gui-libs/xdg-desktop-portal-wlr media-libs/exiftool media-libs/libva-intel-driver media-libs/libva-intel-media-driver net-libs/ldns net-proxy/shadowsocks-libev sci-libs/composable-kernel sci-libs/libqalculate sci-libs/openblas
If the dependuncy is missing from the ebuild, thats a bug.
Test as follows ...
to have portage tell what depends on each package.Code: Select all
emerge -cpv <category>/<package>
If its non-zero output,That will leave the package to be managed by portage and when it's no longer required, it will be --depcleaned.Code: Select all
emerge --deselect <category>/<package>
If portage says that there are no dependences, keep the package in world if you need it for other reasons.
e.g. something installed outside of portage needs it.
Also, most of those are not really libraries.. I don't know why they are in a library category. For example, dev-libs/bemenu provides the bemenu program (replacement for dmenu), media-libs/exiftool provides the exiftool program, net-libs/ldns provides the drill program, sci-libs/libqalculate provides the qalc program.
dev-libs/cyrus-sasl-xoauth2, dev-python/matplotlib I use for other reaons.
sci-libs/composable-kernel already has a bug report opened on some package (not by me). I can't remember which one though, and it still does not need to have been solved. I can depclean it without Portage complaining.
media-libs/libva-intel-driver, media-libs/libva-intel-media-driver, and gui-libs/xdg-desktop-portal-wlr fall into similar categories where they are just implementations for a general API. The Intel drivers are implementations for VA-API, and the wlr portal is an implemention for xdg-desktop-portal. They are detected at runtime and not strictly needed, so I don't think I should open a bug for them.
sci-libs/openblas similar to above where it is an implemention of an API, although it is not really detected at runtime. See <https://wiki.gentoo.org/wiki/Blas-lapack-switch>.
net-proxy/shadowsocks-libev I am not sure why you are pointing this out. It is a system service with its own init scripts. I use it to get around a great firewall which block wireguard.
So you are saying I can put /var/cache/binpkgs/ in an NFS share for example? I don't understand what problem that would solve.logrusx wrote:You're misunderstanding the suggestion. With shared location you don't need a binhost. It is your binhost.ktoupt wrote: I think you misunderstood my question. I already have a "binary package host" set up.
I understand that normally you can't skip dependency checking, but I think if I already have a system with identical configuration which is already updated and gone through the dependency checking, then I should be able to use information from that system to skip Portage's slow dependency resolution. I think my suggestion at the end of my initial post, with refinements thanks to NeddySeagoon, would work. I am not suggesting integrating this into Portage though!logrusx wrote:You can't skip "dependency checking stuff". This is what keeps your system consistent.ktoupt wrote:I am just trying to find a way to use the list of packages installed from one updated system on another to skip all this dependency checking stuff. The updated system already has everything figured out.
You are starting to make me think I am going crazy. It is true! I personally experienced this! Next time I use the system, I will roll-back using btrfs snapshots and post outputs here. Don't expect that anytime soon though... Maybe in a couple of weeks...logrusx wrote:You think what you like but 30 minutes on a 13th generation Intel is way too much. The most I've reached is ~5 minutes back when I had two binhosts enabled. Processing graphs is of M*N complexity where M and N are the number of vertices and edges. Adding a repo increases complexity in a non-linear fashion.ktoupt wrote:I really think it is just due to not updating in a while and having Haskell and Rust packages (especially with Rust subslots, which I think are known to cause Portage to slow down--subslots that is)
Also, dependency resolution is single-threaded, so I am not suprised that it is slow. CPUs haven't really improved single-threaded performance much.
Further, I am not convinced the number of binhosts would affect dependency resolution at all. I think the dependency resolution is the same, and then, at the end, Portage would just check if the packages it needs are in a binhost to merge or if it should compile from source. I haven't read the Portage source code so I can't say for sure.
Similarly, I am not convinced having more repos would non-linearly increase resolution time. Although, yes, I do agree having more repos would be slower.
Again, I haven't read the Portage source code, and I also won't claim I know anything about graph theory or dependency resolution, so I could be wrong.
Sure. Here is emerge --info from an "identical" device, with the definition of "identical" being the one from my first post:logrusx wrote:Post your emerge --info
Code: Select all
Portage 3.0.67 (python 3.13.3-final-0, default/linux/amd64/23.0/musl/hardened, gcc-14, musl-1.2.5-r3, 6.15.2-gentoo-dist-hardened x86_64)
=================================================================
System uname: Linux-6.15.2-gentoo-dist-hardened-x86_64-Intel-R-_Core-TM-_i7-8565U_CPU_@_1.80GHz-with-libc
KiB Mem: 16083328 total, 9626444 free
KiB Swap: 33554428 total, 33554428 free
Timestamp of repository gentoo: Thu, 12 Jun 2025 14:07:23 +0000
Head commit of repository gentoo: 996906d94ab2d1ceeb83d4b0b79fa4dfa0b2ba52
Timestamp of repository guru: Thu, 12 Jun 2025 08:21:48 +0000
Head commit of repository guru: 0d64dddc524dd371bd50bdd3dba28955013b2e97
Timestamp of repository haskell: Wed, 11 Jun 2025 09:07:04 +0000
Head commit of repository haskell: bb2d03b3bc0ca6e132d6463ef5aec9751b8dd5ac
Head commit of repository librewolf: fffef2e4997634eb59179111ce3a4b9841760077
sh dash 0.5.12-r1
ld GNU ld (Gentoo 2.44 p1) 2.44.0
app-misc/pax-utils: 1.3.8::gentoo
app-shells/bash: 5.2_p37::gentoo
dev-build/autoconf: 2.72-r1::gentoo
dev-build/automake: 1.17-r2::gentoo
dev-build/cmake: 3.31.7-r1::gentoo
dev-build/libtool: 2.5.4::gentoo
dev-build/make: 4.4.1-r100::gentoo
dev-build/meson: 1.7.2::gentoo
dev-java/java-config: 2.3.4::gentoo
dev-lang/perl: 5.40.2::gentoo
dev-lang/python: 3.12.10_p2::gentoo, 3.13.3_p2::gentoo
dev-lang/rust: 1.86.0-r2::gentoo
llvm-core/clang: 18.1.8-r6::gentoo, 19.1.7::gentoo
llvm-core/lld: 18.1.8::gentoo, 19.1.7::gentoo
llvm-core/llvm: 18.1.8-r6::gentoo, 19.1.7::gentoo
sys-apps/baselayout: 2.17::gentoo
sys-apps/openrc: 0.56::gentoo
sys-apps/sandbox: 2.46::gentoo
sys-devel/binutils: 2.44-r1::gentoo
sys-devel/binutils-config: 5.5.2::gentoo
sys-devel/gcc: 14.3.0::gentoo
sys-devel/gcc-config: 2.12.1::gentoo
sys-kernel/linux-headers: 6.15::gentoo (virtual/os-headers)
sys-libs/musl: 1.2.5-r3::gentoo
Repositories:
gentoo
location: /var/db/repos/gentoo
sync-type: git
sync-uri: https://github.com/gentoo-mirror/gentoo.git
priority: -1000
volatile: False
sync-git-verify-commit-signature: true
cross-i686-w64-mingw32
location: /var/db/repos/cross-i686-w64-mingw32
masters: gentoo
volatile: False
cross-x86_64-w64-mingw32
location: /var/db/repos/cross-x86_64-w64-mingw32
masters: gentoo
volatile: False
guru
location: /var/db/repos/guru
sync-type: git
sync-uri: https://github.com/gentoo-mirror/guru.git
masters: gentoo
volatile: False
haskell
location: /var/db/repos/haskell
sync-type: git
sync-uri: https://github.com/gentoo-mirror/haskell.git
masters: gentoo
volatile: False
librewolf
location: /var/db/repos/librewolf
sync-type: git
sync-uri: https://codeberg.org/librewolf/gentoo.git
masters: gentoo
volatile: False
mylocalgentoorepo
location: /var/db/repos/mylocalgentoorepo
masters: gentoo
volatile: False
Binary Repositories:
gentoobinhost
priority: 1
sync-uri: rsync://hostname.localdomain/gentoo-x86-64_musl-hardened-binpkg
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-musl"
CFLAGS="-O2 -pipe -fomit-frame-pointer -march=x86-64 -mtune=generic"
CHOST="x86_64-pc-linux-musl"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -fomit-frame-pointer -march=x86-64 -mtune=generic"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--binpkg-changed-deps=n"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe -fomit-frame-pointer -march=x86-64 -mtune=generic"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg buildpkg-live clean-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync merge-wait network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -fomit-frame-pointer -march=x86-64 -mtune=generic"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
INSTALL_MASK="charset.alias /usr/share/locale/locale.alias"
LANG="C.UTF8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs"
LEX="flex"
MAKEOPTS="-j4"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--new-compress"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RUSTFLAGS="-C opt-level=3"
SHELL="/bin/bash"
USE="X a52 aac acl alsa amd64 avif bash-completion blake2 bluetooth brotli bzip2 cdda cdr cet cjk cpudetection crossdev-mingw crypt cups dav1d dbus dist-kernel dri dts dvd dvdr dynamic egl encode eselect-ldso exif exr fdk ffmpeg flac gif gles1 gles2 gles3 hardened harfbuzz heif hip iconv ipv6 jit jpeg jpeg2k jpegxl lame libaom libass libtirpc lto lv2 lz4 lzip lzma lzo mad mng mp3 mp4 mpeg ncurses nls ogg opencl openexr opengl openh264 openmp opus orc pam pango pcre pgo pic pie pipewire png pulseaudio rav1e raw readline rocm screencast seccomp snappy ssl ssp svg svt-av1 test-rust theora tiff truetype udev unicode v4l vaapi vim-syntax vorbis vpx vulkan wavpack wayland webp wmf x264 x265 xattr xcb xft xpm xtpax xxhash zlib zstd" ABI_X86="64" ADA_TARGET="gcc_14" AMDGPU_TARGETS="gfx1031" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 aes pclmul popcnt rdrand sse3 sse4_1 sse4_2 ssse3" ELIBC="musl" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GUILE_SINGLE_TARGET="3-0" GUILE_TARGETS="3-0" INPUT_DEVICES="libinput wacom" KERNEL="linux" L10N="*" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres17" PYTHON_SINGLE_TARGET="python3_13" PYTHON_TARGETS="python3_13" RUBY_TARGETS="ruby32 ruby33" VIDEO_CARDS="amdgpu intel nouveau radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PYTHONPATH, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
I agree with you, musl has no tangible personal benefits in most cases and will probably get in your way. (However, in this case, I do not really see how musl is troubling me. If this is about the binhost, I still don't understand how that is relevant to the problem of slow dependency checking)logrusx wrote:p.s. I've always wondered why people complicate their lives with musl et.c. It doesn't give anything in exchange for the trouble.
I still think things like musl have an overall benefit to the ecosystem as a whole. Diversity is just good in general. Having two implementations of libc and testing software on both of them will make things such as non-standard uses or subtle bugs more obvious. For example, I've seen a use after free bug which crashed on my system but was working fine for everyone else (reported this upstream and they have it fixed now).
In fact, if I could buy a fast enough big-endian system which will not dominiate my energy bill, I would use it just to see what kind of bugs I could shake out. I know it would be more trouble than its worth, but I think thats fine. Not everything needs to be 100% efficient.
I think your remark about musl is somewhat similar to "why use Gentoo when you can just use Debian or Arch".
Single location where you will use binary packages from. You will just run a full world update from any of your devices and not bother with the complications you'll soon find out will come with your initial idea.ktoupt wrote:So you are saying I can put /var/cache/binpkgs/ in an NFS share for example? I don't understand what problem that would solve.logrusx wrote:You're misunderstanding the suggestion. With shared location you don't need a binhost. It is your binhost.ktoupt wrote: I think you misunderstood my question. I already have a "binary package host" set up.
No that won't work, but you're free to bang your head in it. I mean it won't solve your problem (of slow dependency resolution if I finally got it right. BTW you came here with multiple issues/questions, whether you realize it or not).ktoupt wrote:I understand that normally you can't skip dependency checking, but I think if I already have a system with identical configuration which is already updated and gone through the dependency checking, then I should be able to use information from that system to skip Portage's slow dependency resolution. I think my suggestion at the end of my initial post, with refinements thanks to NeddySeagoon, would work. I am not suggesting integrating this into Portage though!logrusx wrote:You can't skip "dependency checking stuff". This is what keeps your system consistent.ktoupt wrote:I am just trying to find a way to use the list of packages installed from one updated system on another to skip all this dependency checking stuff. The updated system already has everything figured out.
It's single threaded everywhere, Intel, AMD and so on. My processor should be slower than yours. At the time I experienced this I had a comparable number of packages. Just below 1600, currently 1605. Although I did a cleanup and had reduced them. But I digress.ktoupt wrote:You are starting to make me think I am going crazy. It is true! I personally experienced this! Next time I use the system, I will roll-back using btrfs snapshots and post outputs here. Don't expect that anytime soon though... Maybe in a couple of weeks...logrusx wrote:You think what you like but 30 minutes on a 13th generation Intel is way too much. The most I've reached is ~5 minutes back when I had two binhosts enabled. Processing graphs is of M*N complexity where M and N are the number of vertices and edges. Adding a repo increases complexity in a non-linear fashion.ktoupt wrote:I really think it is just due to not updating in a while and having Haskell and Rust packages (especially with Rust subslots, which I think are known to cause Portage to slow down--subslots that is)
Also, dependency resolution is single-threaded, so I am not suprised that it is slow. CPUs haven't really improved single-threaded performance much.
Yes you're wrong. Binary packages are additional vertices in the graph. They come with additional edges. It's not so simple as you imagine it. I've peaked in the source code and it's one giant file that handles this stuff. I didn't read it carefully but what I saw as method names et.c. is what I expected to find.ktoupt wrote:Further, I am not convinced the number of binhosts would affect dependency resolution at all. I think the dependency resolution is the same, and then, at the end, Portage would just check if the packages it needs are in a binhost to merge or if it should compile from source. I haven't read the Portage source code so I can't say for sure.
Similarly, I am not convinced having more repos would non-linearly increase resolution time. Although, yes, I do agree having more repos would be slower.
Again, I haven't read the Portage source code, and I also won't claim I know anything about graph theory or dependency resolution, so I could be wrong.
At a first glance, I don't see anything disturbing. But I didn't have such slow dependency resolution even on my Q6600, and even on my dual core equivalent which I don't remember the model for, but it was with E in front.ktoupt wrote:Sure. Here is emerge --info from an "identical" device, with the definition of "identical" being the one from my first post:logrusx wrote:Post your emerge --info
That was just a side note which is totally irrelevant except it prevents you from using ready made binary packages.ktoupt wrote: I think your remark about musl is somewhat similar to "why use Gentoo when you can just use Debian or Arch".
Here's my old thread about the dependency resolution time: viewtopic-t-1171147.html
EDIT: I actually find something off in your emerge --info. To my knowledge haskell overlay it not a thing anymore, but I might be wrong about that. I think they moved most of in in the main tree a while ago. Yes, I used that too and I think it may even have been on my aforementioned old hardware. I needed pandoc.
For comparison:
Code: Select all
~ # ACCEPT_KEYWORDS="~amd64" emerge -pv app-text/pandoc
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 36.82 s (backtrack: 15/20).
...
Total: 237 packages (237 new, 1 uninstall), Size of downloads: 63,029 KiB
Conflict: 3 blocks (all satisfied)
EDIT2: I enabled ::haskell and added ACCEPT_KEYWORDS="~amd64" and -g to make portage think harder. Also added pandoc to the query to make it use packages from ::haskell. Indeed there's noticeable delay, but not that much. Here's what came out:
Code: Select all
~ # ACCEPT_KEYWORDS="~amd64" emerge -DuUpvg @world app-text/pandoc
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 195.95 s (backtrack: 11/20).
...
Total: 581 packages (291 upgrades, 244 new, 15 in new slots, 31 reinstalls, 2 binaries, 2 uninstalls), Size of downloads: 5,524,707 KiB
Conflict: 21 blocks (all satisfied)Georgi
I seriously have no idea what this would solve. I already just do a full world update from any of my devices. The only issue is slow dependency checking on world updates on systems which haven't been updated in a while.logrusx wrote:Single location where you will use binary packages from. You will just run a full world update from any of your devices and not bother with the complications you'll soon find out will come with your initial idea.ktoupt wrote:So you are saying I can put /var/cache/binpkgs/ in an NFS share for example? I don't understand what problem that would solve.logrusx wrote: You're misunderstanding the suggestion. With shared location you don't need a binhost. It is your binhost.
Well, I'm sorry to say you will have to admit you were wrong. Read on...logrusx wrote:No that won't work, but you're free to bang your head in it. I mean it won't solve your problem (of slow dependency resolution if I finally got it right. BTW you came here with multiple issues/questions, whether you realize it or not).ktoupt wrote:I understand that normally you can't skip dependency checking, but I think if I already have a system with identical configuration which is already updated and gone through the dependency checking, then I should be able to use information from that system to skip Portage's slow dependency resolution. I think my suggestion at the end of my initial post, with refinements thanks to NeddySeagoon, would work. I am not suggesting integrating this into Portage though!logrusx wrote: You can't skip "dependency checking stuff". This is what keeps your system consistent.
I just read through that thread. You are the only one insisting that binhost slows down dependency resolution significantly. I do agree with you that there is additional checks (for example, checking USE flags to see if they match), but I doubt it increases dependency resolution time much.logrusx wrote:Yes you're wrong. Binary packages are additional vertices in the graph. They come with additional edges. It's not so simple as you imagine it. I've peaked in the source code and it's one giant file that handles this stuff. I didn't read it carefully but what I saw as method names et.c. is what I expected to find.ktoupt wrote:Further, I am not convinced the number of binhosts would affect dependency resolution at all. I think the dependency resolution is the same, and then, at the end, Portage would just check if the packages it needs are in a binhost to merge or if it should compile from source. I haven't read the Portage source code so I can't say for sure.
Similarly, I am not convinced having more repos would non-linearly increase resolution time. Although, yes, I do agree having more repos would be slower.
Again, I haven't read the Portage source code, and I also won't claim I know anything about graph theory or dependency resolution, so I could be wrong.
Here's my old thread about the dependency resolution time: viewtopic-t-1171147.html
Also, I see you mention having multiple binhosts configured and dropping one of them for faster resolution times (although you don't provide any logs). Assuming you are correct, I do just have a single binhost configured at a time anyway.
See https://www.gentoo.org/support/news-ite ... ation.htmllogrusx wrote:EDIT: I actually find something off in your emerge --info. To my knowledge haskell overlay it not a thing anymore, but I might be wrong about that. I think they moved most of in in the main tree a while ago. Yes, I used that too and I think it may even have been on my aforementioned old hardware. I needed pandoc.
I'm sorry, but these tests are not comparable to a system which has not been updated for months!logrusx wrote:For comparison:
Ryzen 7 5800H - should be slower than yours.Code: Select all
~ # ACCEPT_KEYWORDS="~amd64" emerge -pv app-text/pandoc These are the packages that would be merged, in order: Calculating dependencies... done! Dependency resolution took 36.82 s (backtrack: 15/20). ... Total: 237 packages (237 new, 1 uninstall), Size of downloads: 63,029 KiB Conflict: 3 blocks (all satisfied)
EDIT2: I enabled ::haskell and added ACCEPT_KEYWORDS="~amd64" and -g to make portage think harder. Also added pandoc to the query to make it use packages from ::haskell. Indeed there's noticeable delay, but not that much. Here's what came out:
Best Regards,Code: Select all
~ # ACCEPT_KEYWORDS="~amd64" emerge -DuUpvg @world app-text/pandoc Local copy of remote index is up-to-date and will be used. Local copy of remote index is up-to-date and will be used. These are the packages that would be merged, in order: Calculating dependencies... done! Dependency resolution took 195.95 s (backtrack: 11/20). ... Total: 581 packages (291 upgrades, 244 new, 15 in new slots, 31 reinstalls, 2 binaries, 2 uninstalls), Size of downloads: 5,524,707 KiB Conflict: 21 blocks (all satisfied)
Georgi
The system had to go through Haskell updates (Haskell is ABI unstable so Portage needs to figure out which reverse dependencies it needs to rebuild), some updates related to Rust (Rust utilizes subslots, which are well-known to cause slow dependency resolution times), a Python version update (not patch update), and I think a Ruby version update as well.
Also, I will highlight it again: the slow down is caused by Haskell updates, hence why I mentioned temporarily deselecting it (and uninstalling Haskell stuff) because it just takes less time to completely uninstall and install from scratch than letting Portage try to figure out what needs to be rebuilt. Also reiterating this: when you update regularly, its fine; the issue is with months of updates!
Okay. I'm getting tired of just arguing because there is clearly some miscommunication going on. Here is hard evidence.
System A is the mostly updated (except for librewolf) system with an i7-8565U. System B is a very slightly not up to date system with a Celeron N3060. To be more specific about how non up to date System B is:
Before:
Code: Select all
Timestamp of repository gentoo: Mon, 09 Jun 2025 07:24:18 +0000
Head commit of repository gentoo: 504d21abb17e390ae2264dcfe1250019be511bb9Code: Select all
Timestamp of repository gentoo: Fri, 13 Jun 2025 10:52:25 +0000
Head commit of repository gentoo: a347110eb75b607c242d63b82ad12e51dded8728Code: Select all
b# time emerge -puUDN --with-bdeps=y --quiet-build --keep-going @world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 306.18 s (backtrack: 2/20).
[ebuild r U ] dev-lang/go-1.24.4 [1.24.3]
[ebuild U ~] cross-i686-w64-mingw32/mingw64-runtime-13.0.0 [12.0.0]
[ebuild U ~] cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0 [12.0.0]
[ebuild rR ] dev-go/go-md2man-2.0.6
[ebuild rR ] app-backup/restic-0.18.0
[ebuild rR ] app-shells/fzf-0.61.0
[ebuild rR ] net-p2p/syncthing-1.27.10
[ebuild U ] app-dicts/myspell-uk-6.6.1 [6.5.4]
[ebuild rR ] net-dns/dnscrypt-proxy-2.1.8
[ebuild U ] dev-perl/Capture-Tiny-0.500.0-r1 [0.500.0]
[ebuild U ] dev-perl/Regexp-Common-2024080801.0.0-r1 [2024080801.0.0]
[ebuild U ] dev-perl/YAML-Tiny-1.760.0-r1 [1.760.0]
[ebuild U ] dev-perl/Business-ISSN-1.8.0-r1 [1.8.0]
[ebuild U ] dev-perl/Sub-Name-0.280.0-r1 [0.280.0]
[ebuild U ] dev-perl/Error-0.170.300-r1 [0.170.300]
[ebuild U ] dev-perl/Specio-0.500.0-r2 [0.500.0-r1]
[ebuild U ] dev-perl/libintl-perl-1.350.0-r1 [1.350.0]
[ebuild U ] dev-perl/DateTime-1.660.0-r1 [1.660.0]
[ebuild rR ] dev-vcs/git-lfs-3.6.1
[ebuild U ] dev-perl/libwww-perl-6.780.0-r1 [6.780.0]
[ebuild U ] sys-apps/systemd-utils-255.18 [255.15-r1]
[ebuild U ] dev-perl/XML-LibXSLT-2.3.0-r1 [2.3.0]
[ebuild U ] dev-util/spirv-llvm-translator-19.1.6 [19.1.5-r2]
[ebuild U ~] dev-python/tifffile-2025.6.11 [2025.6.1]
[ebuild U ] net-misc/yt-dlp-2025.06.09 [2025.05.22]
[ebuild U ] dev-python/requests-2.32.4 [2.32.3] PYTHON_TARGETS="(-python3_14)"
[ebuild rR ] app-containers/podman-5.3.2
[ebuild rR ~] mail-client/aerc-0.20.1
[ebuild U ] dev-util/mesa_clc-25.0.7 [25.0.5] LLVM_SLOT="(-20)"
[ebuild U ~] dev-util/hip-6.3.3-r1 [6.3.3]
[ebuild U ] media-libs/mesa-25.0.7 [25.0.5] LLVM_SLOT="(-20)"
[ebuild NS ~] sys-kernel/gentoo-kernel-6.15.2 [6.14.10]
[ebuild r U ~] virtual/dist-kernel-6.15.2 [6.14.10]
[ebuild rR ~] media-video/v4l2loopback-0.15.0
[ebuild U ~] media-gfx/ueberzugpp-2.9.7 [2.9.6]
[ebuild U ~] www-client/librewolf-139.0.4_p1 [139.0.1_p1]
The following packages are causing rebuilds:
(dev-lang/go-1.24.4:0/1.24.4::gentoo, ebuild scheduled for merge) causes rebuilds for:
(app-shells/fzf-0.61.0:0/0::gentoo, ebuild scheduled for merge)
(app-containers/podman-5.3.2:0/0::gentoo, ebuild scheduled for merge)
(net-p2p/syncthing-1.27.10:0/0::gentoo, ebuild scheduled for merge)
(dev-vcs/git-lfs-3.6.1:0/0::gentoo, ebuild scheduled for merge)
(app-backup/restic-0.18.0:0/0::gentoo, ebuild scheduled for merge)
(dev-go/go-md2man-2.0.6:0/0::gentoo, ebuild scheduled for merge)
(mail-client/aerc-0.20.1:0/0::gentoo, ebuild scheduled for merge)
(net-dns/dnscrypt-proxy-2.1.8:0/0::gentoo, ebuild scheduled for merge)
(virtual/dist-kernel-6.15.2:0/6.15.2::gentoo, ebuild scheduled for merge) causes rebuilds for:
(media-video/v4l2loopback-0.15.0:0/0::gentoo, ebuild scheduled for merge)
!!! The following installed packages are masked:
- media-gfx/transfig-3.2.5e-r2::gentoo (masked by: package.mask)
/var/db/repos/gentoo/profiles/package.mask:
# Alexey Sokolov <alexey+gentoo@asokolov.org> (2025-06-09)
# Deprecated and no longer maintained upstream. Open security
# issues. Use media-gfx/fig2dev instead.
# Removal on 2025-07-12. Bug #917279.
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.
real 5m17.836s
user 5m7.557s
sys 0m5.856sCode: Select all
b# time emerge -puUDNg --with-bdeps=y --quiet-build --keep-going @world
Packages
6,809,429 100% 2.35MB/s 0:00:02 (xfr#1, to-chk=0/1)
sent 43 bytes received 6,811,162 bytes 1,946,058.57 bytes/sec
total size is 6,809,429 speedup is 1.00
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 414.32 s (backtrack: 2/20).
[binary U ~] cross-i686-w64-mingw32/mingw64-runtime-13.0.0-1 [12.0.0]
[binary U ~] cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0-1 [12.0.0]
[binary r U ] dev-lang/go-1.24.4-1 [1.24.3]
[snip]
[binary U ~] media-gfx/ueberzugpp-2.9.7-1 [2.9.6]
[ebuild U ~] www-client/librewolf-139.0.4_p1 [139.0.1_p1]
The following packages are causing rebuilds:
[snip]
real 7m18.445s
user 6m49.683s
sys 0m7.359sAnyway, you also mentioned that my idea of skipping dependency resolution would not work. Well, take a look at this:
Code: Select all
a$ ./write-portage-pkg-db >a.txt
a$ sftp b.localdomain
sftp> put a.txt
b$ ./write-portage-pkg-db >b.txt
b$ ./check-updates-from-portage-pkg-db a.txt b.txt 1749445000
b$ wc -l uninstall.txt
27 uninstall.txt
b# emerge -1ag --quiet-build --nodeps $(cat install.txt)
Packages
6,809,429 100% 1.21MB/s 0:00:05 (xfr#1, to-chk=0/1)
sent 43 bytes received 6,811,162 bytes 1,047,877.69 bytes/sec
total size is 6,809,429 speedup is 1.00
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
[binary U ~] cross-i686-w64-mingw32/mingw64-runtime-13.0.0-1 [12.0.0]
[binary U ~] cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0-1 [12.0.0]
[binary R ] sys-apps/hwloc-2.9.3-r2-1
[binary R ] sys-firmware/edk2-202411-1
[binary U ] app-dicts/myspell-uk-6.6.1-1 [6.5.4]
[binary U ] dev-perl/Capture-Tiny-0.500.0-r1-1 [0.500.0]
[binary U ] dev-perl/Regexp-Common-2024080801.0.0-r1-1 [2024080801.0.0]
[binary U ] dev-perl/YAML-Tiny-1.760.0-r1-1 [1.760.0]
[binary U ] dev-perl/Business-ISSN-1.8.0-r1-1 [1.8.0]
[binary U ] dev-perl/Sub-Name-0.280.0-r1-1 [0.280.0]
[binary U ] dev-perl/Error-0.170.300-r1-1 [0.170.300]
[binary U ] dev-perl/Specio-0.500.0-r2-1 [0.500.0-r1]
[binary U ] dev-perl/libintl-perl-1.350.0-r1-1 [1.350.0]
[binary U ] dev-perl/DateTime-1.660.0-r1-1 [1.660.0]
[binary U ] dev-perl/libwww-perl-6.780.0-r1-1 [6.780.0]
[binary U ] dev-perl/XML-LibXSLT-2.3.0-r1-1 [2.3.0]
[binary U ] net-misc/yt-dlp-2025.06.09-1 [2025.05.22]
[binary U ] dev-python/requests-2.32.4-1 [2.32.3] PYTHON_TARGETS="(-python3_14)"
[binary U ] dev-util/spirv-llvm-translator-19.1.6-1 [19.1.5-r2]
[binary U ] dev-util/mesa_clc-25.0.7-1 [25.0.5] LLVM_SLOT="(-20)"
[binary U ] media-libs/mesa-25.0.7-1 [25.0.5] LLVM_SLOT="(-20)"
[binary NS ~] sys-kernel/gentoo-kernel-6.15.2-1 [6.14.10]
[binary U ~] virtual/dist-kernel-6.15.2-1 [6.14.10]
[binary R ~] media-video/v4l2loopback-0.15.0-3
[binary U ~] dev-util/hip-6.3.3-r1-1 [6.3.3]
[binary U ~] media-gfx/ueberzugpp-2.9.7-1 [2.9.6]
[binary U ] sys-apps/systemd-utils-255.18-1 [255.15-r1]
[binary U ] dev-lang/go-1.24.4-1 [1.24.3]
[binary R ] dev-go/go-md2man-2.0.6-2
[binary R ] app-backup/restic-0.18.0-4
[binary R ] app-shells/fzf-0.61.0-2
[binary R ] net-p2p/syncthing-1.27.10-11
[binary R ] net-dns/dnscrypt-proxy-2.1.8-4
[binary R ] dev-vcs/git-lfs-3.6.1-4
[binary U ~] dev-python/tifffile-2025.6.11-1 [2025.6.1]
[binary R ] app-containers/podman-5.3.2-3
[binary R ~] mail-client/aerc-0.20.1-11
Would you like to merge these packages? [Yes/No]
>>> Running pre-merge checks for media-libs/mesa-25.0.7
>>> Running pre-merge checks for sys-kernel/gentoo-kernel-6.15.2
* Assuming you do not have a separate /boot partition.
* Assuming you do not have a separate /efi partition.
* Your /boot/efi partition was detected as being mounted.
* Files will be installed there for gentoo-kernel to function correctly.
* Assuming you do not have a separate /boot/EFI partition.
>>> Emerging binary (1 of 37) cross-i686-w64-mingw32/mingw64-runtime-13.0.0::cross-i686-w64-mingw32
>>> Installing (1 of 37) cross-i686-w64-mingw32/mingw64-runtime-13.0.0::cross-i686-w64-mingw32
>>> Completed (1 of 37) cross-i686-w64-mingw32/mingw64-runtime-13.0.0::cross-i686-w64-mingw32
>>> Emerging binary (2 of 37) cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0::cross-x86_64-w64-mingw32
>>> Installing (2 of 37) cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0::cross-x86_64-w64-mingw32
>>> Completed (2 of 37) cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0::cross-x86_64-w64-mingw32
>>> Emerging binary (3 of 37) sys-apps/hwloc-2.9.3-r2::gentoo
>>> Installing (3 of 37) sys-apps/hwloc-2.9.3-r2::gentoo
>>> Completed (3 of 37) sys-apps/hwloc-2.9.3-r2::gentoo
[snip]
>>> Emerging binary (36 of 37) app-containers/podman-5.3.2::gentoo
>>> Installing (36 of 37) app-containers/podman-5.3.2::gentoo
>>> Completed (36 of 37) app-containers/podman-5.3.2::gentoo
>>> Emerging binary (37 of 37) mail-client/aerc-0.20.1::gentoo
>>> Installing (37 of 37) mail-client/aerc-0.20.1::gentoo
>>> Completed (37 of 37) mail-client/aerc-0.20.1::gentoo
>>> Jobs: 37 of 37 complete Load avg: 1.14, 1.18, 0.97
[snip]
b# emerge -ac
* Always study the list of packages to be cleaned for any obvious
* mistakes. Packages that are part of the world set will always
* be kept. They can be manually added to this set with
* `emerge --noreplace <atom>`. Packages that are listed in
* package.provided (see portage(5)) will be removed by
* depclean, even if they are part of the world set.
*
* As a safety measure, depclean will not remove any packages
* unless *all* required dependencies have been resolved. As a
* consequence of this, it often becomes necessary to run
* `emerge --update --newuse --deep @world` prior to depclean.
Calculating dependencies... done!
>>> Calculating removal order...
>>> These are the packages that would be unmerged:
sys-kernel/gentoo-kernel
selected: 6.14.10
protected: none
omitted: 6.15.2
virtual/perl-Module-Load
selected: 0.360.0-r4
protected: none
omitted: none
All selected packages: =sys-kernel/gentoo-kernel-6.14.10 =virtual/perl-Module-Load-0.360.0-r4
>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.
Would you like to unmerge these packages? [Yes/No]
>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging (1 of 2) sys-kernel/gentoo-kernel-6.14.10...
>>> Unmerging (2 of 2) virtual/perl-Module-Load-0.360.0-r4...
Packages installed: 2053
Packages in world: 211
Packages in system: 52
Required packages: 2053
Number removed: 2
* GNU info directory index is up-to-date.
b$ ./write-portage-pkg-db >b.txt
b$ ./check-updates-from-portage-pkg-db a.txt b.txt 1749445000
b$ wc -l install.txt uninstall.txt
0 install.txt
0 uninstall.txt
0 totalLet me explain:
write-portage-pkg-db is the serializor which looks through /var/db/pkg and writes information about the packages a in textual format so I can sftp it. It just writes the full package name (including version), followed by a space, followed by the content of its BUILD_TIME (I decided BUILD_TIME is more relevant than mtime).
check-updates-from-portage-pkg-db accepts the package list information from both system A and B to figure out what needs to be updated in B and in what order. It follows the logic at the end of my initial post, plus it orders by BUILD_TIME. It writes out the packages in install.txt and uninstall.txt.
So what is this 1749445000 I provide? Well, its just a UNIX timestamp of what is the earliest BUILD_TIME that will be considered. In this case, its sometime in June 9, which you will recall is the last time I updated System B. This timestamp filter just removes a bunch of irrelevant packages which were installed a long time ago (like acct-group and acct-user packages).
Note how it found all the packages I needed to update (except for librewolf since it has not been updated in System A either). Also note how it correctly installed gentoo-kernel before installing the v4l2loopback kernel module. Ditto for dev-lang/go and its reverse dependencies (although, if they were installed out of order it would not matter in the case of dev-lang/go).
One weird thing you might notice is that it also wanted to rebuild sys-apps/hwloc-2.9.3-r2-1 and sys-firmware/edk2-202411-1, which is not needed. This is because I updated those two packages on System B earlier than I did on System A, and, when I updated them on System A, I re-built them from scratch rather than pull them from System B. Thus the BUILD_TIME was higher on System A than B, which my simple tool picks up as a reason to rebuild.
Anyway, as I previously mentioned, the final test for these tools will be on the 13th gen Intel system in a couple of weeks hopefully. I will roll it back, demonstrate the 30+ minutes resolution time, then try my tool and see if it works.
I am a bit hesitant about releasing these two tools. I am afraid someone will misuse them then blame me for their broken system. I also wrote them rather hurriedly and messily using a personal "library" which was also written hurriedly and messily. I would need to at least untangle them from the library (copy parts of it).
I will do so if there is demand for them, but, remember, only use them for updating an identically configured system.
Oh, and if you are curious about the long resolution times here even though the system is not long out of date. Remember, that it is on a Celeron N3060. Here is a comparison with the i7-8565U:
Code: Select all
b# time emerge -puUDNg --with-bdeps=y --quiet-build --keep-going @world
Packages
6,809,429 100% 1.61MB/s 0:00:04 (xfr#1, to-chk=0/1)
sent 43 bytes received 6,811,162 bytes 1,513,601.11 bytes/sec
total size is 6,809,429 speedup is 1.00
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 219.77 s (backtrack: 0/20).
[ebuild U ~] www-client/librewolf-139.0.4_p1 [139.0.1_p1]
real 4m4.538s
user 3m51.057s
sys 0m5.500s
a# time emerge -puUDN --with-bdeps=y --quiet-build --keep-going @world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 47.99 s (backtrack: 1/20).
[ebuild U ~] www-client/librewolf-139.0.4_p1 [139.0.1_p1]
!!! The following installed packages are masked:
- media-gfx/transfig-3.2.5e-r2::gentoo (masked by: package.mask)
/var/db/repos/gentoo/profiles/package.mask:
# Alexey Sokolov <alexey+gentoo@asokolov.org> (2025-06-09)
# Deprecated and no longer maintained upstream. Open security
# issues. Use media-gfx/fig2dev instead.
# Removal on 2025-07-12. Bug #917279.
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.
real 0m52.810s
user 0m51.547s
sys 0m0.991sHuh? Isn't that what I just did?logrusx wrote:I have some knowledge and I'm confident in it. If you can prove me wrong, do it, I'll be happy to learn something new.
You said it is not possible to update my identically configured Gentoo system without going through Portage's dependency resolution, even if I have another system which was updated normally, going through Portage's dependency resolution.
Yet, look at this from the previous post:
Code: Select all
b# emerge -1ag --quiet-build --nodeps $(cat install.txt)
Packages
6,809,429 100% 1.21MB/s 0:00:05 (xfr#1, to-chk=0/1)
sent 43 bytes received 6,811,162 bytes 1,047,877.69 bytes/sec
total size is 6,809,429 speedup is 1.00
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
[binary U ~] cross-i686-w64-mingw32/mingw64-runtime-13.0.0-1 [12.0.0]
[binary U ~] cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0-1 [12.0.0]
[binary R ] sys-apps/hwloc-2.9.3-r2-1
[binary R ] sys-firmware/edk2-202411-1
[binary U ] app-dicts/myspell-uk-6.6.1-1 [6.5.4]
[binary U ] dev-perl/Capture-Tiny-0.500.0-r1-1 [0.500.0]
[binary U ] dev-perl/Regexp-Common-2024080801.0.0-r1-1 [2024080801.0.0]
[binary U ] dev-perl/YAML-Tiny-1.760.0-r1-1 [1.760.0]
[binary U ] dev-perl/Business-ISSN-1.8.0-r1-1 [1.8.0]
[binary U ] dev-perl/Sub-Name-0.280.0-r1-1 [0.280.0]
[binary U ] dev-perl/Error-0.170.300-r1-1 [0.170.300]
[binary U ] dev-perl/Specio-0.500.0-r2-1 [0.500.0-r1]
[binary U ] dev-perl/libintl-perl-1.350.0-r1-1 [1.350.0]
[binary U ] dev-perl/DateTime-1.660.0-r1-1 [1.660.0]
[binary U ] dev-perl/libwww-perl-6.780.0-r1-1 [6.780.0]
[binary U ] dev-perl/XML-LibXSLT-2.3.0-r1-1 [2.3.0]
[binary U ] net-misc/yt-dlp-2025.06.09-1 [2025.05.22]
[binary U ] dev-python/requests-2.32.4-1 [2.32.3] PYTHON_TARGETS="(-python3_14)"
[binary U ] dev-util/spirv-llvm-translator-19.1.6-1 [19.1.5-r2]
[binary U ] dev-util/mesa_clc-25.0.7-1 [25.0.5] LLVM_SLOT="(-20)"
[binary U ] media-libs/mesa-25.0.7-1 [25.0.5] LLVM_SLOT="(-20)"
[binary NS ~] sys-kernel/gentoo-kernel-6.15.2-1 [6.14.10]
[binary U ~] virtual/dist-kernel-6.15.2-1 [6.14.10]
[binary R ~] media-video/v4l2loopback-0.15.0-3
[binary U ~] dev-util/hip-6.3.3-r1-1 [6.3.3]
[binary U ~] media-gfx/ueberzugpp-2.9.7-1 [2.9.6]
[binary U ] sys-apps/systemd-utils-255.18-1 [255.15-r1]
[binary U ] dev-lang/go-1.24.4-1 [1.24.3]
[binary R ] dev-go/go-md2man-2.0.6-2
[binary R ] app-backup/restic-0.18.0-4
[binary R ] app-shells/fzf-0.61.0-2
[binary R ] net-p2p/syncthing-1.27.10-11
[binary R ] net-dns/dnscrypt-proxy-2.1.8-4
[binary R ] dev-vcs/git-lfs-3.6.1-4
[binary U ~] dev-python/tifffile-2025.6.11-1 [2025.6.1]
[binary R ] app-containers/podman-5.3.2-3
[binary R ~] mail-client/aerc-0.20.1-11
Would you like to merge these packages? [Yes/No]
>>> Running pre-merge checks for media-libs/mesa-25.0.7
>>> Running pre-merge checks for sys-kernel/gentoo-kernel-6.15.2
* Assuming you do not have a separate /boot partition.
* Assuming you do not have a separate /efi partition.
* Your /boot/efi partition was detected as being mounted.
* Files will be installed there for gentoo-kernel to function correctly.
* Assuming you do not have a separate /boot/EFI partition.
>>> Emerging binary (1 of 37) cross-i686-w64-mingw32/mingw64-runtime-13.0.0::cross-i686-w64-mingw32
>>> Installing (1 of 37) cross-i686-w64-mingw32/mingw64-runtime-13.0.0::cross-i686-w64-mingw32
>>> Completed (1 of 37) cross-i686-w64-mingw32/mingw64-runtime-13.0.0::cross-i686-w64-mingw32
>>> Emerging binary (2 of 37) cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0::cross-x86_64-w64-mingw32
>>> Installing (2 of 37) cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0::cross-x86_64-w64-mingw32
>>> Completed (2 of 37) cross-x86_64-w64-mingw32/mingw64-runtime-13.0.0::cross-x86_64-w64-mingw32
>>> Emerging binary (3 of 37) sys-apps/hwloc-2.9.3-r2::gentoo
>>> Installing (3 of 37) sys-apps/hwloc-2.9.3-r2::gentoo
>>> Completed (3 of 37) sys-apps/hwloc-2.9.3-r2::gentoo
[snip]
>>> Emerging binary (36 of 37) app-containers/podman-5.3.2::gentoo
>>> Installing (36 of 37) app-containers/podman-5.3.2::gentoo
>>> Completed (36 of 37) app-containers/podman-5.3.2::gentoo
>>> Emerging binary (37 of 37) mail-client/aerc-0.20.1::gentoo
>>> Installing (37 of 37) mail-client/aerc-0.20.1::gentoo
>>> Completed (37 of 37) mail-client/aerc-0.20.1::gentoo
>>> Jobs: 37 of 37 complete Load avg: 1.14, 1.18, 0.97That install.txt file was created by tools I wrote based on my initial post and sorting by BUILD_TIME.
Please actually read my previous post if you want to know the details or at least let me know what was unclear...
- NeddySeagoon
- Administrator
- Posts: 56085
- Joined: Sat Jul 05, 2003 9:37 am
- Location: 56N 3W
ktoupt,
What happens if your build host uses emerge --jobs= or you get packages in the merge wait state.
Does that make a mess of the ordering, so packages are not installed on the build host in BTIME order?
A single demonstration is not a proof that you have a robust solution.
What happens if your build host uses emerge --jobs= or you get packages in the merge wait state.
Does that make a mess of the ordering, so packages are not installed on the build host in BTIME order?
A single demonstration is not a proof that you have a robust solution.
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
I assume it would mess up the ordering because Portage has no idea what the dependency graph is when I pass '--nodeps'. So that is another limitation of this method to keep in mind. Thanks for the constructive feedback.NeddySeagoon wrote:ktoupt,
What happens if your build host uses emerge --jobs= or you get packages in the merge wait state.
Does that make a mess of the ordering, so packages are not installed on the build host in BTIME order?
A single demonstration is not a proof that you have a robust solution.
Okay, I just rolled-back the i9-13900K system to try and demonstrate the 30+ minute dependency resolution, but it seems faster at throwing errors now. Maybe the day I was trying to update, there was something in the Gentoo repo giving Portage a hard time.
Here are the git commits of the Gentoo repo:
Before:
After:
Previously, when I experienced the 30+ minute dependency resolution times, the git commit of the Gentoo repo was 4e0b39dfee4b64451165618ffebb9976f10a2d32. I must be stupid because I cannot find documentation on /etc/portage/repos.conf, so I do not know how to set a specific commit (and I already wasted too much time on this so moving on).
Anyway, currently, I am getting ~5 minute dependency resolution time for that update from 01 March to 27 June, which is much better than before but still annoying. There is still, however, the issue of Portage complaining about stuff which I know are not real problems (because my identically configured and up-to-date system is fine). So it might just be stopping early...
Of course, I do know how to fix those issues by, for example, deselecting packages temporarily (or whatever else) just to get Portage to continue, but I got frustrated and did not want to waste more time just trying to prove to you that this is a problem.
If you do not think this is an actual problem, then what are you doing reading this long thread. For the 2 remaining people who are in the same boat as me and have identically configured systems, where one is not updated as often as the other, and have experienced first-hand the annoying, time-consuming task of trying to get Portage to just update already(!!), read on.
I am now fairly confident that the approach detailed at the end of my initial post along with sorting by BUILD_TIME works at completely skipping dependency resolution times by passing --nodeps (under some situations, like not using --jobs).
For a more robust, but slower, solution, then you just implement what I detailed at the end of my initial post (no need to manually sort by BUILD_TIME) but do not pass --nodeps. Portage will do dependency resolution, but it is much faster than just giving it @world, and it also seems to not run into issues (i.e. those fake problems I previously mentioned) as often. With this slightly slower solution, Portage will correctly order the dependencies, and you can use --jobs if you wish.
I am still a bit hesitant to give the source code for the tools I wrote and risk someone blindly using them without understand the circumstance they work under (identically configured systems). There is also the issue of decoupling them from an internal library.
To solve someone blindly using the tools, I have decided not to give instructions (but if you do read the thread and prove to yourself that you understand the risks and when this tools are supposed to work, then you will find that I essentially gave instructions in a previous post).
To solve the issue about the internal library, I just quickly re-wrote the tools as shell scripts. They are much slower than the original C programs, especially because of the constant greping (hash maps in shell when?), but they are still faster than Portage dependency resolution. They should give you an idea of what I was trying to say at the end of my initial post, so you may want to re-write it in your favorite language if you want the speed boost and have time to spare.
write-portage-pkg-db:
check-updates-from-portage-pkg-db:
Here are the git commits of the Gentoo repo:
Before:
Code: Select all
Timestamp of repository gentoo: Sat, 01 Mar 2025 01:48:43 +0000
Head commit of repository gentoo: a537a2828fba698a32e2735234eb47d370d615deCode: Select all
Timestamp of repository gentoo: Fri, 27 Jun 2025 06:06:54 +0000
Head commit of repository gentoo: 0c44872ca142ededc34c375d3d2bab5235cd242fAnyway, currently, I am getting ~5 minute dependency resolution time for that update from 01 March to 27 June, which is much better than before but still annoying. There is still, however, the issue of Portage complaining about stuff which I know are not real problems (because my identically configured and up-to-date system is fine). So it might just be stopping early...
Of course, I do know how to fix those issues by, for example, deselecting packages temporarily (or whatever else) just to get Portage to continue, but I got frustrated and did not want to waste more time just trying to prove to you that this is a problem.
If you do not think this is an actual problem, then what are you doing reading this long thread. For the 2 remaining people who are in the same boat as me and have identically configured systems, where one is not updated as often as the other, and have experienced first-hand the annoying, time-consuming task of trying to get Portage to just update already(!!), read on.
I am now fairly confident that the approach detailed at the end of my initial post along with sorting by BUILD_TIME works at completely skipping dependency resolution times by passing --nodeps (under some situations, like not using --jobs).
For a more robust, but slower, solution, then you just implement what I detailed at the end of my initial post (no need to manually sort by BUILD_TIME) but do not pass --nodeps. Portage will do dependency resolution, but it is much faster than just giving it @world, and it also seems to not run into issues (i.e. those fake problems I previously mentioned) as often. With this slightly slower solution, Portage will correctly order the dependencies, and you can use --jobs if you wish.
I am still a bit hesitant to give the source code for the tools I wrote and risk someone blindly using them without understand the circumstance they work under (identically configured systems). There is also the issue of decoupling them from an internal library.
To solve someone blindly using the tools, I have decided not to give instructions (but if you do read the thread and prove to yourself that you understand the risks and when this tools are supposed to work, then you will find that I essentially gave instructions in a previous post).
To solve the issue about the internal library, I just quickly re-wrote the tools as shell scripts. They are much slower than the original C programs, especially because of the constant greping (hash maps in shell when?), but they are still faster than Portage dependency resolution. They should give you an idea of what I was trying to say at the end of my initial post, so you may want to re-write it in your favorite language if you want the speed boost and have time to spare.
write-portage-pkg-db:
Code: Select all
#!/bin/sh
set -e
list="$(find /var/db/pkg -name BUILD_TIME)"
for build_time_path in $list; do
build_time="$(cat "$build_time_path")"
pkg="${build_time_path#/var/db/pkg/}"
pkg="${pkg%/BUILD_TIME}"
echo "$pkg $build_time"
doneCode: Select all
#!/bin/sh
a_pkg_db_file="$1"
b_pkg_db_file="$2"
min_build_time="${3:-0}"
install_unsorted="install_unsorted.txt"
test -f "$install_unsorted" && rm "$install_unsorted"
uninstall="uninstall.txt"
test -f "$uninstall" && rm "$uninstall"
install_sorted="install_sorted.txt"
test -f "$install_sorted" && rm "$install_sorted"
install="install.txt"
test -f "$install" && rm "$install"
while read -r a_pkg; do
pkg_name="$(printf '%s' "$a_pkg" | cut -d' ' -f 1)"
a_time="$(printf '%s' "$a_pkg" | cut -d' ' -f 2)"
b_pkg="$(grep -F "$pkg_name " "$b_pkg_db_file")"
if test "$a_time" -gt "$min_build_time"; then
if test -n "$b_pkg"; then
b_time="$(printf '%s' "$b_pkg" | cut -d' ' -f 2)"
if test "$a_time" -gt "$b_time"; then
printf '%d %s\n' "$a_time" "$pkg_name" >>"$install_unsorted"
fi
else
printf '%d %s\n' "$a_time" "$pkg_name" >>"$install_unsorted"
fi
fi
done <"$a_pkg_db_file"
while read -r b_pkg; do
pkg_name="$(printf '%s' "$b_pkg" | cut -d' ' -f 1)"
a_pkg="$(grep -F "$pkg_name " "$a_pkg_db_file")"
if test -z "$a_pkg"; then
printf '%s\n' "$pkg_name" >>"$uninstall"
fi
done <"$b_pkg_db_file"
sort -n "$install_unsorted" -o "$install_sorted"
while read -r line; do
pkg_name="$(printf '%s' "$line" | cut -d' ' -f 2)"
printf '=%s\n' "$pkg_name" >>"$install"
done <"$install_sorted"There's no such option in repos.conf.ktoupt wrote:I must be stupid because I cannot find documentation on /etc/portage/repos.conf, so I do not know how to set a specific commit
Your portage tree is a git repo. You manually go there and an manually it through git commands. On the next sync it'll be reset back to HEAD so you don't need to worry about that.
Best Regards,
Georgi