View previous topic :: View next topic |
Author |
Message |
curmudgeon Veteran
Joined: 08 Aug 2003 Posts: 1741
|
Posted: Tue Apr 24, 2018 2:56 pm Post subject: Why is portage so flaky? |
|
|
I swear this actually happened and that I am not losing my mind. :)
After spending several days rebuilding the system (culminating with an "emerge -e world"), I wanted to check the consistency of the current state. I typed the following commands one after the other with nothing else done in between. The second command was an attempt to determine which package was demanding the older version of readline.
Code: |
# emerge -ep world
[...]
[ebuild NS ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[...]
Total: 871 packages (870 reinstalls, 1 in new slot), Size of downloads: 2.024 KiB
# emerge -ept world
[...]
Total: 870 packages (870 reinstalls), Size of downloads: 0 KiB
# emerge -ep world
[...]
Total: 870 packages (870 reinstalls), Size of downloads: 0 KiB
|
So I would have to guess that there is some cache somewhere that got updated (and I never will find out what package thought it needed readline-5.2_p14).
Just maddening to get different output from running the same command almost back to back. |
|
Back to top |
|
|
fturco Veteran
Joined: 08 Dec 2010 Posts: 1181 Location: Italy
|
Posted: Wed Apr 25, 2018 12:17 pm Post subject: |
|
|
I don't know an answer to your problem, but I noticed you're using "world" instead of "@world". Nowadays sets such as "@world" or "@system" should all begin with an "@" sign. |
|
Back to top |
|
|
Tony0945 Watchman
Joined: 25 Jul 2006 Posts: 5127 Location: Illinois, USA
|
Posted: Wed Apr 25, 2018 1:05 pm Post subject: |
|
|
fturco wrote: | I don't know an answer to your problem, but I noticed you're using "world" instead of "@world". Nowadays sets such as "@world" or "@system" should all begin with an "@" sign. | Why? Seriously. |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Apr 25, 2018 1:33 pm Post subject: |
|
|
For consistency and to establish the appropriate muscle memory if nothing else. "world" and "system" existed in Portage as informal sets before sets were formalized in the implementation. To preserve backwards compatibility with legacy muscle memory (and legacy documentation), the "@" can be omitted on those sets but on all other sets the "@" is required. It's like preferring "/etc/portage/package.keywords" over "/etc/portage/keywords". Old timers know that the latter is supported but should also realize it's deprecated.
Just FYI, I use the @selected (only the contents of the /var/lib/portage/world file) set a lot.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9677 Location: almost Mile High in the USA
|
Posted: Wed Apr 25, 2018 2:02 pm Post subject: |
|
|
Aside from the @ issue, I have seen that --pretend frequently gives different results than without --pretend, and haven't really found why. I chalked it to some random timing behavior from a hardware point of view but the excuse made for myself makes no sense.
From what I *feel* is the issue is that sometimes packages get chosen to be installed in a random order if portage feels like they all have the same weight. Due to the ordering, a different set of USE flags are needed and sometimes more than one solution to solve the NP problem is found, and --pretend tends to choose that other solution.
As I rarely use --emptytree I don't know about the characteristics of such, but I do know it affects --fetchonly. I frequently use --fetchonly with --pretend and have posted this in the past but elicited no responses. However the thing is that usually I do get a usable list for the --pretend but frequently end up with results that were predicted incorrectly.
Sorry, just putting my 2 cents in here, just commiserating on similar issues... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
krinn Watchman
Joined: 02 May 2003 Posts: 7470
|
Posted: Wed Apr 25, 2018 2:13 pm Post subject: |
|
|
John R. Graham wrote: | Just FYI, I use the @selected (only the contents of the /var/lib/portage/world file) set a lot. |
It's not "purely" true
@selected include @selected-packages and @selected-sets, which include @system, so there's no real diffence with @world or world
See: Code: | >grep gcc /var/lib/portage/world
>emerge -pv -e @selected-packages --nodeps | grep gcc
* Last emerge --sync was...
>emerge -pv -e @selected --nodeps | grep gcc
* Last emerge...
[binary R *] sys-devel/gcc-7.3.0-r1...
>wc -l /var/lib/portage/world
120 /var/lib/portage/world
>emerge -pv -e @selected-packages --nodeps
...
Total: 120 packages
>emerge -pv -e @selected --nodeps
...
Total: 162 packages
|
So technically, only @selected-packages do only include contents from /var/lib/portage/world
For op question: i have never seen that myself, nor i have any explains except bug. |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Apr 25, 2018 2:28 pm Post subject: |
|
|
Jeez. I'm pretty sure that used to be true. Guess I need to go head down again into the Portage source, which I'm starting to do on this issue anyway. Thanks for keeping me honest.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
krinn Watchman
Joined: 02 May 2003 Posts: 7470
|
Posted: Wed Apr 25, 2018 2:51 pm Post subject: |
|
|
couldn't help with this one, never really use myself @ (any), i keep using world and system.
ps: you remember --inject ? (if you remember --inject, congrats you're old!) |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Apr 25, 2018 3:29 pm Post subject: |
|
|
krinn,
Notice how polite Portage is. It appears to let you keep the definition of @selected at the time of your install: Code: | ~ # emerge -pv -e @selected --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 131 packages (131 reinstalls), Size of downloads: 0 KiB
~ # emerge -pv -e @selected-packages --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 131 packages (131 reinstalls), Size of downloads: 0 KiB
~ # wc -l /var/lib/portage/world
131 /var/lib/portage/world | Or at least I so surmise. I think this means that my Gentoo install is older than yours. What do you think it means?
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Tony0945 Watchman
Joined: 25 Jul 2006 Posts: 5127 Location: Illinois, USA
|
Posted: Wed Apr 25, 2018 3:39 pm Post subject: |
|
|
eccerr0r wrote: | Aside from the @ issue, I have seen that --pretend frequently gives different results than without --pretend, and haven't really found why. I chalked it to some random timing behavior from a hardware point of view but the excuse made for myself makes no sense. |
I haven't seen that, but I did notice that changing --autounmask-write=y to --autounmask-write=n gives a much cleaner view of blockers and what is really blocked. My guess is that portage tries to solve a blocker by unmasking something then gives a totally erroneous blocking message based on what it unblocked which was something I definitely wanted blocked. |
|
Back to top |
|
|
krinn Watchman
Joined: 02 May 2003 Posts: 7470
|
Posted: Wed Apr 25, 2018 3:45 pm Post subject: |
|
|
Hard to say, this gentoo (the one i use to provide the examples) is my desktop computer, i have install gentoo 3 times on this "one" ("one" because i have kept migrating it over new m/b, cpu and hdd).
in 2002
in 2009~ (hdd died, and no backup as since 2002 i kept ignoring NeddySeagoon advise for my /)
in 2017 (migration to amd64)
But it must be that.
My server also exhibit your behaviour because its world_sets is empty, while my desktop have @system in world_sets
But look at my server log directory date (which is not totally true, because in 2002 i had only a desktop, and the server itself was built later, but using my desktop gentoo as stage4)
Code: | drwxr-xr-x 14 root root 4096 2002-01-01 01:06:20.000000000 +0100 log
|
|
|
Back to top |
|
|
freke l33t
Joined: 23 Jan 2003 Posts: 974 Location: Somewhere in Denmark
|
Posted: Wed Apr 25, 2018 6:48 pm Post subject: |
|
|
Must be something other than age that defines @selected/selected-packages, my young gentoo-server exhibits same behaviour as Graham's: Code: | lamp ~ # emerge -pv -e @selected --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 34 packages (34 reinstalls), Size of downloads: 0 KiB
lamp ~ # emerge -pv -e @selected-packages --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 34 packages (34 reinstalls), Size of downloads: 0 KiB
lamp ~ # wc -l /var/lib/portage/world
34 /var/lib/portage/world |
Initial sync during instalation Code: | lamp ~ # head -6 /var/log/emerge.log
1497722118: Started emerge on: Jun 17, 2017 17:55:17
1497722118: *** emerge --sync
1497722118: === sync
1497722118: >>> Syncing repository 'gentoo' into '/opt/portage'...
1497722118: >>> Starting rsync with rsync://[2a01:488:67:1000:b01c:3277:0:1]/gentoo-portage
1497722151: === Sync completed for gentoo |
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Apr 25, 2018 6:58 pm Post subject: |
|
|
Well, we always knew krinn was an anomaly.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6747
|
Posted: Wed Apr 25, 2018 7:18 pm Post subject: |
|
|
When sets had been introduced, portage automatically added @system to the world_set file.
Much later there was a time when it was decided that portage should better add this implicitly. Since then, portage no longer did this; in some cases it was even removed (emaint?).
And then not much later @selected was introduced. |
|
Back to top |
|
|
krinn Watchman
Joined: 02 May 2003 Posts: 7470
|
Posted: Wed Apr 25, 2018 7:54 pm Post subject: |
|
|
John R. Graham wrote: | Well, we always knew krinn was an anomaly. |
We say "French" here |
|
Back to top |
|
|
Tony0945 Watchman
Joined: 25 Jul 2006 Posts: 5127 Location: Illinois, USA
|
Posted: Wed Apr 25, 2018 9:22 pm Post subject: |
|
|
krinn wrote: | John R. Graham wrote: | Well, we always knew krinn was an anomaly. |
We say "French" here |
LOL! Mes ami! |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Thu Apr 26, 2018 3:01 am Post subject: Re: Why is portage so flaky? |
|
|
curmudgeon wrote: | So I would have to guess that there is some cache somewhere that got updated (and I never will find out what package thought it needed readline-5.2_p14).
Just maddening to get different output from running the same command almost back to back. |
I guess it could in theory happen if some package resolution was affected by the iteration order of a hash or a race between reading two files, but things like that shouldn't be possible in python since it's single-threaded.
Might be better for everyone's sanity if we just accept portage gets it wrong sometimes and don't look at the code directly to figure out why... |
|
Back to top |
|
|
curmudgeon Veteran
Joined: 08 Aug 2003 Posts: 1741
|
Posted: Sat Apr 28, 2018 1:25 am Post subject: |
|
|
It's baaaaack...
So today, synced, and then did emerge -Dpu world. No problems there. Downloaded all of the sources required for updates (I always do that separately and first, just to remove the possibility of something going wrong in the fetch process). Readline was not among those. Checked for consistency before actually starting the update, and...
Code: |
# emerge -Dpu world
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild U ] dev-libs/vala-common-0.36.13::gentoo [0.36.7::gentoo] 0 KiB
[ebuild NS ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[ebuild U ] dev-libs/libxml2-2.9.8:2::gentoo [2.9.7:2::gentoo] USE="icu ipv6 lzma python readline -debug -examples -static-libs {-test}" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 -python3_4 -python3_5 -python3_6" 0 KiB
[ebuild U ] dev-libs/libinput-1.10.5:0/10::gentoo [1.9.4:0/10::gentoo] USE="-doc {-test}" INPUT_DEVICES="-wacom" 0 KiB
[ebuild U ] x11-libs/xcb-util-cursor-0.1.3-r2::gentoo [0.1.3-r1::gentoo] USE="-doc -static-libs {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild U ] media-libs/libgphoto2-2.5.16:0/6::gentoo [2.5.14:0/6::gentoo] USE="exif jpeg -doc -examples -gd -nls -serial" ABI_X86="(64) -32 (-x32)" CAMERAS="ptp2 -adc65 -agfa_cl20 -aox -ax203 -barbie -canon -casio_qv -clicksmart310 -digigr8 -digita -dimagev -dimera3500 -directory -enigma13 -fuji -gsmart300 -hp215 -iclick -jamcam -jd11 -jl2005a -jl2005c -kodak_dc120 -kodak_dc210 -kodak_dc240 -kodak_dc3200 -kodak_ez200 -konica -konica_qm150 -largan -lg_gsm -mars -mustek -panasonic_coolshot -panasonic_dc1000 -panasonic_dc1580 -panasonic_l859 -pccam300 -pccam600 -pentax -polaroid_pdc320 -polaroid_pdc640 -polaroid_pdc700 -ricoh -ricoh_g3 -samsung -sierra -sipix_blink2 -sipix_web2 -smal -sonix -sony_dscf1 -sony_dscf55 -soundvision -spca50x -sq905 -st2205 -stv0674 -stv0680 -sx330z -topfield -toshiba_pdrm11 -tp6801" 0 KiB
[ebuild U ] x11-libs/gdk-pixbuf-2.36.12:2::gentoo [2.36.11:2::gentoo] USE="X jpeg tiff -debug -introspection -jpeg2k {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild U ] media-libs/harfbuzz-1.7.6:0/0.9.18::gentoo [1.7.2:0/0.9.18::gentoo] USE="cairo fontconfig glib icu truetype -debug -graphite -introspection -static-libs {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild U ] x11-libs/gtk+-3.22.29:3::gentoo [3.22.19:3::gentoo] USE="X cups xinerama (-aqua) -broadway -cloudprint -colord -examples -introspection {-test} -vim-syntax -wayland" ABI_X86="(64) -32 (-x32)" 0 KiB
Total: 9 packages (8 upgrades, 1 in new slot), Size of downloads: 2,024 KiB
|
This time portage revealed the culprit (skipping the tree version of the output):
Code: |
# emerge -Dpu libxml2
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild NS ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[ebuild U ] dev-libs/libxml2-2.9.8:2::gentoo [2.9.7:2::gentoo] USE="icu ipv6 lzma python readline -debug -examples -static-libs {-test}" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 -python3_4 -python3_5 -python3_6" 0 KiB
Total: 2 packages (1 upgrade, 1 in new slot), Size of downloads: 2,024 KiB
|
A quick "<=sys-libs/readline-6" in /etc/portage/package.mask will prevent that from showing up in the future (but I don't really understand why it is happening in the first place).icu ipv6 lzma python readline -debug -examples -static-libs {-test} |
|
Back to top |
|
|
zmedico Developer
Joined: 02 Jan 2004 Posts: 352 Location: California USA
|
Posted: Sat Apr 28, 2018 9:02 pm Post subject: |
|
|
curmudgeon wrote: |
This time portage revealed the culprit (skipping the tree version of the output):
Code: |
# emerge -Dpu libxml2
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild NS ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[ebuild U ] dev-libs/libxml2-2.9.8:2::gentoo [2.9.7:2::gentoo] USE="icu ipv6 lzma python readline -debug -examples -static-libs {-test}" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 -python3_4 -python3_5 -python3_6" 0 KiB
Total: 2 packages (1 upgrade, 1 in new slot), Size of downloads: 2,024 KiB
|
|
The doesn't show the true cause, because there's no reason for dev-libs/libxml2-2.9.8:2 to pull in an older readline.
A debug log created as follows would reveal the answer:
Code: |
emerge -Dpu libxml2 --debug 2>&1 > debug.log
xz -9 debug.log
|
_________________ Zac |
|
Back to top |
|
|
curmudgeon Veteran
Joined: 08 Aug 2003 Posts: 1741
|
Posted: Sat Apr 28, 2018 11:13 pm Post subject: |
|
|
zmedico wrote: | The doesn't show the true cause, because there's no reason for dev-libs/libxml2-2.9.8:2 to pull in an older readline.
A debug log created as follows would reveal the answer:
Code: |
emerge -Dpu libxml2 --debug 2>&1 > debug.log
xz -9 debug.log
|
|
I already did the upgrades, and now (of course), I can't reproduce it. I have removed the line from package.mask, and will report if it ever occurs again. |
|
Back to top |
|
|
|