Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Why is portage so flaky?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1485

PostPosted: Tue Apr 24, 2018 2:56 pm    Post subject: Why is portage so flaky? Reply with quote

I swear this actually happened and that I am not losing my mind. :)

After spending several days rebuilding the system (culminating with an "emerge -e world"), I wanted to check the consistency of the current state. I typed the following commands one after the other with nothing else done in between. The second command was an attempt to determine which package was demanding the older version of readline.

Code:

# emerge -ep world
[...]
[ebuild  NS    ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[...]
Total: 871 packages (870 reinstalls, 1 in new slot), Size of downloads: 2.024 KiB


# emerge -ept world
[...]
Total: 870 packages (870 reinstalls), Size of downloads: 0 KiB


# emerge -ep world
[...]
Total: 870 packages (870 reinstalls), Size of downloads: 0 KiB


So I would have to guess that there is some cache somewhere that got updated (and I never will find out what package thought it needed readline-5.2_p14).

Just maddening to get different output from running the same command almost back to back.
Back to top
View user's profile Send private message
fturco
Guru
Guru


Joined: 08 Dec 2010
Posts: 529
Location: Italy

PostPosted: Wed Apr 25, 2018 12:17 pm    Post subject: Reply with quote

I don't know an answer to your problem, but I noticed you're using "world" instead of "@world". Nowadays sets such as "@world" or "@system" should all begin with an "@" sign.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2457
Location: Illinois, USA

PostPosted: Wed Apr 25, 2018 1:05 pm    Post subject: Reply with quote

fturco wrote:
I don't know an answer to your problem, but I noticed you're using "world" instead of "@world". Nowadays sets such as "@world" or "@system" should all begin with an "@" sign.
Why? Seriously.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9936
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Apr 25, 2018 1:33 pm    Post subject: Reply with quote

For consistency and to establish the appropriate muscle memory if nothing else. "world" and "system" existed in Portage as informal sets before sets were formalized in the implementation. To preserve backwards compatibility with legacy muscle memory (and legacy documentation), the "@" can be omitted on those sets but on all other sets the "@" is required. It's like preferring "/etc/portage/package.keywords" over "/etc/portage/keywords". Old timers know that the latter is supported but should also realize it's deprecated.

Just FYI, I use the @selected (only the contents of the /var/lib/portage/world file) set a lot.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 6658
Location: almost Mile High in the USA

PostPosted: Wed Apr 25, 2018 2:02 pm    Post subject: Reply with quote

Aside from the @ issue, I have seen that --pretend frequently gives different results than without --pretend, and haven't really found why. I chalked it to some random timing behavior from a hardware point of view but the excuse made for myself makes no sense.

From what I *feel* is the issue is that sometimes packages get chosen to be installed in a random order if portage feels like they all have the same weight. Due to the ordering, a different set of USE flags are needed and sometimes more than one solution to solve the NP problem is found, and --pretend tends to choose that other solution.

As I rarely use --emptytree I don't know about the characteristics of such, but I do know it affects --fetchonly. I frequently use --fetchonly with --pretend and have posted this in the past but elicited no responses. However the thing is that usually I do get a usable list for the --pretend but frequently end up with results that were predicted incorrectly.

Sorry, just putting my 2 cents in here, just commiserating on similar issues...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6649

PostPosted: Wed Apr 25, 2018 2:13 pm    Post subject: Reply with quote

John R. Graham wrote:
Just FYI, I use the @selected (only the contents of the /var/lib/portage/world file) set a lot.

It's not "purely" true

@selected include @selected-packages and @selected-sets, which include @system, so there's no real diffence with @world or world
See:
Code:
>grep gcc /var/lib/portage/world
>emerge -pv -e @selected-packages --nodeps | grep gcc
* Last emerge --sync was...
>emerge -pv -e @selected --nodeps | grep gcc
 * Last emerge...
[binary   R   *] sys-devel/gcc-7.3.0-r1...
>wc -l /var/lib/portage/world
120 /var/lib/portage/world
>emerge -pv -e @selected-packages --nodeps
...
Total: 120 packages
>emerge -pv -e @selected --nodeps
...
Total: 162 packages


So technically, only @selected-packages do only include contents from /var/lib/portage/world ;)


For op question: i have never seen that myself, nor i have any explains except bug.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9936
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Apr 25, 2018 2:28 pm    Post subject: Reply with quote

Jeez. I'm pretty sure that used to be true. Guess I need to go head down again into the Portage source, which I'm starting to do on this issue anyway. Thanks for keeping me honest. ;)

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6649

PostPosted: Wed Apr 25, 2018 2:51 pm    Post subject: Reply with quote

couldn't help with this one, never really use myself @ (any), i keep using world and system.

ps: you remember --inject ? :D (if you remember --inject, congrats you're old!)
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9936
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Apr 25, 2018 3:29 pm    Post subject: Reply with quote

krinn,

Notice how polite Portage is. It appears to let you keep the definition of @selected at the time of your install:
Code:
~ # emerge -pv -e @selected --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 131 packages (131 reinstalls), Size of downloads: 0 KiB
~ # emerge -pv -e @selected-packages --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 131 packages (131 reinstalls), Size of downloads: 0 KiB
~ # wc -l /var/lib/portage/world
131 /var/lib/portage/world
Or at least I so surmise. I think this means that my Gentoo install is older than yours. What do you think it means?

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2457
Location: Illinois, USA

PostPosted: Wed Apr 25, 2018 3:39 pm    Post subject: Reply with quote

eccerr0r wrote:
Aside from the @ issue, I have seen that --pretend frequently gives different results than without --pretend, and haven't really found why. I chalked it to some random timing behavior from a hardware point of view but the excuse made for myself makes no sense.

I haven't seen that, but I did notice that changing --autounmask-write=y to --autounmask-write=n gives a much cleaner view of blockers and what is really blocked. My guess is that portage tries to solve a blocker by unmasking something then gives a totally erroneous blocking message based on what it unblocked which was something I definitely wanted blocked.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6649

PostPosted: Wed Apr 25, 2018 3:45 pm    Post subject: Reply with quote

Hard to say, this gentoo (the one i use to provide the examples) is my desktop computer, i have install gentoo 3 times on this "one" ("one" because i have kept migrating it over new m/b, cpu and hdd).
in 2002
in 2009~ (hdd died, and no backup as since 2002 i kept ignoring NeddySeagoon advise for my /)
in 2017 (migration to amd64)

But it must be that.
My server also exhibit your behaviour because its world_sets is empty, while my desktop have @system in world_sets
But look at my server log directory date :) (which is not totally true, because in 2002 i had only a desktop, and the server itself was built later, but using my desktop gentoo as stage4)
Code:
drwxr-xr-x 14 root     root 4096 2002-01-01 01:06:20.000000000 +0100 log
Back to top
View user's profile Send private message
freke
Guru
Guru


Joined: 23 Jan 2003
Posts: 352
Location: Somewhere in Denmark

PostPosted: Wed Apr 25, 2018 6:48 pm    Post subject: Reply with quote

Must be something other than age that defines @selected/selected-packages, my young gentoo-server exhibits same behaviour as Graham's:
Code:
lamp ~ # emerge -pv -e @selected --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 34 packages (34 reinstalls), Size of downloads: 0 KiB
lamp ~ # emerge -pv -e @selected-packages --nodeps 2>/dev/null | grep '^Total: [[:digit:]]* packages'
Total: 34 packages (34 reinstalls), Size of downloads: 0 KiB
lamp ~ # wc -l /var/lib/portage/world
34 /var/lib/portage/world

Initial sync during instalation
Code:
lamp ~ # head -6 /var/log/emerge.log
1497722118: Started emerge on: Jun 17, 2017 17:55:17
1497722118:  *** emerge  --sync
1497722118:  === sync
1497722118: >>> Syncing repository 'gentoo' into '/opt/portage'...
1497722118: >>> Starting rsync with rsync://[2a01:488:67:1000:b01c:3277:0:1]/gentoo-portage
1497722151: === Sync completed for gentoo
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9936
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Apr 25, 2018 6:58 pm    Post subject: Reply with quote

Well, we always knew krinn was an anomaly. ;)

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6070

PostPosted: Wed Apr 25, 2018 7:18 pm    Post subject: Reply with quote

When sets had been introduced, portage automatically added @system to the world_set file.
Much later there was a time when it was decided that portage should better add this implicitly. Since then, portage no longer did this; in some cases it was even removed (emaint?).
And then not much later @selected was introduced.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6649

PostPosted: Wed Apr 25, 2018 7:54 pm    Post subject: Reply with quote

John R. Graham wrote:
Well, we always knew krinn was an anomaly. ;)

We say "French" here :)
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2457
Location: Illinois, USA

PostPosted: Wed Apr 25, 2018 9:22 pm    Post subject: Reply with quote

krinn wrote:
John R. Graham wrote:
Well, we always knew krinn was an anomaly. ;)

We say "French" here :)

LOL! Mes ami!
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5010

PostPosted: Thu Apr 26, 2018 3:01 am    Post subject: Re: Why is portage so flaky? Reply with quote

curmudgeon wrote:
So I would have to guess that there is some cache somewhere that got updated (and I never will find out what package thought it needed readline-5.2_p14).

Just maddening to get different output from running the same command almost back to back.

I guess it could in theory happen if some package resolution was affected by the iteration order of a hash or a race between reading two files, but things like that shouldn't be possible in python since it's single-threaded.

Might be better for everyone's sanity if we just accept portage gets it wrong sometimes and don't look at the code directly to figure out why...
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1485

PostPosted: Sat Apr 28, 2018 1:25 am    Post subject: Reply with quote

It's baaaaack...

So today, synced, and then did emerge -Dpu world. No problems there. Downloaded all of the sources required for updates (I always do that separately and first, just to remove the possibility of something going wrong in the fetch process). Readline was not among those. Checked for consistency before actually starting the update, and...

Code:

# emerge -Dpu world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] dev-libs/vala-common-0.36.13::gentoo [0.36.7::gentoo] 0 KiB
[ebuild  NS    ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[ebuild     U  ] dev-libs/libxml2-2.9.8:2::gentoo [2.9.7:2::gentoo] USE="icu ipv6 lzma python readline -debug -examples -static-libs {-test}" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 -python3_4 -python3_5 -python3_6" 0 KiB
[ebuild     U  ] dev-libs/libinput-1.10.5:0/10::gentoo [1.9.4:0/10::gentoo] USE="-doc {-test}" INPUT_DEVICES="-wacom" 0 KiB
[ebuild     U  ] x11-libs/xcb-util-cursor-0.1.3-r2::gentoo [0.1.3-r1::gentoo] USE="-doc -static-libs {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U  ] media-libs/libgphoto2-2.5.16:0/6::gentoo [2.5.14:0/6::gentoo] USE="exif jpeg -doc -examples -gd -nls -serial" ABI_X86="(64) -32 (-x32)" CAMERAS="ptp2 -adc65 -agfa_cl20 -aox -ax203 -barbie -canon -casio_qv -clicksmart310 -digigr8 -digita -dimagev -dimera3500 -directory -enigma13 -fuji -gsmart300 -hp215 -iclick -jamcam -jd11 -jl2005a -jl2005c -kodak_dc120 -kodak_dc210 -kodak_dc240 -kodak_dc3200 -kodak_ez200 -konica -konica_qm150 -largan -lg_gsm -mars -mustek -panasonic_coolshot -panasonic_dc1000 -panasonic_dc1580 -panasonic_l859 -pccam300 -pccam600 -pentax -polaroid_pdc320 -polaroid_pdc640 -polaroid_pdc700 -ricoh -ricoh_g3 -samsung -sierra -sipix_blink2 -sipix_web2 -smal -sonix -sony_dscf1 -sony_dscf55 -soundvision -spca50x -sq905 -st2205 -stv0674 -stv0680 -sx330z -topfield -toshiba_pdrm11 -tp6801" 0 KiB
[ebuild     U  ] x11-libs/gdk-pixbuf-2.36.12:2::gentoo [2.36.11:2::gentoo] USE="X jpeg tiff -debug -introspection -jpeg2k {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U  ] media-libs/harfbuzz-1.7.6:0/0.9.18::gentoo [1.7.2:0/0.9.18::gentoo] USE="cairo fontconfig glib icu truetype -debug -graphite -introspection -static-libs {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U  ] x11-libs/gtk+-3.22.29:3::gentoo [3.22.19:3::gentoo] USE="X cups xinerama (-aqua) -broadway -cloudprint -colord -examples -introspection {-test} -vim-syntax -wayland" ABI_X86="(64) -32 (-x32)" 0 KiB

Total: 9 packages (8 upgrades, 1 in new slot), Size of downloads: 2,024 KiB


This time portage revealed the culprit (skipping the tree version of the output):

Code:

# emerge -Dpu libxml2

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  NS    ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[ebuild     U  ] dev-libs/libxml2-2.9.8:2::gentoo [2.9.7:2::gentoo] USE="icu ipv6 lzma python readline -debug -examples -static-libs {-test}" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 -python3_4 -python3_5 -python3_6" 0 KiB

Total: 2 packages (1 upgrade, 1 in new slot), Size of downloads: 2,024 KiB


A quick "<=sys-libs/readline-6" in /etc/portage/package.mask will prevent that from showing up in the future (but I don't really understand why it is happening in the first place).icu ipv6 lzma python readline -debug -examples -static-libs {-test}
Back to top
View user's profile Send private message
zmedico
Developer
Developer


Joined: 02 Jan 2004
Posts: 333
Location: California USA

PostPosted: Sat Apr 28, 2018 9:02 pm    Post subject: Reply with quote

curmudgeon wrote:


This time portage revealed the culprit (skipping the tree version of the output):

Code:

# emerge -Dpu libxml2

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  NS    ] sys-libs/readline-5.2_p14:5::gentoo [6.2_p5-r2:0::gentoo] ABI_X86="(64) -32 (-x32)" 2,024 KiB
[ebuild     U  ] dev-libs/libxml2-2.9.8:2::gentoo [2.9.7:2::gentoo] USE="icu ipv6 lzma python readline -debug -examples -static-libs {-test}" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 -python3_4 -python3_5 -python3_6" 0 KiB

Total: 2 packages (1 upgrade, 1 in new slot), Size of downloads: 2,024 KiB


The doesn't show the true cause, because there's no reason for dev-libs/libxml2-2.9.8:2 to pull in an older readline.

A debug log created as follows would reveal the answer:
Code:

emerge -Dpu libxml2 --debug 2>&1 > debug.log
xz -9 debug.log

_________________
Zac
Back to top
View user's profile Send private message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1485

PostPosted: Sat Apr 28, 2018 11:13 pm    Post subject: Reply with quote

zmedico wrote:
The doesn't show the true cause, because there's no reason for dev-libs/libxml2-2.9.8:2 to pull in an older readline.

A debug log created as follows would reveal the answer:
Code:

emerge -Dpu libxml2 --debug 2>&1 > debug.log
xz -9 debug.log


I already did the upgrades, and now (of course), I can't reproduce it. I have removed the line from package.mask, and will report if it ever occurs again.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum