Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
emerge is non-deterministic?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 509

PostPosted: Fri May 11, 2018 10:49 pm    Post subject: emerge is non-deterministic? Reply with quote

I'm experiencing something of a curiosity with portage-2.3.24-r1.

As there's still no recent stable gentoo-sources, I still have 4.12.12 installed. emerge suggests to also install 4.9.95, which is stable. But here's the kicker - not always!

I can run

Code:
emerge -pvDNu world


multiple times, and there's roughly a 50:50 chance that gentoo-sources will be in the list of merge candidates.

Huh??
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Sat May 12, 2018 12:36 am    Post subject: Reply with quote

Can we see the output of
Code:
grep -Fn sys-kernel /var/lib/portage/world
and the usual emerge --info please.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 509

PostPosted: Sat May 12, 2018 12:52 am    Post subject: Reply with quote

sure thing

Code:
# grep -Fn sys-kernel /var/lib/portage/world
195:sys-kernel/gentoo-sources
196:sys-kernel/gentoo-sources:4.12.12
197:sys-kernel/linux-firmware


I remember adding 4.12.12 a long time ago to prevent it from getting unmerged by depclean when I also had 4.14 installed (I don't have it anymore)

Code:
# emerge --info
Portage 2.3.24 (python 3.5.5-final-0, default/linux/amd64/17.0/desktop, gcc-6.4.0, glibc-2.25-r11, 4.12.12-gentoo-H x86_64)
=================================================================
System uname: Linux-4.12.12-gentoo-H-x86_64-Intel-R-_Core-TM-_i7-3920XM_CPU_@_2.90GHz-with-gentoo-2.4.1
KiB Mem:    24302688 total,   3594640 free
KiB Swap:   33554428 total,  32954860 free
Head commit of repository gentoo: 06fbdeb1f6f84e8a375a850426038ff227a9b280

sh bash 4.4_p12
ld GNU ld (Gentoo 2.29.1 p3) 2.29.1
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.5::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.15.1-r2::gentoo
sys-devel/binutils:       2.29.1-r1::gentoo
sys-devel/gcc:            6.4.0-r1::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.16-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.25-r11::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/repo/gentoo.git
    priority: -1000

bumblebee
    location: /var/db/repos/bumblebee
    sync-type: laymansync
    sync-uri: git://github.com/Bumblebee-Project/bumblebee-gentoo
    masters: gentoo
    priority: 50

jorgicio
    location: /var/db/repos/jorgicio
    sync-type: laymansync
    sync-uri: https://github.com/jorgicio/jorgicio-gentoo.git
    masters: gentoo
    priority: 50

rust
    location: /var/db/repos/rust
    sync-type: laymansync
    sync-uri: https://github.com/gentoo/gentoo-rust.git
    masters: gentoo
    priority: 50

spring
    location: /var/db/repos/spring
    sync-type: laymansync
    sync-uri: git://github.com/springlobby/overlay.git
    masters: gentoo
    priority: 50

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: laymansync
    sync-uri: https://github.com/anyc/steam-overlay.git
    masters: gentoo
    priority: 50

vmware
    location: /var/db/repos/vmware
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/proj/vmware.git
    masters: gentoo
    priority: 50

vortex
    location: /var/db/repos/vortex
    sync-type: laymansync
    sync-uri: git://github.com/nE0sIghT/vortex-overlay.git
    masters: gentoo
    priority: 50

x11
    location: /var/db/repos/x11
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/proj/x11
    masters: gentoo
    priority: 50

haarp
    location: /var/db/repos/haarp
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/user/haarp.git
    masters: gentoo
    priority: 100

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe -fno-stack-protector"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe -fno-stack-protector"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--ask-enter-invalid --alert --read-news --keep-going --quiet-unmerge-warn"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs cgroup config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles ipc-sandbox merge-sync multilib-strict news noinfo parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
INSTALL_MASK="/etc/systemd /lib/systemd /lib64/systemd /usr/lib/systemd /usr/lib64/systemd"
LANG="en_DK.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j8"
PKGDIR="/var/cache/packages"
PORTAGE_COMPRESS="xz"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi aio alsa amd64 aot archive berkdb bluetooth bluray branding bzip2 cairo caps cdda cdr cgroups cli consolekit crypt cryptsetup cups custom-cflags custom-optimization cxx dbus dri dts dvd dvdr emboss encode exif fam fat ffmpeg flac fuse gdbm geoip gif git glamor gphoto2 gtk gtkstyle hidpi http2 iconv ipv6 jpeg keybinder lcms libass libcanberra libinput libnotify lm_sensors lz4 lzo mad mng modemmanager modules mp3 mp4 mpeg mtp multilib ncurses networkmanager nls nptl nsplugin ntfs ntfsdecrypt offensive ogg opengl openmp pam pango pcre pdf pgo pm-utils png policykit ppds pulseaudio qrcode qt3support rar readline run-as-root s3tc samba sdl seccomp sftp smp ssh ssl staging startup-notification suid svg symlink syslog tcmalloc tcpd threads thunar tiff truetype udev udisks unicode upower usb v4l vaapi vdpau vorbis wifi wxwidgets x264 x265 xattr xcb xml xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx mmx mmxext sse sse2 sse3 ssse3 sse4 sse4_1 sse4_2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc efi-64" INPUT_DEVICES="libinput wacom" KERNEL="linux" L10N="en de" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="vesa intel nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Sat May 12, 2018 3:57 pm    Post subject: Reply with quote

My gut reaction is: with all those web overlays taking precedence, it's no wonder it's "undeterministic".

When it's changing behaviour in front of you, with no sync or other changes in between, can you get the output of
Code:
emerge -pv sys-kernel/gentoo-sources
Can you mask out sys-kernel/* from other overlays? Might need to mask it across all, and then unmask ::gentoo.
See what changes you get, also on an emerge -pv @system (then @world), keeping an eye out for overlay changes. (::fubar -> ::gentoo)
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 509

PostPosted: Sat May 12, 2018 5:12 pm    Post subject: Reply with quote

Yes, it's changing behavior during consecutive runs, without any changes to configs, syncing or anything else.
Code:
$ emerge -pv sys-kernel/gentoo-sources

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  NS    ] sys-kernel/gentoo-sources-4.9.95:4.9.95::gentoo [4.12.12:4.12.12::gentoo] USE="experimental symlink -build" 92,990 KiB

Total: 1 package (1 in new slot), Size of downloads: 92,990 KiB

No surprises here.

Let's mask sys-kernel/* and unmask sys-kernel/*::gentoo. This also unmasks sys-kernel/hardened-sources-4.8.17-r2::gentoo. Now emerge -pvDNu world randomly switches between wanting to merge gentoo-sources and hardened-sources. :lol:

Since hardened-sources are not part of my worldfile, it's presence indicates that this might be the work of virtual/linux-sources-3::gentoo, which I have installed.


To exclude the possibility of overlays being responsible, I've disabled all of them, expect for ::gentoo itself. It still oscillates.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Sun May 13, 2018 1:07 pm    Post subject: Reply with quote

haarp wrote:
To exclude the possibility of overlays being responsible, I've disabled all of them, expect for ::gentoo itself. It still oscillates.
OK, at this point (if it were my machine) I would /join #gentoo-portage on IRC: chat.freenode.org or .net and wait for zmedico to be around.
Weird bugs like this come up all the time, and while it might just be a config snafu, it could also be a resolver bug.
They tend not to show up, unless under specific circumstance, and it's much easier if the coder can ask questions and get files in realtime (for them; not for the reporter) at the time it's happening.
It's also the quickest way to find the config snafu.

You could file a bug, if you wanted, but it will take an awful lot longer (I'd estimate a couple of months, vs a couple of hours.)
Another approach is to file the bug, then try IRC after you get a response, and it looks like it's going to need more and more info, or there doesn't seem to be much response.

Other than that, hopefully genone will see this topic.

Sorry I cannot be of more help; I don't code on portage, and I've never seen it oscillate between package choice like that.
It smells like a bug to me, fwtw; even if it's just under certain circumstance, it should complain instead, imo.

With hardened-sources, either mask it again specifically after the unmask for ::gentoo, or unmask sys-kernel/gentoo-sources and linux-firmware (not sys-kernel/*)

edit: Hmm, how come you're getting a lower slot for gentoo-sources?
Back to top
View user's profile Send private message
saboya
Guru
Guru


Joined: 28 Nov 2006
Posts: 395
Location: Brazil

PostPosted: Sun May 13, 2018 3:22 pm    Post subject: Reply with quote

Apparently --pretend is non-deterministic. Yours is not the first post I've seen about this.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 509

PostPosted: Mon May 14, 2018 7:21 am    Post subject: Reply with quote

Interesting. I indeed found another thread about it - https://forums.gentoo.org/viewtopic-t-1047456-start-0.html

Still weird tho. Isn't dependency resolving single-threaded? Can't be a race-condition then.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Mon May 14, 2018 2:32 pm    Post subject: Reply with quote

Non-deterministic wrt the final order it spits out (which isn't precise under --jobs anyway.)

Missing updates are another matter.

Oscillation smells like a bug to me, likely triggered under specific config, with a specific combination of installed packages, etc.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9221
Location: beyond the rim

PostPosted: Wed May 16, 2018 1:51 pm    Post subject: Reply with quote

steveL wrote:
Oscillation smells like a bug to me, likely triggered under specific config, with a specific combination of installed packages, etc.


If --pretend results change without any (significant) change to system state it's definitely a bug of some kind, as it completely breaks the definition of --pretend. Could simply be a codepath iterating over an unordered data structure, and depending on the actual order packages are included/excluded in the final result.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6148

PostPosted: Wed May 16, 2018 2:32 pm    Post subject: Reply with quote

Genone wrote:
Could simply be a codepath iterating over an unordered data structure

I would be surprised if that (i.e. preallocated size or choice of hash function) would be non-deterministic in python if the order of insertion of the data is deterministic. The latter seems likely unless multiple threads are used. And once the metadata is calculated (which presumably spans multiple threads but happens only at the first (few) call(s)): Why should portage use multiple threads before the end of dependency resolution in the presence of GIL? (Which BTW is IMHO a major reason why python was not a fortunate choice as a language for a PM).
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Wed May 16, 2018 4:22 pm    Post subject: Reply with quote

Genone wrote:
If --pretend results change without any (significant) change to system state it's definitely a bug of some kind, as it completely breaks the definition of --pretend.
Thanks for confirming.

@mv: I used to think that, but then I tried pkgcore.
Back to top
View user's profile Send private message
Dr.Willy
Guru
Guru


Joined: 15 Jul 2007
Posts: 453
Location: NRW, Germany

PostPosted: Wed May 16, 2018 5:19 pm    Post subject: Reply with quote

steveL wrote:
@mv: I used to think that, but then I tried pkgcore.

Well yes, python is great if you replace enough of it with c.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Wed May 16, 2018 9:25 pm    Post subject: Reply with quote

Genone wrote:
If --pretend results change without any (significant) change to system state it's definitely a bug of some kind, as it completely breaks the definition of --pretend.

Yes, but it wouldn't be really a surprise as this scenario is causing trouble as user has define this rule:
#1 i want gentoo-sources stable and latest
#2 i want gentoo-sources-4.12.12 which is not stable

If you take try to solve the problem, looking at need #2 is easy, he want 4.12.12 just give him 4.12.12
But if you look at #1, technically user have 0 stable gentoo-sources install (4.12.12 is not stable, and 4.9.95 is a good candidate then), but at the same time, if you just look at "any gentoo sources install?", then 4.12.12 met this.

To me the right answer is what portage is offering "sometimes", having 4.12.12 as per user wish, but also having 4.9.95 (that's the answer to met the "gentoo-sources" put in his world).

The usual, mixing stable and unstable is never good, also for portage it seem.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 509

PostPosted: Wed May 16, 2018 9:30 pm    Post subject: Reply with quote

Yes, my mix of stable/unstable is less than ideal. I accept that some breakage may occur with weird combinations. But I expect breakage to be consistent at least :P This bug is not critical, but quite peculiar.

I hope I'll get around to filing a bug report in the coming week.

And thanks for your responses!
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Thu May 17, 2018 1:31 am    Post subject: Reply with quote

haarp wrote:
This bug is not critical, but quite peculiar.

I hope I'll get around to filing a bug report in the coming week.

Agree, this should be fix, mixing stable/unstable make it a bit a pain for user, but portage shouldn't had any handling that.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12619

PostPosted: Thu May 17, 2018 1:40 am    Post subject: Reply with quote

Non-deterministic dictionary key ordering is standard in recent versions of Python. There was a big scare a few years ago about web applications initializing dictionaries with untrusted user input as keys, because a malicious user could submit a set of keys that would cause collisions and force extremely bad lookup performance. The workaround from the language authors was to bias the key->bucket transformation with a random value not knowable to the attacker (generally picked once at program initialization time). For a simple demonstration, run the below program multiple times and observe the output. On patched versions of Python, you get a different ordering of words each time.
Code:
d = {}
with open('/usr/share/dict/words', 'rt') as fp:
   for i, line in enumerate(fp):
      d[line] = i

print(''.join(d.keys()))
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6148

PostPosted: Thu May 17, 2018 1:03 pm    Post subject: Reply with quote

Hu wrote:
Non-deterministic dictionary key ordering is standard in recent versions of Python.

Thank you very much for this interesting information. Also the reason for this makes sense.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Thu May 17, 2018 3:20 pm    Post subject: Reply with quote

steveL wrote:
@mv: I used to think that, but then I tried pkgcore.
Dr.Willy wrote:
Well yes, python is great if you replace enough of it with c.
Heh, that's one way of putting it.
But python works well as a high-level language, interfacing to C for speed where needed; and ferringb has done an excellent job with pkgcore.

So I don't think we can say "python is a bad choice", given the stonking proof of the opposite: the best package manager that ever came out of all the years of Gentoo.

Obviously, that's just my opinion, but it is based on scripting against all of them.

Here's another: ferringb is the most talented programmer ever to accept a Gentoo badge.
Gentoo should have been proud of nurturing his talent, instead of hiding it under a tonne of McCreesh shite.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5101
Location: The Peanut Gallery

PostPosted: Thu May 17, 2018 3:22 pm    Post subject: Reply with quote

Hu wrote:
Non-deterministic dictionary key ordering is standard in recent versions of Python.
Good info, but I really don't see what that has to do with missing a package.

Even if it is part of the symptom, it's still a bug, and I don't think ordering will be relevant once it's fixed (by definition.)
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6148

PostPosted: Thu May 17, 2018 3:41 pm    Post subject: Reply with quote

steveL wrote:
But python works well as a high-level language, interfacing to C for speed where needed

If all of the dependency resolver is written in C (or C++) then one can argue whether C (or C++) is actually the main implementation language of the pm.
If any part is not written in C/C++ then this part cannot be parallelised.
The problem is that one "step" in dependency resolving is NP complete, and no matter how you do it, this "step" is quite large and (no matter how well the implementation is written) can get roughly 8 times the speed if 8 cores are used...
As someone here and on some ml suggested, this step can be a SAT solver written in a different language (maybe pkgcore goes this route?) but I would then tend to argue that the language in which this solver is implemented is the "main" language of the pm. The other languages (python and shell) or then only "kit" holding the thing together.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12619

PostPosted: Fri May 18, 2018 2:11 am    Post subject: Reply with quote

steveL wrote:
Hu wrote:
Non-deterministic dictionary key ordering is standard in recent versions of Python.
Good info, but I really don't see what that has to do with missing a package.
It may not. I was responding to the discussion about why the ordering changes across runs:
Genone wrote:
steveL wrote:
Oscillation smells like a bug to me
Could simply be a codepath iterating over an unordered data structure, and depending on the actual order packages are included/excluded in the final result.
mv wrote:
I would be surprised if that (i.e. preallocated size or choice of hash function) would be non-deterministic in python if the order of insertion of the data is deterministic.
I cannot comment on why non-deterministic ordering would influence the decision to include/exclude packages, although I like Genone's theory in that regard. For example, if the solver iterates over an unordered dictionary and builds a secondary data structure from visited elements, and the presence of package P1 in that secondary structure eliminates the need for package P2 when it pulls P3 from the dictionary, then the per-run ordering of the dictionary would control whether P2 is used based on whether P1 is found before P3. I am not saying the package manager should work this way, only that it's very plausible that the current implementation does.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6148

PostPosted: Fri May 18, 2018 4:21 am    Post subject: Reply with quote

Hu wrote:
I cannot comment on why non-deterministic ordering would influence the decision to include/exclude packages

Here is a concrete example: Assume that there are packages A B in world with dependencies || ( C D ) and || ( D C )), respectively (neither C nor D being installed nor having further dependencies for simplicity). Then whether C or D is preferred depends on whether A or B is processed first.

However, in the case of hardened-sources it really seems to be a bug: eix --deps hardened-sources shows that only virtual/gentoo-sources might pull it in, and gentoo-sources has no || dependencies (and all of the dependencies of gentoo-sources should already be satisfied, anyway). So portage should never have a reason to pull in hardened-sources. (Assuming the information is right that this happens without overlays.)
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9221
Location: beyond the rim

PostPosted: Fri May 18, 2018 2:13 pm    Post subject: Reply with quote

steveL wrote:
Hu wrote:
Non-deterministic dictionary key ordering is standard in recent versions of Python.
Good info, but I really don't see what that has to do with missing a package.

Even if it is part of the symptom, it's still a bug, and I don't think ordering will be relevant once it's fixed (by definition.)


Well, I was just thinking loud. If the input (=system state) of a function (=emerge call) doesn't change, what else could cause it to return different results? Some kind of (implicit) RNG application. And I had recently stumbled across a comment about the internal python hash function to be salted per process leading to different orders within unordered data structures. So that seemed like a realistic cause which could be easily overlooked if it only triggered in rare situations.

Constant input should lead to constant results for --pretend, no matter if the result is correct. Otherwise --pretend is broken by definition as you cannot rely on portage to actually show the action that would be performed.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Fri May 18, 2018 4:05 pm    Post subject: Reply with quote

Hu wrote:
For a simple demonstration, run the below program multiple times and observe the output. On patched versions of Python, you get a different ordering of words each time.

it randomize with python3 but not when i use python2, so that bug could be in portage for years (but not seen as long as user use python2)

that's not a good point for python, i could understand they need to fix this, but changing a function like that is not good ; why didn't they create a new randomize function doing the same but with salt? Users in need for it have to use the new one and everyone is happy.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum