Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
`emerge --info` is not deterministic
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
nmel
n00b
n00b


Joined: 12 Mar 2017
Posts: 14

PostPosted: Wed May 23, 2018 5:59 am    Post subject: `emerge --info` is not deterministic Reply with quote

Hi,

I noticed that my `emerge --info` output became not deterministic a few weeks ago. In particular, the last two lines change order from run to run:

Code:
gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-metamanifest: no
    sync-rsync-extra-opts:


I tracked down the system state and it appears after upgrade from sys-apps/portage-2.3.19-r1 to sys-apps/portage-2.3.24-r1 (the "sync-rsync-verify-metamanifest: no" appeared in the output at the same time). Is it a bug or `emerge --info` is not guaranteed to be deterministic?
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 534
Location: Frankfurt, Germany

PostPosted: Wed May 23, 2018 11:36 am    Post subject: Reply with quote

Please try:
Code:
PYTHONHASHSEED=0 emerge --info

Does this bring back determinism? I saw this trick in this thread, posts 30-33.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Wed May 23, 2018 4:30 pm    Post subject: Reply with quote

you could also set a python2 version per default (salt was add in 3+)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12607

PostPosted: Thu May 24, 2018 2:06 am    Post subject: Reply with quote

Technically, hash salting was added to the late 2.x series. It only defaults to enabled in 3.3 and later though. You can enable it in 2.7.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9220
Location: beyond the rim

PostPosted: Thu May 24, 2018 2:52 pm    Post subject: Reply with quote

Probably nobody so far bothered about the order regarding repo options of --info output. And yes, it's most certainly the hash issue again, but as most keys have specific codepaths for output it won't trigger that often:
Code:
      for o, v in self.module_specific_options.items():
         if v is not None:
            repo_msg.append(indent + o + ": " + v)

dict.items() returns a list, but doesn't define any specific order for it.
Back to top
View user's profile Send private message
nmel
n00b
n00b


Joined: 12 Mar 2017
Posts: 14

PostPosted: Mon May 28, 2018 4:50 am    Post subject: Reply with quote

Quote:
PYTHONHASHSEED=0 emerge --info

Unfortunately, this workaround doesn't work for me.

I ended up using the following workaround so far
Code:
sudo emerge --info | grep -Ev '(Mem|Swap):' | sed -e '/^    sync-rsync-extra-opts: $/d'

However it's just a temporary and ugly fix.

Genone wrote:
Probably nobody so far bothered about the order regarding repo options of --info output.

I care because I track system configuration this way (as a part of a larger tracking setup).

Genone wrote:

And yes, it's most certainly the hash issue again, but as most keys have specific codepaths for output it won't trigger that often:
Code:
      for o, v in self.module_specific_options.items():
         if v is not None:
            repo_msg.append(indent + o + ": " + v)

dict.items() returns a list, but doesn't define any specific order for it.


Can we change to
Code:
sorted(self.module_specific_options.items())

?
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9220
Location: beyond the rim

PostPosted: Mon May 28, 2018 8:32 am    Post subject: Reply with quote

You'll have to file a bug about it, I have nothing to do with Gentoo development anymore.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5096
Location: The Peanut Gallery

PostPosted: Wed May 30, 2018 6:10 pm    Post subject: Reply with quote

nmel wrote:
However it's just a temporary and ugly fix.
..
I care because I track system configuration this way (as a part of a larger tracking setup).
I would definitely submit your one-line patch, but I must ask: what are you using to read this info? It is trivial to read in both sh and awk; the latter would be my preference, especially if you are already using grep in any fashion (since you can combine that into awk.) The point being that your script can be robust against ordering changes; awk associative arrays (aka hash-tabs) are very useful.

Having said that, you can combine the above into one grep call:
Code:
grep -Ev '(Mem|Swap|sync-rsync-extra-opts):'
Nonetheless, I highly recommend #awk on IRC: chat.freenode.org or .net
Back to top
View user's profile Send private message
nmel
n00b
n00b


Joined: 12 Mar 2017
Posts: 14

PostPosted: Sun Jun 03, 2018 5:09 am    Post subject: Reply with quote

steveL wrote:
I would definitely submit your one-line patch

Thanks!!

steveL wrote:
but I must ask: what are you using to read this info? It is trivial to read in both sh and awk; the latter would be my preference, especially if you are already using grep in any fashion (since you can combine that into awk.) The point being that your script can be robust against ordering changes; awk associative arrays (aka hash-tabs) are very useful.

Yes, awk is very nice, indeed. However I don't use anything to read the info. I simply store it and do diff with the current version (to be precise, git does this for me). I'm tracking whether my system has changes or not, so I'm able to find an unexpected change. That's why I need determinism here.

steveL wrote:
Having said that, you can combine the above into one grep call:
Code:
grep -Ev '(Mem|Swap|sync-rsync-extra-opts):'
Nonetheless, I highly recommend #awk on IRC: chat.freenode.org or .net

Thanks for your suggestion. My initial version is more robust in a sense that if I do modify sync-rsync-extra-opts, it will show up in the output. I don't want to blindly strip it.
Back to top
View user's profile Send private message
nmel
n00b
n00b


Joined: 12 Mar 2017
Posts: 14

PostPosted: Sun Jun 24, 2018 5:45 am    Post subject: Reply with quote

nmel wrote:
Quote:
PYTHONHASHSEED=0 emerge --info

Unfortunately, this workaround doesn't work for me.

I finally figured out why. It's because I used sudo to run the command, i.e. I actually run
Code:
PYTHONHASHSEED=0 sudo emerge --info

This is not correct because the env var is not passed to sudo environment. The correct way is the following.
Code:
sudo PYTHONHASHSEED=0 emerge --info

and you need to enable the env var in sudoerc like this:
Code:
Defaults:%wheel   env_keep += PYTHONHASHSEED

Just in case someone comes across this thread. BTW, my previous workaround stopped working since now there are many options that jump back and forth.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5096
Location: The Peanut Gallery

PostPosted: Sun Jun 24, 2018 11:05 pm    Post subject: Reply with quote

steveL wrote:
Having said that, you can combine the above into one grep call:
Code:
grep -Ev '(Mem|Swap|sync-rsync-extra-opts):'
nmel wrote:
Thanks for your suggestion. My initial version is more robust in a sense that if I do modify sync-rsync-extra-opts, it will show up in the output. I don't want to blindly strip it.
Then just roll in the whole sed regex into the egrep call:
Code:
grep -Ev '(Mem:|Swap:|^    sync-rsync-extra-opts: $)'
It's an ERE already, so you might as well save a process.

Does sound like that patch needs to go in, though, or it needs to be raised for an alternative correction.
Might want to file a bug with the patch, if you didn't already.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum