Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Proposal: new emerge option --older-than
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Mon Nov 28, 2011 12:48 am    Post subject: Proposal: new emerge option --older-than Reply with quote

Hi. I think a new emerge option "--older-than" would facilitate things, at least for people like me: I rebuild (almost) all _only_ once a year. I never use "emerge -e world" which pulls in too many at once. I like piecemeal way; linux-headers first, then glibc, etc. But it's cumbersome.

Here comes --older-than. $ emerge -e --older-than sys-kernel/linux-headers sys-apps/util-linux should be read as "emerge util-linux, rebuilding dependent packages older-than linux-headers."; it will upgrade/rebuild all that util-linux needs, recursively solving dependency, except linux-headers and newer. Obviously --older-than is only useful when it's used with -e (=--emptytree) or -D (= --deep).

This is extremely useful. For example you can ensure that all Xorg related packages are rebuilt (or emacs dependency, or whichever) but without touching gcc, glibc, and so on.

Maybe three flavors would be good: --older-than-package / -file / -date, but I'm not sure.

Technically, it's already possible using "--excludes" option with find (1), but I'm speaking of convenience, not the possibility. ;)

# There's a subtlety. Suppose you run emerge -e --older-than foo new-pkg. If foo depends on bar, and bar is older than foo, then should it rebuild bar? Dunno, not an expert.

# Subtlety of the name: If new-pkg depends on baz, and baz is not installed, then baz should be pulled in, even though it's not "older-than" foo. But the accurate name --except-newer-and is daunting.

What do you think?


Last edited by teika on Thu Dec 01, 2011 8:55 am; edited 1 time in total
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2521
Location: Richmond, BC, Canada

PostPosted: Mon Nov 28, 2011 3:28 pm    Post subject: Reply with quote

Portage-2.2 already can:

http://forums.gentoo.org/viewtopic-t-828918-highlight-ageset.html
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 7667
Location: Somewhere over Atlanta, Georgia

PostPosted: Mon Nov 28, 2011 3:40 pm    Post subject: Reply with quote

I'm not saying that it would be beautiful, but I would like to see a syntax whereby these types of sets could be ephemerally created on the command line. It couldn't possibly be uglier than, say, the C++ pointer to a function returning a reference to an array syntax.

- John
_________________
This space intentionally left blank.
Back to top
View user's profile Send private message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Wed Nov 30, 2011 1:48 am    Post subject: Reply with quote

@dol-sen

I see, but it seems the opposite from what I want; --older-than is to *cut down* the rebuild/upgrade dependency which e.g. emacs requires. AgeSet is to choose *all* old packages, right? Or is it possible to choose both in AgeSet and dependency, i.e. a set intersection? (Seems no from a glance of the latest portage-2.2.)

Well, you may think it's not necessary, but I never do emerge [opts] world, so it'd be nice for me. Of course AgeSet benefits me, too. (I upgrade all once a year, breaking things into pieces.)

@John R. Graham
You said in the thread cited by dol-sen that -e doesn't update but simply rebuild. But it does seem to upgrade to the latest version. Right? (-e pretends nothing is installed, according to man emerge. If it doesn't upgrade, then the manual needs fix.)
I tried emerge -pe system. For example sys-apps/kbd is in system, and emerge says it'll be upgraded.
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2521
Location: Richmond, BC, Canada

PostPosted: Wed Nov 30, 2011 2:23 am    Post subject: Reply with quote

No, -e does not specifically cause it to update. But since the tree is constantly changing, ebuilds, pkgs come and go and deps are changed. The process of running the -e can require some updates, which may trigger more updates. The main update stimulus is the -u and -D options.

No the AgeSet does not do set intersection like you want. A more common use of AgeSet is to rebuild all packages built with an older gcc/glibc toolchain for instance.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Wed Nov 30, 2011 1:06 pm    Post subject: Reply with quote

Hmm, perhaps sometimes portage developers define:
* rebuild: emerge again.
* upgrade: emerge if there's a new version, otherwise do nothing.

If so:
I understand what JRG meant: -e is not upgrade, only -u does. But this wording is different from ordinary users'. I understand the sentence "-e does not upgrade but rebuild" as "rebuild the same version".

For developers there must be reasons for this distinction, and it's good that there's nomenclature that allows quick and precise communication. But they're internal jargons, and you've got be careful.

My name is Yoda. This familyname is not rare in Japan. - 依田某
_________________
Easy Shift / Ctrl / AltGr ... hack; save your pinkies, type without drudge: topic 865313


Last edited by teika on Thu Dec 01, 2011 8:17 am; edited 2 times in total
Back to top
View user's profile Send private message
krinn
Advocate
Advocate


Joined: 02 May 2003
Posts: 4151

PostPosted: Wed Nov 30, 2011 2:25 pm    Post subject: Reply with quote

teika wrote:

I tried emerge -pe system. For example sys-apps/kbd is in system, and emerge says it'll be upgraded.


Because it's rebuild kbd, and not rebuild kbd-0.9, and if you have then kbd-0.9 install, but the kbd-0.9 ebuild isn't in portage anymore, then rebuild kdb will rebuild (upgrade) to a newer version.

If you push in your world, not kbd, but kbd-0.9 instead, you'll get -e to try to rebuild kbd-0.9 and get a message that this ebuild doesn't exist in the tree anymore.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4009

PostPosted: Mon Aug 27, 2012 6:10 pm    Post subject: Reply with quote

dol-sen wrote:
Portage-2.2 already can:

http://forums.gentoo.org/viewtopic-t-828918-highlight-ageset.html

Sorry to reopen that old thread again, but I just recently saw the ageset, and unfortunately it seems it cannot do what you claim:
It can only update something older than a specified number of days, but neither something older than a specified date (unless you manually change the set definition every day) nor something older than a particular installation date.
After major gcc upgrades e.g. it would really be nice to see what was not yet recompiled with the new gcc.
I would not mind to specify the desired date once manually, but changing it every day makes no sense.
Of course, there is a solution (e.g. the eix-installed-after script with appropriate options, and of course the output of that script can be cast into a set), but a portage-only solution would be nice.
Perhaps it exists already and I just cannot see the obvious?
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2521
Location: Richmond, BC, Canada

PostPosted: Tue Aug 28, 2012 1:53 am    Post subject: Reply with quote

Yeah, unfortunately I had nothing to do with the AgeSet's creation. I believe it was genone, who did the original set programming in portage.

I also agree with you that it would be good to have a native portage capability to perform age related rebuilds which can understand dates/times, num-of-days, etc.. Not have to edit the config each time. Now that my emaint rewrite has been merged into portage. It could be possible to create a new emaint module that is capable of taking the date/time cli options and modify the AgeSet config parameters within portage's set handling code to do this. I don't really know the best way to go about it until I've been poking around that code more.

Can you open an enhancement bug for this so it doesn't get forgotten about. I have several other things I need to work on before adding this to my chore list :)
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9005
Location: beyond the rim

PostPosted: Tue Aug 28, 2012 2:46 pm    Post subject: Reply with quote

While AgeSet takes "number of days" as filter argument it would be almost trivial to extend it to accept a date or also packagename as reference (or create a new subclass to to simplify argument handling).
For reference, this is the whole definition of the AgeSet class. All you'd have to do is parse a date string (or packagename) in the singleBuilder() method, pass that into the constructor and in _filter() compare it to the path mtime.
Code:
 327 class AgeSet(EverythingSet):
 328         _operations = ["merge", "unmerge"]
 329
 330         def __init__(self, vardb, mode="older", age=7):
 331                 super(AgeSet, self).__init__(vardb)
 332                 self._mode = mode
 333                 self._age = age
 334
 335         def _filter(self, atom):
 336         
 337                 cpv = self._db.match(atom)[0]
 338                 path = self._db.getpath(cpv, filename="COUNTER")
 339                 age = (time.time() - os.stat(path).st_mtime) / (3600 * 24)
 340                 if ((self._mode == "older" and age <= self._age) \
 341                         or (self._mode == "newer" and age >= self._age)):
 342                         return False
 343                 else:
 344                         return True
 345         
 346         def singleBuilder(cls, options, settings, trees):
 347                 mode = options.get("mode", "older")
 348                 if str(mode).lower() not in ["newer", "older"]:
 349                         raise SetConfigError(_("invalid 'mode' value %s (use either 'newer' or 'older')") % mode)
 350                 try:
 351                         age = int(options.get("age", "7"))
 352                 except ValueError as e:
 353                         raise SetConfigError(_("value of option 'age' is not an integer"))
 354                 return AgeSet(vardb=trees["vartree"].dbapi, mode=mode, age=age)
 355
 356         singleBuilder = classmethod(singleBuilder)


As for the idea of creating sets on the commandline, I had actually implemented that long ago together with string based set operators. The latter were later removed in favor of atom-based operators (no idea if those are implemented by now), the former still exists. To use the above "rebuild all packages that were not rebuilt since last gcc update" example the syntax would be like this (untested):
Code:
emerge -p '@gcc-rebuild{class=dbapi.AgeSet,mode=older,refpackage=sys-devel/gcc}'
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4009

PostPosted: Sat Sep 01, 2012 6:49 pm    Post subject: Reply with quote

Genone wrote:
(or create a new subclass to to simplify argument handling).

I have written a corresponding (separate) DateSet class in which the date can be specified in several ways:
  • Explicitly (in a format which can also be specified).
  • Explicitly in seconds since epoch.
  • Implicitly as the installation date of a package.
  • Implicitly as a timestamp of a file.
The latter appears most useful to me, since e.g. you can touch some specified auxiliary file once you make a major gcc ugrade (and then minor gcc upgrades will not change this date).

Once bugzilla works again, I will open an enhancement bug with the patch. (Not sure whether the new class should be merged with the AgeSet class - as you remarked, it would be trivial to do so).

BTW: Is it really the proper way to check the date of a package by considering the timestamp of COUNTER? After all, portage supports BUILD_TIME since quite a while, but packages built with ancient portage versions not necessarily have it; isn't there a cleaner interface to get the build time?

Edit: Bugzilla is up again: Enhancement bug submitted.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9005
Location: beyond the rim

PostPosted: Mon Sep 03, 2012 7:45 am    Post subject: Reply with quote

mv wrote:
BTW: Is it really the proper way to check the date of a package by considering the timestamp of COUNTER? After all, portage supports BUILD_TIME since quite a while, but packages built with ancient portage versions not necessarily have it; isn't there a cleaner interface to get the build time?

Well, it didn't exist back when I wrote this stuff so COUNTER was the best option then, and I haven't really followed portage development over the last few years.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4009

PostPosted: Mon Sep 03, 2012 7:38 pm    Post subject: Reply with quote

Genone wrote:
Well, it didn't exist back when I wrote this stuff so COUNTER was the best option then, and I haven't really followed portage development over the last few years.

Thanks. Some investigation showed that the "portable" way is probably to use the BUILD_TIME key with aux_get - and if it doesn't exist, it is probably reasonable to assume that the build time is older than any chosen date/age. If you do not protest, I will file a corresponding bug (with patch) for the AgeSet and DateSet classes.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9005
Location: beyond the rim

PostPosted: Tue Sep 04, 2012 6:40 am    Post subject: Reply with quote

Feel free to update anything you think can be improved.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum