Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[portage feature request] Ebuild mtime dependant remerge
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722
Location: Cluj-Napoca, Romania

PostPosted: Wed Jul 21, 2004 6:41 pm    Post subject: [portage feature request] Ebuild mtime dependant remerge Reply with quote

Seeing as ebuilds sometimes (ok, often :D) get updated without bumping the version number of the package, I'd like to propose / request a portage feature that allows the user to specify whether to re-merge packages when the ebuild modification time changes into the future. The way I'd see this would be to:

  • record the original ebuild modification time under /var/db/pkg/category/package-version/SOMEFILE
  • have a make.conf FEATURE flag (defaulting to off of course ;-) ) that would cause portage to check the /usr/portage ebuild against the recorded mtime and offer a remerge if the /usr/portage ebuild is newer
  • have portage show up a distinct letter when -pv-ing (a la [ T ], similar to [ U ] [ N] etc) to signal that the /usr/portage ebuild is newer.


Any thoughts on this?
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
cold_flame
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jun 2003
Posts: 88
Location: Australia

PostPosted: Thu Jul 22, 2004 1:57 pm    Post subject: Reply with quote

i'm not too sure about how mtimes work, but wouldn't they reflect the last time the rsync mirror was synched? not when the package was added?

in which case you would always have ebuild with newer mtimes?

or are mtimes static when copied onto a new filesystem?
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Thu Jul 22, 2004 3:35 pm    Post subject: Reply with quote

rsync will only modify the file if the server-side copy has changed, so if the ebuild hasn't been updated, the mtime on your local copy should stay constant.
Back to top
View user's profile Send private message
robmoss
Retired Dev
Retired Dev


Joined: 27 May 2003
Posts: 2634
Location: Jesus College, Oxford

PostPosted: Thu Jul 22, 2004 3:47 pm    Post subject: Reply with quote

Sounds like a great idea to me! Of course, it would be nice if keyword changes unrelated to the current arch could be ignored (perhaps a remerge would be "warranted" by marking something stable though).
_________________
Reality is for those who can't face Science Fiction.

emerge -U will kill your Gentoo
ecatmur, Lord of Portage Bash Scripts
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Thu Jul 22, 2004 5:17 pm    Post subject: Reply with quote

OK, apply this patch to emerge, then run 'emerge -p changes'. It's a bit hackish, in that it just ignores any package that causes it problems, but it seems to have the desired effect. It's been tested to the extent of touching the xterm ebuild and seeing it show up in the list, and on that count it works.
Back to top
View user's profile Send private message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722
Location: Cluj-Napoca, Romania

PostPosted: Thu Jul 22, 2004 8:30 pm    Post subject: Reply with quote

Whoa, that was fast! :D Gotta love Python. Thanks, thebell, testing.
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
pkdawson
Retired Dev
Retired Dev


Joined: 16 Jul 2004
Posts: 146
Location: Long Island or Buffalo, NY

PostPosted: Thu Jul 22, 2004 8:38 pm    Post subject: Re: [portage feature request] Ebuild mtime dependant remerge Reply with quote

moocha wrote:
Seeing as ebuilds sometimes (ok, often :D) get updated without bumping the version number of the package, I'd like to propose / request a portage feature that allows the user to specify whether to re-merge packages when the ebuild modification time changes into the future.


It's not really necessary. Gentoo policy is that when ebuilds get updated without a revision bump, the change is usually small and only affects a minority of users. For example, there's absolutely no reason you'd want to recompile when all that was added is a compile fix for those who couldn't compile it in the first place.

See:
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=3&chap=1

If you're obsessive-compulsive about staying on the bleeding edge, well, okay :) But it doesn't make a lot of sense to "upgrade" when you're not actually upgrading at all.
Back to top
View user's profile Send private message
robmoss
Retired Dev
Retired Dev


Joined: 27 May 2003
Posts: 2634
Location: Jesus College, Oxford

PostPosted: Thu Jul 22, 2004 9:38 pm    Post subject: Reply with quote

In some cases, it is necessary. x11-base/xorg-x11-6.7.0, for example, has had more changes without a revision bump than I could reasonably expect to count - that ebuild tends to get updated every couple of hours at least. Now, this is great, because there's loads of patches that do all sorts of useful things, and I'm the sort of person who'd actually use them, but right now I'm having to use ecatmur's revcheck.sh script to check for differences in the ebuild.

The Gentoo policy exists for a reason - to make sure that people don't spend more time compiling than their computers can manage, unless they have a really unfeasibly slow machine. But if I've got an Athlon64 and it only takes half an hour to build X.org, why should I worry about recompiling it? Particularly when that's the biggest package I have (I think).
_________________
Reality is for those who can't face Science Fiction.

emerge -U will kill your Gentoo
ecatmur, Lord of Portage Bash Scripts
Back to top
View user's profile Send private message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722
Location: Cluj-Napoca, Romania

PostPosted: Thu Jul 22, 2004 9:47 pm    Post subject: Re: [portage feature request] Ebuild mtime dependant remerge Reply with quote

pkdawson wrote:
moocha wrote:
Seeing as ebuilds sometimes (ok, often :D) get updated without bumping the version number of the package, I'd like to propose / request a portage feature that allows the user to specify whether to re-merge packages when the ebuild modification time changes into the future.


It's not really necessary. Gentoo policy is that when ebuilds get updated without a revision bump, the change is usually small and only affects a minority of users. For example, there's absolutely no reason you'd want to recompile when all that was added is a compile fix for those who couldn't compile it in the first place.

See:
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=3&chap=1

If you're obsessive-compulsive about staying on the bleeding edge, well, okay :) But it doesn't make a lot of sense to "upgrade" when you're not actually upgrading at all.


Or when you really need to be on the bleeding edge because you're maintaining a package that depends one some other package whose ebuild has been silently updated.

Think before you type :)
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
Lv
Retired Dev
Retired Dev


Joined: 22 Jun 2003
Posts: 352

PostPosted: Thu Jul 22, 2004 9:54 pm    Post subject: Re: [portage feature request] Ebuild mtime dependant remerge Reply with quote

pkdawson wrote:

It's not really necessary. Gentoo policy is that when ebuilds get updated without a revision bump, the change is usually small and only affects a minority of users. For example, there's absolutely no reason you'd want to recompile when all that was added is a compile fix for those who couldn't compile it in the first place.


not necessarily true, major amd64 fixes have gone into ebuilds without bumping because arch maintainers arent generally allowed to bump ebuilds... especially since said bugs tend to only effect one arch.

this has bothered me so much i wrote a script to do revision checking... http://dev.gentoo.org/~lv/revcheck.sh
and then ecatmur updated it a bit to handle overlays: http://dev.gentoo.org/~lv/revcheck2.sh
Back to top
View user's profile Send private message
Lv
Retired Dev
Retired Dev


Joined: 22 Jun 2003
Posts: 352

PostPosted: Thu Jul 22, 2004 10:36 pm    Post subject: Reply with quote

thebell wrote:
OK, apply this patch to emerge, then run 'emerge -p changes'. It's a bit hackish, in that it just ignores any package that causes it problems, but it seems to have the desired effect. It's been tested to the extent of touching the xterm ebuild and seeing it show up in the list, and on that count it works.


doesnt work here.

Code:
ayanami bin # emerge -p changes
 
These are the packages that I would merge, in order:
 
Calculating changes dependencies \Traceback (most recent call last):
  File "/usr/bin/emerge", line 2677, in ?
    if not mydepgraph.xcreate(myaction):
  File "/usr/bin/emerge", line 1201, in xcreate
    myeb=portage.portdb.xmatch("bestmatch-visible",mydep)
  File "/usr/lib/portage/pym/portage.py", line 5338, in xmatch
    mydep=dep_expand(origdep,self)
  File "/usr/lib/portage/pym/portage.py", line 3689, in dep_expand
    return prefix+cpv_expand(mydep,mydb,use_cache=use_cache)+postfix
  File "/usr/lib/portage/pym/portage.py", line 3633, in cpv_expand
    raise ValueError, matches
ValueError: ['app-xemacs/ocaml', 'dev-lang/ocaml']


after changing mylines.append("="+my_pv) to mylines.append("="+my_p+"/"+my_pv) -

Code:
ayanami bin # touch /portcvs/sys-libs/libstdc++-v3/libstdc++-v3-3.3.3-r1.ebuild ayanami bin # emerge -p changes                                                                                 
These are the packages that I would merge, in order:
 
Calculating changes dependencies  ...done!
 
ayanami bin # touch /usr/portage/sys-libs/libstdc++-v3/libstdc++-v3-3.3.3-r1.ebuild
ayanami bin # emerge -p changes                                                                                 
These are the packages that I would merge, in order:
 
Calculating changes dependencies  ...done!
Back to top
View user's profile Send private message
Lv
Retired Dev
Retired Dev


Joined: 22 Jun 2003
Posts: 352

PostPosted: Thu Jul 22, 2004 10:38 pm    Post subject: Reply with quote

i would also like to note that ferringb doesnt seem to mind the idea of including this patch into portage proper once it works right. so if you want to see this feature... get to work! :)

Quote:
<ferringb> Lv: mm'kay (come back with a working patch) :)
<ferringb> hmm.
<ferringb> Lv: patch needs work, should be calling aux_get(cat/pkg-ver, ["_MTIME_"]) instead of pulling from the ebuild

stuff, stuff...
Quote:
<ferringb> yeah, it could be used to hunt for QA violations also :)
<ferringb> Lv: well, right now it's not catching adjustments in the tree (abusing touch); if that's ironed out (along w/ a few other tweaks) it's possible


i would also suggest something that filters out changes we dont care about... like comment typo fixes and keyword changes. whitespace changes. etc. poke at http://dev.gentoo.org/~lv/revcheck2.sh for a general idea of what i mean. :)
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Thu Jul 22, 2004 11:17 pm    Post subject: Reply with quote

OK, I'll get down to improving it tomorrow, with any luck. ;) I did write it in about quarter of an hour with hardly any knowledge of portage internals, so you can't expect that much from it. ;)
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9004
Location: beyond the rim

PostPosted: Fri Jul 23, 2004 6:54 am    Post subject: Reply with quote

IIRC Nick was against that functionality, see bug 39501 :roll:
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Fri Jul 23, 2004 9:54 am    Post subject: Reply with quote

OK, I've got a newer version up here that uses portdbapi.aux_get instead of crawling through the tree, but the vardbapi.aux_get function doesn't seem to do mtimes afaict, so it still pulls that mtime from the filesystem.

@Genone: AFAICT, Nick's objections were that (1) people would be rebuilding unnecessarily, and (2) that it would add too much complexity. This approach seems to get around them both, since it's seperate from a normal update, and is actually fairly simple to implement.

@Lv: were you copy+pasting the patch? I've found that Python really doesn't like that. ;) I can't seem to reproduce that here (on portage-2.0.51_pre13 if it's of interest).
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Fri Jul 23, 2004 12:10 pm    Post subject: Reply with quote

Nicer version here which checks ebuild contents rather than mtimes, doesn't manually walk the package database, and also (in theory) ignores comments and KEYWORDS changes.
Back to top
View user's profile Send private message
Lv
Retired Dev
Retired Dev


Joined: 22 Jun 2003
Posts: 352

PostPosted: Sun Jul 25, 2004 1:40 am    Post subject: Reply with quote

it also ignores possibly important changes since it only counts lines... and it may cause false positives from removing whitespace. at least that' what i get from just taking a quick peek at it.
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Mon Aug 09, 2004 11:03 am    Post subject: Reply with quote

Sorry for the delayed response-- I've been away for the last 2 weeks. ;)

What it does in the latest version is to count the lines, and if they differ, add it to the list. If they have the same number of lines, then it goes through and actually compares them. If the ebuilds differ on a line that doesn't begin 'KEYWORDS=', it adds that package to the list. As for whitespace changes, portage.grabfile ought to handle that.

BTW: update here; I just removed some debug statements I'd missed earlier...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum