| View previous topic :: View next topic |
| Author |
Message |
moocha
Watchman
Joined: 21 Oct 2003
Posts: 5722
|
 Posted: Wed Jul 21, 2004 6:41 pm Post subject: [portage feature request] Ebuild mtime dependant remerge |
 |
|
Seeing as ebuilds sometimes (ok, often  ) get updated without bumping the version number of the package, I'd like to propose / request a portage feature that allows the user to specify whether to re-merge packages when the ebuild modification time changes into the future. The way I'd see this would be to:
- record the original ebuild modification time under /var/db/pkg/category/package-version/SOMEFILE
- have a make.conf FEATURE flag (defaulting to off of course
 ) that would cause portage to check the /usr/portage ebuild against the recorded mtime and offer a remerge if the /usr/portage ebuild is newer
- have portage show up a distinct letter when -pv-ing (a la [ T ], similar to [ U ] [ N] etc) to signal that the /usr/portage ebuild is newer.
Any thoughts on this?
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
| Back to top |
|
 |
cold_flame
Tux's lil' helper
Joined: 23 Jun 2003
Posts: 88
Location: Australia
|
| Posted: Thu Jul 22, 2004 1:57 pm Post subject: |
|
|
i'm not too sure about how mtimes work, but wouldn't they reflect the last time the rsync mirror was synched? not when the package was added?
in which case you would always have ebuild with newer mtimes?
or are mtimes static when copied onto a new filesystem? |
|
| Back to top |
|
|
spb
Retired Dev
Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK
|
| Posted: Thu Jul 22, 2004 3:35 pm Post subject: |
|
|
| rsync will only modify the file if the server-side copy has changed, so if the ebuild hasn't been updated, the mtime on your local copy should stay constant. |
|
| Back to top |
|
|
robmoss
Retired Dev
Joined: 27 May 2003
Posts: 2634
Location: Jesus College, Oxford
|
| Posted: Thu Jul 22, 2004 3:47 pm Post subject: |
|
|
Sounds like a great idea to me! Of course, it would be nice if keyword changes unrelated to the current arch could be ignored (perhaps a remerge would be "warranted" by marking something stable though).
_________________
Reality is for those who can't face Science Fiction.
emerge -U will kill your Gentoo
ecatmur, Lord of Portage Bash Scripts |
|
| Back to top |
|
|
spb
Retired Dev
Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK
|
| Posted: Thu Jul 22, 2004 5:17 pm Post subject: |
|
|
| OK, apply this patch to emerge, then run 'emerge -p changes'. It's a bit hackish, in that it just ignores any package that causes it problems, but it seems to have the desired effect. It's been tested to the extent of touching the xterm ebuild and seeing it show up in the list, and on that count it works. |
|
| Back to top |
|
|
moocha
Watchman
Joined: 21 Oct 2003
Posts: 5722
|
| Posted: Thu Jul 22, 2004 8:30 pm Post subject: |
|
|
Whoa, that was fast! Gotta love Python. Thanks, thebell, testing.
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
| Back to top |
|
|
pkdawson
Retired Dev
Joined: 16 Jul 2004
Posts: 146
Location: Long Island or Buffalo, NY
|
| Posted: Thu Jul 22, 2004 8:38 pm Post subject: Re: [portage feature request] Ebuild mtime dependant remerge |
|
|
| moocha wrote: | | Seeing as ebuilds sometimes (ok, often ) get updated without bumping the version number of the package, I'd like to propose / request a portage feature that allows the user to specify whether to re-merge packages when the ebuild modification time changes into the future. |
It's not really necessary. Gentoo policy is that when ebuilds get updated without a revision bump, the change is usually small and only affects a minority of users. For example, there's absolutely no reason you'd want to recompile when all that was added is a compile fix for those who couldn't compile it in the first place.
See:
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=3&chap=1
If you're obsessive-compulsive about staying on the bleeding edge, well, okay  But it doesn't make a lot of sense to "upgrade" when you're not actually upgrading at all. |
|
| Back to top |
|
|
robmoss
Retired Dev
Joined: 27 May 2003
Posts: 2634
Location: Jesus College, Oxford
|
| Posted: Thu Jul 22, 2004 9:38 pm Post subject: |
|
|
In some cases, it is necessary. x11-base/xorg-x11-6.7.0, for example, has had more changes without a revision bump than I could reasonably expect to count - that ebuild tends to get updated every couple of hours at least. Now, this is great, because there's loads of patches that do all sorts of useful things, and I'm the sort of person who'd actually use them, but right now I'm having to use ecatmur's revcheck.sh script to check for differences in the ebuild.
The Gentoo policy exists for a reason - to make sure that people don't spend more time compiling than their computers can manage, unless they have a really unfeasibly slow machine. But if I've got an Athlon64 and it only takes half an hour to build X.org, why should I worry about recompiling it? Particularly when that's the biggest package I have (I think).
_________________
Reality is for those who can't face Science Fiction.
emerge -U will kill your Gentoo
ecatmur, Lord of Portage Bash Scripts |
|
| Back to top |
|
|
moocha
Watchman
Joined: 21 Oct 2003
Posts: 5722
|
| Posted: Thu Jul 22, 2004 9:47 pm Post subject: Re: [portage feature request] Ebuild mtime dependant remerge |
|
|
| pkdawson wrote: | | moocha wrote: | | Seeing as ebuilds sometimes (ok, often ) get updated without bumping the version number of the package, I'd like to propose / request a portage feature that allows the user to specify whether to re-merge packages when the ebuild modification time changes into the future. |
It's not really necessary. Gentoo policy is that when ebuilds get updated without a revision bump, the change is usually small and only affects a minority of users. For example, there's absolutely no reason you'd want to recompile when all that was added is a compile fix for those who couldn't compile it in the first place.
See:
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=3&chap=1
If you're obsessive-compulsive about staying on the bleeding edge, well, okay But it doesn't make a lot of sense to "upgrade" when you're not actually upgrading at all. |
Or when you really need to be on the bleeding edge because you're maintaining a package that depends one some other package whose ebuild has been silently updated.
Think before you type
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
| Back to top |
|
|
Lv
Retired Dev
Joined: 22 Jun 2003
Posts: 352
|
| Posted: Thu Jul 22, 2004 9:54 pm Post subject: Re: [portage feature request] Ebuild mtime dependant remerge |
|
|
| pkdawson wrote: |
It's not really necessary. Gentoo policy is that when ebuilds get updated without a revision bump, the change is usually small and only affects a minority of users. For example, there's absolutely no reason you'd want to recompile when all that was added is a compile fix for those who couldn't compile it in the first place. |
not necessarily true, major amd64 fixes have gone into ebuilds without bumping because arch maintainers arent generally allowed to bump ebuilds... especially since said bugs tend to only effect one arch.
this has bothered me so much i wrote a script to do revision checking... http://dev.gentoo.org/~lv/revcheck.sh
and then ecatmur updated it a bit to handle overlays: http://dev.gentoo.org/~lv/revcheck2.sh |
|
| Back to top |
|
|
Lv
Retired Dev
Joined: 22 Jun 2003
Posts: 352
|
| Posted: Thu Jul 22, 2004 10:36 pm Post subject: |
|
|
| thebell wrote: | | OK, apply this patch to emerge, then run 'emerge -p changes'. It's a bit hackish, in that it just ignores any package that causes it problems, but it seems to have the desired effect. It's been tested to the extent of touching the xterm ebuild and seeing it show up in the list, and on that count it works. |
doesnt work here.
| Code: | ayanami bin # emerge -p changes
These are the packages that I would merge, in order:
Calculating changes dependencies \Traceback (most recent call last):
File "/usr/bin/emerge", line 2677, in ?
if not mydepgraph.xcreate(myaction):
File "/usr/bin/emerge", line 1201, in xcreate
myeb=portage.portdb.xmatch("bestmatch-visible",mydep)
File "/usr/lib/portage/pym/portage.py", line 5338, in xmatch
mydep=dep_expand(origdep,self)
File "/usr/lib/portage/pym/portage.py", line 3689, in dep_expand
return prefix+cpv_expand(mydep,mydb,use_cache=use_cache)+postfix
File "/usr/lib/portage/pym/portage.py", line 3633, in cpv_expand
raise ValueError, matches
ValueError: ['app-xemacs/ocaml', 'dev-lang/ocaml']
|
after changing mylines.append("="+my_pv) to mylines.append("="+my_p+"/"+my_pv) -
| Code: | ayanami bin # touch /portcvs/sys-libs/libstdc++-v3/libstdc++-v3-3.3.3-r1.ebuild ayanami bin # emerge -p changes
These are the packages that I would merge, in order:
Calculating changes dependencies ...done!
ayanami bin # touch /usr/portage/sys-libs/libstdc++-v3/libstdc++-v3-3.3.3-r1.ebuild
ayanami bin # emerge -p changes
These are the packages that I would merge, in order:
Calculating changes dependencies ...done!
|
|
|
| Back to top |
|
|
Lv
Retired Dev
Joined: 22 Jun 2003
Posts: 352
|
| Posted: Thu Jul 22, 2004 10:38 pm Post subject: |
|
|
i would also like to note that ferringb doesnt seem to mind the idea of including this patch into portage proper once it works right. so if you want to see this feature... get to work!
| Quote: | <ferringb> Lv: mm'kay (come back with a working patch)
<ferringb> hmm.
<ferringb> Lv: patch needs work, should be calling aux_get(cat/pkg-ver, ["_MTIME_"]) instead of pulling from the ebuild |
stuff, stuff...
| Quote: | <ferringb> yeah, it could be used to hunt for QA violations also
<ferringb> Lv: well, right now it's not catching adjustments in the tree (abusing touch); if that's ironed out (along w/ a few other tweaks) it's possible |
i would also suggest something that filters out changes we dont care about... like comment typo fixes and keyword changes. whitespace changes. etc. poke at http://dev.gentoo.org/~lv/revcheck2.sh for a general idea of what i mean. |
|
| Back to top |
|
|
spb
Retired Dev
Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK
|
| Posted: Thu Jul 22, 2004 11:17 pm Post subject: |
|
|
| OK, I'll get down to improving it tomorrow, with any luck. I did write it in about quarter of an hour with hardly any knowledge of portage internals, so you can't expect that much from it. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9522
Location: beyond the rim
|
| Posted: Fri Jul 23, 2004 6:54 am Post subject: |
|
|
IIRC Nick was against that functionality, see bug 39501  |
|
| Back to top |
|
|
spb
Retired Dev
Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK
|
| Posted: Fri Jul 23, 2004 9:54 am Post subject: |
|
|
OK, I've got a newer version up here that uses portdbapi.aux_get instead of crawling through the tree, but the vardbapi.aux_get function doesn't seem to do mtimes afaict, so it still pulls that mtime from the filesystem.
@Genone: AFAICT, Nick's objections were that (1) people would be rebuilding unnecessarily, and (2) that it would add too much complexity. This approach seems to get around them both, since it's seperate from a normal update, and is actually fairly simple to implement.
@Lv: were you copy+pasting the patch? I've found that Python really doesn't like that. I can't seem to reproduce that here (on portage-2.0.51_pre13 if it's of interest). |
|
| Back to top |
|
|
spb
Retired Dev
Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK
|
| Posted: Fri Jul 23, 2004 12:10 pm Post subject: |
|
|
| Nicer version here which checks ebuild contents rather than mtimes, doesn't manually walk the package database, and also (in theory) ignores comments and KEYWORDS changes. |
|
| Back to top |
|
|
Lv
Retired Dev
Joined: 22 Jun 2003
Posts: 352
|
| Posted: Sun Jul 25, 2004 1:40 am Post subject: |
|
|
| it also ignores possibly important changes since it only counts lines... and it may cause false positives from removing whitespace. at least that' what i get from just taking a quick peek at it. |
|
| Back to top |
|
|
spb
Retired Dev
Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK
|
| Posted: Mon Aug 09, 2004 11:03 am Post subject: |
|
|
Sorry for the delayed response-- I've been away for the last 2 weeks.
What it does in the latest version is to count the lines, and if they differ, add it to the list. If they have the same number of lines, then it goes through and actually compares them. If the ebuilds differ on a line that doesn't begin 'KEYWORDS=', it adds that package to the list. As for whitespace changes, portage.grabfile ought to handle that.
BTW: update here; I just removed some debug statements I'd missed earlier... |
|
| Back to top |
|
|
|
Portage & Programming
