| View previous topic :: View next topic |
| Author |
Message |
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
 Posted: Fri Jan 09, 2004 1:47 am Post subject: Portage drawback |
 |
|
Rules of the house:
1. Please don't flame me...
2. Please read carefully.
I think kernel patching in Gentoo/Portage sucks. Consider this scenario:
I emerge development-sources-2.6.0 and have my 2.6 system up and running. I apply some patches to get acpi, bootsplash etc. working. Along come some security threats and next moment you know people have a fix ready. Fixed kernel is available as development-sources-2.6.0_rc1 (or something like that). Now, if I emerge the new sources, it emerges as a whole tar and creates directory /usr/src/...2.6.0_rc1/. Now I have to apply all patches on top of this one and re-compile my kernel with .config from /usr/src/...2.6.0/.
Few problems: 1.) I might have forgotten about some patches. 2.) I have to maintain two source trees for a while. 3.) Tomorrow _rc2 comes along I have to do the same thing again.
Other option of finding the patches on internet and applying the patches manually is a security threat in itself and I don't think anyone Gentoo will encourage it. And mind you, those patches have to be against development-sources-2.6.0.
Its a "source only" package. Why not release these as patches on top of base branch, instead of whole tar and provide "--patch" option to emerge for such packages to just update the same base kernel source tree?
No, I am not a bleeding edge guy. Just that security fixes are important.
I just want to add that I LOVE PORTAGE, so much so that I took time to port it to fedora on another system that I have. But nothing has ever stopped a man from asking for more, now, has it?...  |
|
| Back to top |
|
 |
adaptr
Watchman
Joined: 06 Oct 2002
Posts: 6730
Location: Rotterdam, Netherlands
|
| Posted: Fri Jan 09, 2004 2:04 am Post subject: |
|
|
'S not your house - it's a fun-fair!
But I agree completely - for a true "from-source" distribution, EVERY ebuild should follow the same policy:
1. pure source tree
-> + 2. gentoo patches
-----> + 3. my patches
This goes double for whoppers like the kernel, where I have to download a big fat 30 MB for changes that amount to less than 0.001% of the source code!
That's idiotic. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9523
Location: beyond the rim
|
| Posted: Fri Jan 09, 2004 3:15 am Post subject: |
|
|
| check GLEP 9 / app-portage/deltup, I don't know what's the current status of it but you might want to ask teh author about it. The major problem is that portage would have to know that tarball-1.0 + patch-to-1.1 is the same as tarball-1.1, that's what deltup was written for. (As I said, I don't know the status so don't ask me about it) |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Jan 09, 2004 6:11 am Post subject: |
|
|
deltup is not updated very much these days...and I think tarball diff is a bad idea anyway.
what is wrong with generating just a plain text patch file of the changes from a base version and then zipping it up? Changes in patchset 2 include changes in patchset 1 on top of the base system. adaptr has put in a 3 level good description of it.
This diff would be so very easy to create if something like cvs is used to manage these sources.
Can somebody at least look at doing this for kernel sources because in this case the package is not even compiled by emerge?
What is the standard way other people are updating their Gentoo systems with security fixes? |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9523
Location: beyond the rim
|
| Posted: Fri Jan 09, 2004 5:07 pm Post subject: |
|
|
well, code a working solution and we can talk about it.
Problems involved are mirror space, md5 checks and equivalence checking of different patch paths. |
|
| Back to top |
|
|
stevena
n00b
Joined: 17 Apr 2003
Posts: 50
|
| Posted: Fri Jan 09, 2004 5:22 pm Post subject: |
|
|
Can't we do this in portage by splitting a current kernel ebuild in a kernel-source-ebuild and a patch-ebuild. And let them depend on each other?
Patching the kernel will be done in another ebuild.
Or is this to simplified? |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9523
Location: beyond the rim
|
| Posted: Fri Jan 09, 2004 5:38 pm Post subject: |
|
|
| ask the kernel people about it (= file a enhancement bug and assign it to x86-kernel@gentoo.org) |
|
| Back to top |
|
|
adaptr
Watchman
Joined: 06 Oct 2002
Posts: 6730
Location: Rotterdam, Netherlands
|
| Posted: Sat Jan 10, 2004 6:00 pm Post subject: |
|
|
| Genone wrote: | | Problems involved are mirror space |
You're kidding, right?
How could maintaining patches against a pure kernel.org kernel EVER take more space than maintaining separate kernel source trees for a DOZEN different -sources? |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9523
Location: beyond the rim
|
| Posted: Sat Jan 10, 2004 7:09 pm Post subject: |
|
|
because
a) all those -sources packages use the vanilla tarball and apply patches to add functionality, so that is not an argument
b) the old way still has to be supported, so we can't just drop old tarballs from the mirrors |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Thu Jan 15, 2004 9:48 pm Post subject: |
|
|
| devsk wrote: | | deltup is not updated very much these days...and I think tarball diff is a bad idea anyway. |
Guessing you mean a gnudiff patch generated via diff?
| devsk wrote: | | what is wrong with generating just a plain text patch file of the changes from a base version and then zipping it up? |
Depends on what type of patch it is; unified is utterly wasteful due to including the deleted line; added it lacks a command to state 'move this line 4 lines forward'. Further, a one character modification in a line results in the entire line being included in the patch, rather then just that one character.
Try one of the binary differencers like xdelta or diffball and difference a few srcs out of distfiles- xdelta will usually beat out diff by a good margin, and diffball will 95% of the time beat the crap out of xdelta (note xdelta's patches are generated compressed- run xdelta w/ -0 to disable compression and use either bzip2/gzip for comparison).
Also, plain text won't work since to apply it, you'd have to pull the tarball apart, apply patch, then reconstruct the file. This breaks the md5 in a way that is not easy to fix. A thread detailing the problems w/ using plaintext is here.
| devsk wrote: | | Can somebody at least look at doing this for kernel sources because in this case the package is not even compiled by emerge? |
If you're using portage to pull the src, compilation isn't the issue- md5 perfect reconstruction is the problem (detailing in the link above, here, and here). Reconstructing the target perfectly isn't hard, but dealing w/ the fact recompressing it may result in a different file is... it's pretty much the main problem I've been working on the last few weeks...
| devsk wrote: | | What is the standard way other people are updating their Gentoo systems with security fixes? |
Genone's baby, glep 14 would be worth watching then |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Thu Jan 15, 2004 10:59 pm Post subject: |
|
|
I was talking about simpler stuff than people are talking about in those posts. For a package like gentoo-dev-sources and going from 2.6.0 to 2.6.1, why can't they keep just the patches tar (genpatches tar keeps tracks of patches for gentoo-dev-sources on top of development-sources, just keep adding one to that tar). ebuild for this patch checks if gentoo-dev-sources-2.6.0 is installed, it then just epatch'es all the patches in that tar, so existing sources get overwritten with the changes that were part of 2.6.0 -> 2.6.1 bringing my current source tree to latest. kernel.org releases patches on top of development-sources.
If acpi patch (plus others) can be applied on top of development-sources-2.6.0 and it can be called gentoo-dev-sources-2.6.0, why not add another patch in that tar and make it gentoo-development-sources-2.6.1. It can reduce time to download for security fixes, reduce time for to configure and build the kernel again, save space for me by overwriting existing source tree.
For all other packages in portage, this doesn't apply because source anyway is in a tar and your "diff" or "patch" based system has to be complex. |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Fri Jan 16, 2004 12:09 am Post subject: |
|
|
| devsk wrote: | | I was talking about simpler stuff than people are talking about in those posts. For a package like gentoo-dev-sources and going from 2.6.0 to 2.6.1, why can't they keep just the patches tar (genpatches tar keeps tracks of patches for gentoo-dev-sources on top of development-sources, just keep adding one to that tar). ebuild for this patch checks if gentoo-dev-sources-2.6.0 is installed, it then just epatch'es all the patches in that tar, so existing sources get overwritten with the changes that were part of 2.6.0 -> 2.6.1 bringing my current source tree to latest. kernel.org releases patches on top of development-sources. |
What you're proposing it a bit of a pita; to do it _correctly_, you need to determine what reference/base version exists, from there, verify the file is valid (md5 check). Determining a base and patches needed isn't too hard, I've already written out code that does that for my work w/ glep9. So, say 2.6.0 was determined as the base; for _strictly_ packages using genpatches, specifically using a common variable UNIPATCH_(DOC|LIST) we'll say, you'd have to either make the ebuild able to determine the base, and patch from there, or stick that functionality in an eclass.
That sounds nice and fuzzy, but has some rather annoying problems- untar the 2.6.0 kernel, it drops it's source in linux-2.6.0; now you have to make the ebuild use the old work dir (linux-2.6.0 rather then linux-2.6.1). That's extra cruft that needs to be jammed into an eclass/ebuilds.
Hokay lets just say that the issues above and a couple others I've left out for brevity have been dealt with- now there is the annoyance of having to repatch the kernel every single time w/ the 2.6.1 patch; minor, but such a setup gets old _quick_ when you get upto 2.6.5 and you have to apply the entire .1 -> .2 -> .3 -> .4 -> .5 set of patches.
To head off the 'why not just use a patch that goes from .1 directly to .5', jamming the logic to handle that into an eclass/ebuild is utterly problematic. To enable such behaviour, you need shortest path determination, which is not something you want to try in bash.
| devsk wrote: | | It can reduce time to download for security fixes, reduce time for to configure and build the kernel again |
Time to download for security fixes? It takes time for a security fix to be tested, then pushed into the tree; from there, there is the time involved in syncing, and building- you're worried about downloading? For a security fix, most of the time it's an additional patch, not the full src. Please elaborate- I might be missing what you're getting at w/ that statement.
| devsk wrote: | | For all other packages in portage, this doesn't apply because source anyway is in a tar and your "diff" or "patch" based system has to be complex. |
How does it not apply? The kernel is distributed in tar; the extra patches apply for the various kernel versions are non-vanilla tweaks, not version bumps. Fundamentally, patching the kernel src tarball is no different then patching any other src tarball.
Unless I'm missing something what you're getting at there.
Either way, basically either a lot of rather nasty work must be done for one particular set of ebuilds, or glep9 gets finished.
The general route imo is the better way; added the patch size to be pulled down is typically much smaller then pulling the diff patch (mentioned above in the binary differencer vs gnudiff discussions in threads listed above). |
|
| Back to top |
|
|
adaptr
Watchman
Joined: 06 Oct 2002
Posts: 6730
Location: Rotterdam, Netherlands
|
| Posted: Fri Jan 16, 2004 1:08 am Post subject: |
|
|
| Genone wrote: | because
a) all those -sources packages use the vanilla tarball and apply patches to add functionality, |
Heh - didn't catch that one  |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Jan 16, 2004 1:45 am Post subject: |
|
|
| ferringb wrote: |
| devsk wrote: | | It can reduce time to download for security fixes, reduce time for to configure and build the kernel again |
Time to download for security fixes? It takes time for a security fix to be tested, then pushed into the tree; from there, there is the time involved in syncing, and building- you're worried about downloading? For a security fix, most of the time it's an additional patch, not the full src. Please elaborate- I might be missing what you're getting at w/ that statement. |
This is what I have to do to go from 2.6.0 to 2.6.1:
1. emerge 2.6.1. downloads 30MB and untars into another 160MB dir. There are two source dirs now, one with 2.6.0 and another with 2.6.1.
2. update link /usr/src/linux.
3. copy .config and make oldconfig, make menuconfig...blah
4. 2.6.1 sits there untarred. I can remove 2.6.0 by unmerging it.
If I am the kind of guy who is installing his own kernel, I would probably leave these sources around even after the kernel has been built and is running properly. When 2.6.0 was sitting there untarred, why couldn't the ebuild just apply patch on top of it?
That's why I said it applied perfectly to the kernel sources because they just sit there untarred most of time.
Think of it like this: Going from gentoo-dev-sources-2.6.0 to gentoo-dev-sources-2.6.1 is same(apart from the content of the diffs) as going from development-sources-2.6.0 to gentoo-dev-sources-2.6.0. If the latter can be done with a package called genpatches, why not the former by just adding one more patch to genpatches and making an ebuild out of genpatches? gentoo-dev-sources goes to a predefined well known location, genpatches ebuild knows that.
multiple patchsets can be released as multiple genpatches ebuilds.
you are saying that extracting 2.6.0 and 2.6.5 from the installed and available version numbers and determining which genpatches ebuild should be used is difficult in bash? if genpatches-1.0.0 applies to 2.6.0, then genpatches-1.0.5 applies to 2.6.5 i.e. the patches are cumulative and always apply on top of base version.
Of course, all this can't be done externally w/o the help of the portage guys and the kernel source package maintainers. Somebody needs to generate those patch ebuilds(with all its diff problems with binary files, zero size files etc...) and portage needs to have metadata to know which(name) patch ebuild to refer for which package e.g. it needs to know to use genpatches ebuild for gentoo-dev-sources package. version, of course, it can determine easily(another diffing...).
PS: after writing all this, I felt it was too much work to do for saving some space and download times...so, let's be happy and cherish the beauty the portage is, as it is... |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Tue Jan 20, 2004 10:44 pm Post subject: |
|
|
| devsk wrote: | | If I am the kind of guy who is installing his own kernel, I would probably leave these sources around even after the kernel has been built and is running properly. When 2.6.0 was sitting there untarred, why couldn't the ebuild just apply patch on top of it? |
It all depends on what patches have been applied; if it's a straight vanilla src, then yes, applying a version patch isn't an issue; if it's not straight vanilla, you'd have to back out the extra patches first (which can be tricky), apply the version patch, then apply the new genpatches.
Applying the appropriate patch on a livefs I'd be inclined to leave upto users who manage the kernel src themselves (myself for instance); I'm working on pulling down just a patch for the actual src, but what you're talking about is beyond what I'm working towards.
Feel free to give it a shot of course, just strikes me as it's going to be rather hard. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
