proposal: a more intelligent approach to kernel sources

Message

stimuli · Post by **stimuli** » Sat Jun 12, 2004 4:50 am

I use gentoo on PPC, but this applies to all architectures.

Fact is, of the entire 40MB or so of gzipped kernel sources, I use maybe 5-10% of the source code in there. Same is true of x86, PPC64, Alpha, MIPS, etc.

Other architectures have no use for the PPC code. As a PPC user, I have no use for the code for the other architectures. There is no point in me downloading it, as doing so is a waste of bandwidth. There is no point in me having it, as doing so wastes hundreds of MB's of disk space.

So here is my proposal: Break gentoo-sources into smaller subsets like:
gentoo-sources-ppc
gentoo-sources-arm
etc, so that we only d/l the required files for building our arch's kernel, skipping the vast majority of files in the kernel.

This would:
- relieve the generous mirror servers of flagrant bandwidth wasting ("ooh, ooh, 2.6.x is out! Time to d/l another 40MB.")
- greatly speed up un-gzipping of kernel source code
- the previous two points == speeding up "emerge kernel" operations
- save disk space ( uncompressed linux kernel is ~300MB)
- show mercy to those who are on dialup... can you imagine?

That 40MB is a lot of source code to download, heck, that represents about 30 d/ls of say, w3m. It places an unneccesary burden on those who are kind enough to be our mirrors. The fact is, for EVERY gentoo user out there, 90%+ of it is useless. Kernel developers can grab the complete source from kernel.org if they feel they need it.

Failing that, for god's sake, at least use rsync to update kernel sources, so that if 1% of files have changed, 1% is downloaded. The redundancy of our current method is killing me.

Just a thought, hoping to make the raddest distro even better. cheers,

rm

Elm0 · Post by **Elm0** » Sat Jun 12, 2004 10:21 am

The problem is with the average gentoo users insatiable appetite for about a million different flavours of kernel splitting them up might take a LOT of time and effort - effort better spent doing projects that can really make gentoo better.

At the moment it really is as simple for the kernel devs as using wget to grab the file straight off the kernel.org site and sticking it on ibiblio.org (which I think is the main gentoo mirror). It also becomes much easier for us Gentoo users to verify the authenticity of the kernel sources because we can go back to the md5/gpg sums of the original creator - kernel.org

I would also question whether a problem really exists on bandwidth. There are some absolutely HUGE mirrors serving gentoo files, including ibilio, mirror.ac.uk and various US academic institutions with GBs of bandwidth. Certainly my downloads have never dropped below the max download speed of my DSL line.

Disk space, you may have a point there, but with most people having a 60GB drive minimum nowadays, and geeks (gentoo users

) having silly amounts like 200-300GB I don't think this is a huge problem.

[UK]Superdude · Post by **[UK]Superdude** » Sun Jun 13, 2004 3:26 am

I was thinking about this type of thing when I was on dialup. Trying out new kernel sources can be a pain due to the high download sizes. I was wondering about having a CVS type of thing with the entire kernel tree being unpacked on a server, and having some kind of tool that you use to configure the kernel. It then connects to the server and downloads the bits you need to configure and build a kernel with whatever you have selected. The advantage of this would also be updating you kernel would only fetch whatever has been modified in the new release.
Still is only an idea, I'd be interested to hear what others though tho. Is it even remotely possible?

stimuli · Post by **stimuli** » Sun Jun 13, 2004 3:42 am

Doing things my proposed way would be as simple as writing a bash script that untars the kernel source, then tar-gzips the first subdirectories within the main one. Depending on the arch, the 'emerge linux-kernel' would fetch all common directories (Documentation, etc) and one architecture-specific directory. That information (user arch.) is already accessible to portage.
The script could be a cron-job that checks kernel.org for newer kernels once per day, and if it finds them, executes as above.

I've never really found downloading huge source code tarballs to be slow (I'm on DSL too), with the odd exception here or there, I'm just amazed at the inefficiency of our current setup. VERY few files change from, say, 2.6.4-2.6.5, and to download so much excess every time seems so... wasteful? Maybe it is just me, but I think this proposal would be easy and efficient.

Another alternative would be to have a linux-kernel-latest ebuild that rsyncs the latest code from somewhere, although to be honest, rsyncing the latest MIPS and ARM and x86 and Sparc, etc, code on a PPC box is a minor waste of bandwidth and (remote server) processing power as well.

placeholder · Post by **placeholder** » Sun Jun 13, 2004 5:46 am

This all sounds a lot more like a request than a proposal... is using connotation. lol

gigel · Post by **gigel** » Sun Jun 13, 2004 6:16 am

why not download only the patches from kernel.org??
instead of more than 30megs why not just download 1-2 megs?
i suggest the ebuilds should only fetch the patches(for the vanilla kernel)
for other kernels where the patching is more complex it's good the way it is
now,my current /usr/src/linux is about 270 megs
and /usr/src/linux/arch is 52...
and /usr/src/linux/arch/i386 is 15..
i guess we can live with that...

slarti` · Post by **slarti`** » Sun Jun 13, 2004 9:48 am

mortix wrote:why not download only the patches from kernel.org??
instead of more than 30megs why not just download 1-2 megs?

This is exactly what kernel-2.eclass does.

Essentially, you are only downloading one full source tarball for each major kernel version and patching up the tarball you have inside its work directory each time a new version is available. The various flavours of patches are then downloaded and patched on the kernel.org source.

My only gripe is that old (like, ages old) source trees aren't purged. This can really use up a lot of disk space whilst doing no particular good. However, I can't really think of a good way to do this because some people may like to keep older sources around. Still. If there are files in your kernel tree that you aren't using in the slightest, delete them.

The old argument with the Portage tree comes into it too. Although kernel ebuilds are usually very small (because they inherit kernel-2.eclass and aren't left with much else to do), having a seperate ebuild for each architecture is going to slow down rsync a bit, which is expensive on resources on the client and server. Also, rsync resource usage is highest when traversing directories and calculating and differentiating contents - the transfer is the easy bit... and if there is not just gentoo-sources-arch but anyother-sources-arch, that'll waste time.

Sastraxi · Post by **Sastraxi** » Mon Jun 14, 2004 12:05 am

I don't think any of this can be done "cleanly" without portage having some kind of change in order to make package-patching more acceptable. Perhaps a new part of the ebuild that does different things, you check the version you're upgrading from and if it matches the possible upgrade-from versions then it will execute whatever commands needed... exactly the same way an ebuild is normally done. After that all of the files updated are added to the original package's "file-list" and the version is upgraded (after all, if everything was upgraded correctly, it would be *exactly* the same as a base install of the more recent version).

I know it's a rant, but I think it could be done.

stimuli · Post by **stimuli** » Mon Jun 14, 2004 8:31 am

I'm down with mortix's mentioning of patches... I'm a dolt for not thinking of that! It is obvious, simple, straightforward. It's available from kernel.org, I'm sure it has checksums (doesn't it?) and it would save crazy amounts of bandwidth.