zetagrid style gentoo compile sharing?!?

hook

an idea once emerged in my head:

would it be possible to share the compile time with other gentoo boxen? ...like seti and zetagrid do. i mean, that would make us really a community plus users with older machines would benefit from it

_________________
tea+free software+law=hook

(deep inside i'm still a tux's little helper)

bsolar · Bodhisattva Joined: 12 Jan 2003 Posts: 2764

That's intresting. A lot interesting...

If possible (I don't know exactly what would be involved to do such a thing) why sharing only for compilation? Let's share raw horse-processor-power to who needs it...

I think this makes sense also because I heard somewhere in the forum about openmosix and about how it would be inefficient because of the short time a compilation process lives.

hook · Posted: Sat Feb 01, 2003 11:34 pm Post subject:

hmmm ....a gentoo world-wide super-computer? ...tempting

_________________
tea+free software+law=hook

(deep inside i'm still a tux's little helper)

TheCoop · Posted: Sat Feb 01, 2003 11:49 pm Post subject:

are there any possible software available to do it? Where do we start?

If it is openmosix, itll be sensible to merge the openmosix patch into the main gentoo-sources...
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock

idl · Posted: Sun Feb 02, 2003 12:19 am Post subject:

lemme get this straight, say someone on a 1ghz wanted to compile KDE.. would this system then send one person the instruction to compile kdegames for him then send back the binarys?

TheCoop · Posted: Sun Feb 02, 2003 8:57 am Post subject:

I suspect a distccd-like arrangement, but adapted for the internet (perhaps using the zetagrid code, is that open-source?), ive hda a quick look at openmosix and that is built for use on networks, not on the internet...

It also requires the same kernel version on all machines (2.4.20) but for most ppl that shouldnt be a problem
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock

Lovechild · Posted: Sun Feb 02, 2003 9:51 am Post subject:

Security might be an issue, no ?

TheCoop · Posted: Sun Feb 02, 2003 11:04 am Post subject:

yes, that is one issue. I'm not sure how to get around it thou...
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock

Tantive · Retired Dev Joined: 14 May 2002 Posts: 72

I think openmosix is not better than any other solution for this task because:
- Openmosix needs low latency (which you have NOT over an internet connection)
- migrating processes needs huge amount of bandwidth (having bandwidths of 6 MBytes/s occurs quite often)

Any other solution that tries to distribute the work would have the same problems.
The only thing that might be possible is to set up a kind of binary packages build system where systems owned by gentoo user build binary packages and store them on a main server where other users could download and use them (in a automated way). Or maybe they could be distributed over a peer-to-peer network.

Why zetagrid and seti are running so good on the internet can be explained by:
- The task are highly parallelizable.
- The input (the workunit) is small compared to the computation time (and in case of zetagrid compared to the output).

Just my 2 cents...

TaT

bsolar · Bodhisattva Joined: 12 Jan 2003 Posts: 2764

Yes, with the average bandwidth in Internet today this works only for few specific tasks, but the idea is still interesting at least teorically and I don't know how much time will take to get an Internet bandwith so high to start using such technologies with other tasks.

Tantive · Retired Dev Joined: 14 May 2002 Posts: 72

I was just thinking about what compression of data could get out of openmosix as i think the processes are going 1:1 through the wire atm.

hook · Posted: Sun Feb 02, 2003 6:33 pm Post subject:

i thought of those problems even before i opened this thread ...i posted it anyway, because i was sure together we would find a solution

_________________
tea+free software+law=hook

(deep inside i'm still a tux's little helper)

Matje · Posted: Mon Feb 03, 2003 12:26 am Post subject:

If it were available I wouldn't do it anyway, I think, just for security reasons. It is very easy for someone to send you back compromised binaries...
_________________
Life is like a box of chocolates... Before you know it, it's empty...

iplayfast · Posted: Fri Feb 21, 2003 4:47 pm Post subject:

OK, here is a possible solution. (Not that I really know anything about it)

User Mode Linux made available as a pool of computers. If you want to use the pool, it's just another box on the network for your use. I think this solves a number of security issues doesn't it?

distcc across the pool, or emerge -u world using the pool to parallel compile things that have had their dependancies satisfied.

An to make it worthwhile, a group login so that trusted individuals can create their own pools. This way, if you want to compile kde with -03 athlon whatever, if the binaries have been made before in your pool, the result is downloaded.

I can see this useful for groups of friends who trust each other, or corporate groups. Imagine an office full of secrataries and clerks, and having access to a user mode linux pool on their boxen!

ksteddom · Posted: Mon Feb 24, 2003 4:26 am Post subject: ccache

ccache will help with some of this. I want my build settings to be slightly different than someone elses. However, most of the build will be the same. If the pool of computers is using both distcc and ccache then "possibly" I could use some of the build they did previously.

Also, could MD5 be used to check for compromised biniaries?

Just a thought.