| View previous topic :: View next topic |
| Author |
Message |
Sublime
n00b
Joined: 09 Apr 2002
Posts: 7
|
 Posted: Tue Apr 09, 2002 10:40 pm Post subject: Downloading and compiling at same time |
 |
|
Hey everyone,
I know some other build from scratch distros. That when you are compiling source code it is actually downloading the nex things.
Example
say if you are downloading gnome but don't have X it will download the first package of X then download the thing that is depended on the first thing. So when the first is done compiling, it can start the next oen right up. This would save time, this also helps for the people on modems (God help them  )
Just an idea
Sublime |
|
| Back to top |
|
 |
Roel
n00b
Joined: 09 Apr 2002
Posts: 31
Location: Helmond, The netherlands
|
| Posted: Tue Apr 09, 2002 10:46 pm Post subject: |
|
|
The gentoo people are verry bright, there is the option --fetchonly:
| Quote: |
--fetchonly, -f
Instead of doing any package building, just per-
form fetches for all packages (main package as well
as all dependencies.)
|
Its not realy at the same time, but it saves some time for the poor telephone users..
(You will have to run the same emerge line without -f after the fetching, iirc)
_________________
Woei. |
|
| Back to top |
|
|
Sublime
n00b
Joined: 09 Apr 2002
Posts: 7
|
| Posted: Tue Apr 09, 2002 11:16 pm Post subject: downloading and compiling |
|
|
yeah i know there is a --fetchonly thing, but you have to run emerge twice, would be nice if by a config option was avialbe --compile-download or something along those lines. Just a suggestion
Sublime |
|
| Back to top |
|
|
m3thos
n00b
Joined: 09 Apr 2002
Posts: 46
Location: Portugal
|
| Posted: Wed Apr 10, 2002 12:28 am Post subject: yup, totally parallel: compile | download code |
|
|
I also desire this for a long time.. why shouldn't portage start downloading the second package, just after finished downloading the first, and at the same time it's compiling the first?
* It's parallel
* It's a performance enhancement
* It's finer and more elegant strategy
* It's usefull for everyone!
Dificulties:
* It would be better to implement this thing with some kind of semaphore or a similar dependency system, and.. semaphores can be tricky!
But, this would put portage in a superior level, and with it, other parallel and independent procedures could be implemented.
_________________
Miguel Sousa Filipe
handle: m3thos
More human than human |
|
| Back to top |
|
|
fatty
n00b
Joined: 09 Apr 2002
Posts: 6
Location: Raleigh, NC
|
| Posted: Wed Apr 10, 2002 2:48 am Post subject: semaphores?? to complicated! |
|
|
I hate to namedrop, but I am switching over from Sorcerer GNU Linux (experimentally) and they have a system where all source files are downloaded WHILE others are compiling... the best part is that sorcery (their version of emerge) is written as a bash script.
There really shouldn't be any need for semaphores or similarly complicated systems... just download the bleedin' files in the background in order of which are needed first in the dependecy structure, then begin compiling/installing as soon as they are available and all previous dependencies have been installed. That's not so tough, is it?
_________________
I wouldn't recommend sex, drugs or insanity for everyone, but they've always worked for me.
-Dr. Hunter S. Thompson |
|
| Back to top |
|
|
Sublime
n00b
Joined: 09 Apr 2002
Posts: 7
|
| Posted: Wed Apr 10, 2002 5:12 am Post subject: Sorcerer |
|
|
| I know Sorcerer has the feature. My question is how can it be anymore complicated then it already is? i mean gentoo gets the dependencies and lists them already might as well take it a step further and make it start downloading all needed packages one by one. |
|
| Back to top |
|
|
fghellar
Bodhisattva
Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR
|
| Posted: Wed Apr 10, 2002 11:17 pm Post subject: downloading and compiling |
|
|
I think it could be done using FIFO (First In, First Out) structures.
I personally do not know (yet) the internals of the Portage system, but the way I see it, it does already implement the idea of a FIFO structure when merging a new package and its dependencies. It is basically a top-down list of things to do. Currently, this list, if one could see it, would look like this:
| Code: | download package1
compile package1
download package2
compile package2
download package3
compile package3
... |
Above, each entry would be written in the beggining of the process.
Now, I think it wouldn't be that hard to split that list into two separeted lists, one for downloading and one for compiling. They would look like this:
| Code: | download package1 compile package1
download package2 compile package2
download package3 compile package3
download package4 ...
... |
Above, each entry in the dowload list would be written in the beggining of the process, and each entry in the compile list would be written by the corresponding entry in the download list once it had finnished downloading that package. This would allow the downloading process to move ahead independently from the compiling process. |
|
| Back to top |
|
|
m3thos
n00b
Joined: 09 Apr 2002
Posts: 46
Location: Portugal
|
| Posted: Sat Apr 13, 2002 12:46 am Post subject: it can be simpler, but we can start of with something more.. |
|
|
..powerfull and flexible, more geared towards future enhancements.
| fatty wrote: | I hate to namedrop, but I am switching over from Sorcerer GNU Linux (experimentally) and they have a system where all source files are downloaded WHILE others are compiling... the best part is that sorcery (their version of emerge) is written as a bash script.
There really shouldn't be any need for semaphores or similarly complicated systems... just download the bleedin' files in the background in order of which are needed first in the dependecy structure, then begin compiling/installing as soon as they are available and all previous dependencies have been installed. That's not so tough, is it? |
Yes..you're right.. it can be made simpler, I was thinking this wrong.. something like: downloading dependencies while compiling a program that needed those dependencies.. and he would stop at a semafore for the dependant aplication to be installed and then would proceed...
but.. for instance.. imagine we allready downloaded a dependency and we're compiling it.. and also finished the download of the aplication, now.. if we start unpacking and runing "configure && make, make install" the aplication without the dependency finishes tit's install there will be trouble! Right?
So.. the aplication must wait for the dependency to finish it's install!..
but.. since we're talking streamlined compiles and streamlined downloads in parallel this won't happen. but.. if we want this in the future... it would be good to start thinking about this sincronization problems, implementing some solution and use it in this kind of paralelism too!
_________________
Miguel Sousa Filipe
handle: m3thos
More human than human |
|
| Back to top |
|
|
fghellar
Bodhisattva
Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR
|
| Posted: Sat Apr 13, 2002 1:30 am Post subject: downloading and compiling |
|
|
| m3thos wrote: | | if we want this in the future... it would be good to start thinking about this sincronization problems, implementing some solution and use it in this kind of paralelism too! |
This problem is solved if it's done the way I described above. |
|
| Back to top |
|
|
I18N
n00b
Joined: 09 Apr 2002
Posts: 25
Location: Auckland NZ
|
| Posted: Mon Apr 15, 2002 7:37 pm Post subject: |
|
|
I'm a modem user. when I firsst started using gentoo I thought this would have been fantastic... but wouldn't it result in massive fragmentation of the harddisk? for all I know the journalling filesystems may be intelligent enough to solve this, it's just a thought that occurred to me.
another thing that would be good:
if there was an install cd that came with a large selection of the distfiles needed for a complete system.
using emerge -f I grabbed all the distfiles at a friends house via dsl, then built the system at home. my distfiles folder is under 400mb for a complete system, and could easily fit on a cd
_________________
Rarr. |
|
| Back to top |
|
|
tvon
n00b
Joined: 10 Apr 2002
Posts: 46
Location: Baltimore MD, USA
|
| Posted: Tue Apr 16, 2002 4:27 pm Post subject: Portage2 |
|
|
This has been a highly requested feature of Portage that will not be in version 1. Portage2 is going to be a complete rewrite of the code and will allow for downloads while compiling.
Some newish Gentoo developer who came from Sorcerer posted a bug to bugs.gentoo.org about this, but for the life of me I cant remember his name or find the bug.
Granted, I'm not a dev and I have no idea how things like Bevin's departure are affecting Portage2 or when it might be beta-ready.
-T |
|
| Back to top |
|
|
carpaski
Retired Dev
Joined: 04 Aug 2002
Posts: 43
Location: Chicago, IL
|
| Posted: Sun Aug 04, 2002 1:47 pm Post subject: |
|
|
Talk of Portage2 is on the wrong track. But anyway...
I've got code that does concurrent downloads...
I need more testing before I can tell drobbins that it's done.
Help me test it, please?
https://bugs.gentoo.org/show_bug.cgi?id=1661
_________________
Nicholas Jones <carpaski@gentoo.org>
Gentoo Linux Developer |
|
| Back to top |
|
|
|
Gentoo Chat
