| View previous topic :: View next topic |
| Author |
Message |
Brian Ågren
n00b
Joined: 11 Jun 2002
Posts: 11
Location: Odense, Denmark
|
 Posted: Sat Jul 06, 2002 9:11 am Post subject: Speed up installation |
 |
|
to speed up an clean installation:
start downloading the packages (in a different shell) by typing emerge -f system ... and when the first download is done start emerging.
that way the download will be done parallel to the compilation
 BUT .. if you connection is slow or your computer is too fast, the emerge will crash if the download process is downloading the package currently being emerged.
i'm quite new at gentoo so let me know if this is too stupid  |
|
| Back to top |
|
 |
alec
Apprentice
Joined: 19 Apr 2002
Posts: 270
Location: Here
|
| Posted: Sat Jul 06, 2002 6:58 pm Post subject: |
|
|
It isn't too stupid at all. I do it. Just give yourself some lead time - the first few packages compile quite quickly.
It won't speed up things that much, though... |
|
| Back to top |
|
|
delta407
Bodhisattva
Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL
|
| Posted: Sat Jul 06, 2002 8:38 pm Post subject: |
|
|
If you're on a modem, it will. 
This is especially true for huge packages (like KDE); just start an emerge -f kde, give it a little bit of a head start, and emerge kde.
_________________
I don't believe in witty sigs. |
|
| Back to top |
|
|
alec
Apprentice
Joined: 19 Apr 2002
Posts: 270
Location: Here
|
| Posted: Sat Jul 06, 2002 9:16 pm Post subject: |
|
|
| /me is spoiled by broadband |
|
| Back to top |
|
|
freshy98
Apprentice
Joined: 11 Jul 2002
Posts: 274
Location: The Netherlands
|
| Posted: Thu Jul 11, 2002 2:12 pm Post subject: |
|
|
some questions...
- stupid question perhaps, but is the "-f" the same as "--fetchonly" ?
- when the fetch is done, where does it put it? seems to me in the standard download directory for compiling, but then again how do I start the compiling? how do I start the compiling? what parameters should I use?
- Brian: with "system" do you mean the a program to be installed like KDE or something else?
this all is very nice since I'm only home in the weekends, most weekends anyway, I could then compile it all back home where I have no internet.
gonna install the Gentoo base system tommorow.
thanks in advance!
PS: yes, I am a Linux newbie but willing to learn. Gentoo already really helps me to understand Linux.
_________________
Mac Pro single quad 2.8GHz, 6GB RAM, 8800GT. MacBook. Plus way too many SUN/Cobatl/SGI and a lonely Alpha. |
|
| Back to top |
|
|
tomas
n00b
Joined: 05 Jul 2002
Posts: 62
Location: Sweden
|
| Posted: Thu Jul 11, 2002 2:26 pm Post subject: |
|
|
freshy98: Yes -f is the same as --fetchonly. You can use man command to get information on a command. For example:
|
|
| Back to top |
|
|
alec
Apprentice
Joined: 19 Apr 2002
Posts: 270
Location: Here
|
| Posted: Thu Jul 11, 2002 3:34 pm Post subject: |
|
|
| freshy98 wrote: | | when the fetch is done, where does it put it? seems to me in the standard download directory for compiling, but then again how do I start the compiling? how do I start the compiling? what parameters should I use? |
'emerge -f package' puts package in /usr/portage/distfiles. To start compiling, just 'emerge package' - portage looks in /usr/portage/distfiles for the files it will use to compile, and if they are not there, will go and fetch them. However, since they're there, it will skip fetching them and go straight to compiling. Use whatever parameters you'd normally use when emerging. |
|
| Back to top |
|
|
freshy98
Apprentice
Joined: 11 Jul 2002
Posts: 274
Location: The Netherlands
|
| Posted: Fri Jul 12, 2002 8:17 am Post subject: |
|
|
ah great! thanks a lot. gonna use that one after I set my base system.
gonna crunch on KDE and other stuff next week when I'm back home without internet.
is it also possible to get source files from the internet and copy them into the packages directory using a CDROM?
that way I can download those files at my work and put them on my machine at home.
or am I asking to much now?
_________________
Mac Pro single quad 2.8GHz, 6GB RAM, 8800GT. MacBook. Plus way too many SUN/Cobatl/SGI and a lonely Alpha. |
|
| Back to top |
|
|
tomas
n00b
Joined: 05 Jul 2002
Posts: 62
Location: Sweden
|
| Posted: Fri Jul 12, 2002 8:24 am Post subject: |
|
|
| Yes |
|
| Back to top |
|
|
carpaski
Retired Dev
Joined: 04 Aug 2002
Posts: 43
Location: Chicago, IL
|
| Posted: Sun Aug 04, 2002 1:25 pm Post subject: |
|
|
'emerge -f' running at the same time as 'emerge' is a bad idea. You are likely to cause problems in the database, compile directory, or distfiles.
If you want to test the proper way to do this then test this for me:
https://bugs.gentoo.org/show_bug.cgi?id=1661
_________________
Nicholas Jones <carpaski@gentoo.org>
Gentoo Linux Developer |
|
| Back to top |
|
|
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
| Posted: Tue Aug 06, 2002 10:52 am Post subject: Re: Speed up installation |
|
|
| Brian Ågren wrote: | to speed up an clean installation:
start downloading the packages (in a different shell) by typing emerge -f system ... and when the first download is done start emerging.
that way the download will be done parallel to the compilation
BUT .. if you connection is slow or your computer is too fast, the emerge will crash if the download process is downloading the package currently being emerged.
i'm quite new at gentoo so let me know if this is too stupid |
Damn, it is a VERY GOOD idea, already tried your idea and works like a champ
_________________
All for one and one for All
--
MACPRO machine... |
|
| Back to top |
|
|
AlterEgo
Veteran
Joined: 25 Apr 2002
Posts: 1619
|
| Posted: Wed Aug 07, 2002 8:34 pm Post subject: |
|
|
| carpaski wrote: | 'emerge -f' running at the same time as 'emerge' is a bad idea. You are likely to cause problems in the database, compile directory, or distfiles.
If you want to test the proper way to do this then test this for me:
https://bugs.gentoo.org/show_bug.cgi?id=1661 |
1) couldn't agree more. You will break dependencies (I've tried).
2) Will try on narrowband-internet  connection next week.
Interesting stuff ! |
|
| Back to top |
|
|
neeyo
n00b
Joined: 20 Jul 2002
Posts: 32
|
| Posted: Fri Aug 16, 2002 11:49 pm Post subject: |
|
|
Good idea, bad application.
If your program has 2 dependencies that are small, and your final program is huge (like mozilla), then you will screw it up bigtime. You will download the 2 dep's and begin to compile them, while getting mozilla in the background. Unfortunately, it will take you a while to get mozilla, and only about 2 minutes to compile the 2 deps.
Yu0 = scr3w3d. |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Fri Nov 15, 2002 10:43 am Post subject: |
|
|
| neeyo wrote: | If your program has 2 dependencies that are small, and your final program is huge (like mozilla), then you will screw it up bigtime. You will download the 2 dep's and begin to compile them, while getting mozilla in the background. Unfortunately, it will take you a while to get mozilla, and only about 2 minutes to compile the 2 deps.
Yu0 = scr3w3d. |
Well, not fatally. You just wait until the download completes and run emerge mozilla again... Unless the "compile" emerge tries to download the file on top of/simultaneously with the "download" emerge... Either way it will never try to install because the md5's wouldn't match. I can't see how you could POSSIBLY damage the dbs because -f just downloads files... it doesn't write to the database... |
|
| Back to top |
|
|
vaevictus
n00b
Joined: 24 Feb 2003
Posts: 20
|
| Posted: Mon Feb 24, 2003 9:23 pm Post subject: error code checking and return codes and all sorts |
|
|
It would seem to me, that i'd be feasible to write a wrapper around the emerge, to break it out in chunks, and to multithread the compiling and downloading, without much difficulty or danger (but check me on the danger part),
basically...
emerge -p <thing> >> thingoutput
foreach subthing in thingoutput {
emerge --fetchonly subthing &
}
foreach subthing in thingoutput {
while ( ! emerge --fetchonly subthing )
{
sleep 5
}
emerge subthing
}
this way, it starts a download of all dependencies initially, then waits for each one to finish before it starts to compile. Unfortunately, I can find a way to get around the while (! emerge --fetchonly subthing) part starting up another instance of fetch (colliding with the initial one).
with a simple emerge option: --has_been_fetched, or something, this would become clean and usable pseudocode.
Thoughts?
n8 |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Tue Feb 25, 2003 1:37 am Post subject: Re: error code checking and return codes and all sorts |
|
|
| vaevictus wrote: | It would seem to me, that i'd be feasible to write a wrapper around the emerge, to break it out in chunks, and to multithread the compiling and downloading, without much difficulty or danger (but check me on the danger part),
basically... |
See below...
Of course I see a few problems with the entire thing... there is a lot of overhead in running emerge all the time (some smart BASH scripting and caching of results should be used extensively).
Also, the first for foreach loop... starting ALL the downloads at once on a 56k dial-up connection will assuredly guarantee that none of the files will be downloaded quickly... or at all... I think any solution should work for both high and low bandwidth Gentoo users, even if us low speed  users are in the minority.
I'll give this some thought...
(BASH/psuedocode)
| Code: | emerge -p <thing> >> thingoutput
check_files()
{
result=0
#this will return the same file a bunch so uniq should be used somewhere
# although the overhead of checking for existence shouldn't be bad
for file in `emerge -fp $1 | grep -v Calculating`; do
file=`basename $file`;
if [ ! -e $file ]; then
result=1
fi
done;
return $result
}
foreach subthing in thingoutput {
emerge --fetchonly subthing &
}
foreach subthing in thingoutput {
while ( ! check_files subthing )
{
sleep 30s #longer fo average modem user
}
emerge subthing
}
|
|
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Tue Feb 25, 2003 2:16 am Post subject: |
|
|
Warning, this is BASH/pseudo code... you can't copy and paste and run it
It would need a "working" directory somewhere to keep track of state... but it looks VERY workable to me... what does everyone think?
Actually I don't think it would take much work to convert into REAL bash... does anyone see any show stopped before I would write it up?
| Code: |
emerge thing -p >> all_things
# subroutine
download_files()
{
for subthing in all_things do
emerge -f subthing
touch subthing.complete
done
}
# subroutine
have_all_files()
{
check for existence of subthing($1).complete
return success if found, failure if not
}
# start the downloads (one at a time)
download_files &
for subthing in all_things do
while not have_all_files subthing
{
sleep 5s;
}
emerge subthing
done
|
|
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Tue Feb 25, 2003 3:08 am Post subject: |
|
|
OK... I have REAL code, and it seems to work just fine. But first, how would everyone like to handle the simultanous output of both the downloads and the emerges?
If you just mixed them all it would be one pretty big mess...
Any ideas? Give me some thoughts, meanwhile I'll clean up my code and post it. |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
|
| Back to top |
|
|
|
Portage & Programming
