View previous topic :: View next topic |
Author |
Message |
bobby Guest
|
Posted: Sat May 18, 2002 7:50 pm Post subject: Can I make emerge -s go faster? |
|
|
Is there a way to make emerge -s go faster? I want to search for things, but I hate to wait.
If not, ideas on strategies for making it faster? A cache, perhaps?
Shanks,
Bobby
quicker faster better |
|
Back to top |
|
|
rommel Veteran
Joined: 19 Apr 2002 Posts: 1145 Location: Williamsburg Virginia
|
Posted: Sat May 18, 2002 7:56 pm Post subject: |
|
|
more ram faster cpu ....raid?
i just did it after reading your thread and it took like a second to see the version of portage that i had and the latest out....so post something more specific about your setup...internet connection cpu ddr or sdram...how much of it...cpu speed...blah blah blanh
ciao |
|
Back to top |
|
|
Guest
|
Posted: Sun May 19, 2002 1:18 am Post subject: |
|
|
honestly I hate waiting too ... my system is a 333 K6-2 with 128 megs of ram
it takes long enough for the blasted emege --pretend statments and what
not. a cache tied into the rsync command would be nice ... something ala
locate and updatedb
emege --cache --clean rsync
or something like that ... so that you can use it if you want
but arn't forced to |
|
Back to top |
|
|
BradN Advocate
Joined: 19 Apr 2002 Posts: 2391 Location: Wisconsin (USA)
|
Posted: Sun May 19, 2002 3:06 am Post subject: |
|
|
I'm on a P3 515mhz with 448 megs of ram... emerge takes a long time to load it's package list (or whatever it may be) - about 10 seconds or so roughly, but only takes a couple seconds the next time you run it because it's all cached in the filesystem. A cache would probably help a lot here. |
|
Back to top |
|
|
coshaw n00b
Joined: 12 May 2002 Posts: 7
|
Posted: Sun May 19, 2002 3:49 am Post subject: |
|
|
I am not sure, but it seems if you use XFS, emerge -s is faster, especially compared with ext3. I didnt try reiserfs. |
|
Back to top |
|
|
WhyteWolf Apprentice
Joined: 01 May 2002 Posts: 168 Location: Las Vegas
|
Posted: Sun May 19, 2002 4:27 am Post subject: |
|
|
with a proper cacheing system it would be even faster ... sence it wouldn't
hafta read every file just the cache ...
dependency checking would take a matter of nanoseconds
and searching could be much cleaner |
|
Back to top |
|
|
TheWart Guru
Joined: 10 May 2002 Posts: 432 Location: Nashville,TN - USA
|
Posted: Sun May 19, 2002 4:48 am Post subject: |
|
|
IIRC, they are working on some sort of quicker cache thing fro portage 2.0.... _________________ Face it, we are all noobs.
On the box it said it was designed for Win XP or better, so why won't it work with Linux? |
|
Back to top |
|
|
AutoBot l33t
Joined: 22 Apr 2002 Posts: 968 Location: Usually Out
|
Posted: Sun May 19, 2002 4:54 am Post subject: |
|
|
coshaw wrote: | I am not sure, but it seems if you use XFS, emerge -s is faster, especially compared with ext3. I didnt try reiserfs. |
It doesn't seem to be faster on XFS compared to ext3 it is, when I ran gentoo on XFS emerge rsync and emerge -s <foopackage> ran almost instantanious. But now on ext3 I get major lag when running either, enough to *almost* warrant a back up and a change over to XFS again but not until the preemp gets worked out properly.
My Gentoo Box For Reference:
10gig 5200rpm HD --- hdparm Cached: 94.38 MB/sec Buffered: 25.45 MB/sec
40gig USB 2.0 Buslink HD
533 Celeron
192 Meg Ram
GeForce II MX 400
Cable: 300kbs Down / 50kbs Up _________________ This message self destructed a long time ago. |
|
Back to top |
|
|
sibn n00b
Joined: 15 May 2002 Posts: 36
|
Posted: Sun May 19, 2002 4:18 pm Post subject: |
|
|
rm -rf /usr/portage should do the trick. Of course, you should not do this because it will break portage (and you'll have to have some kind of backup plan for restoring the ebuild tree), and every search you do will come up dry. But each search will take less than a second.
You have been warned: Don't do this. _________________ Hey um, this is my signature. |
|
Back to top |
|
|
rommel Veteran
Joined: 19 Apr 2002 Posts: 1145 Location: Williamsburg Virginia
|
Posted: Sun May 19, 2002 7:37 pm Post subject: |
|
|
i did notice a big difference with lsr after i had gentoo the first time installed on just a normal partition then the seccond install i created a raid 0 array so the /mnt/gentoo dir was created adn mounted on /dev/md0 ....from that there was a big improvement on anything related to emerge....read transfers went from 38mb/s on a normal ibm 7200 60gxp to 72mb/s with using two of these drives and the lsr.
try it you'll like it
ciao |
|
Back to top |
|
|
klieber Bodhisattva
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Sun May 19, 2002 9:42 pm Post subject: |
|
|
TheWart wrote: | IIRC, they are working on some sort of quicker cache thing fro portage 2.0.... |
Well, they're working on a real database for a portage backend, as opposed to trying to parse through a bunch of text files. This will make things like 'emerge -s' much faster.
You can learn a bit more about portage 2.0 here, though be warned that at least some of that info is out of date. (for instance, last I heard, portage 2.0 was still going to be python-based, not C++ based -- definitely a sore subject among some members of the gentoo dev team.) So, take what you read with a grain of salt.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
|
bobby Guest
|
Posted: Sun May 19, 2002 11:08 pm Post subject: |
|
|
Ok, thanks, all.
To sum up (so far), it sounds like hardware bottlenecks (memory) and faster (for this app) filesystems are the best options. The issue is mostly tied to the scanning of many file in the filesystem -- so disk and filesystem issues are the main optimizers for the file scanning stuff.
And, there's talk of software improvements in portage 2.0. |
|
Back to top |
|
|
Jyrinx Tux's lil' helper
Joined: 03 May 2002 Posts: 92 Location: Carleton College - Northfield, MN
|
Posted: Mon May 20, 2002 6:09 am Post subject: |
|
|
(off-topic) klieber: You sure they've switched back to Python? That would make me quite happy ... I'm still trying to figure out how they went from the practical, pragmatic Python to their ultra-anal, ideology-driven "let's use 90% of our brainpower making sure we're following the style guide by using GNU syntax for stack allocated variables and ours for others" style of C++ ... I knew I would never contribute if I had to do that.
Jyrinx
jyrinx@mindspring.com |
|
Back to top |
|
|
Target Apprentice
Joined: 25 Apr 2002 Posts: 200
|
Posted: Mon May 20, 2002 8:19 am Post subject: |
|
|
Requiring any kind of scripting engine also makes it impossible to shoehorn Gentoo into some applications, which may be the reason for all the hubaloo. |
|
Back to top |
|
|
klieber Bodhisattva
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Mon May 20, 2002 12:15 pm Post subject: |
|
|
Jyrinx wrote: | klieber: You sure they've switched back to Python? |
Reasonably sure, yes. This interview with Daniel Robbins seems to confirm it.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
|
athlonkmf n00b
Joined: 02 Jun 2002 Posts: 37
|
Posted: Tue Jun 04, 2002 9:53 am Post subject: |
|
|
What if there was an option to create a list of ebuilds emerge can search through? much like debian. It seems that emerge now simply scans the complete portage-dir to find stuff. |
|
Back to top |
|
|
lx Veteran
Joined: 28 May 2002 Posts: 1012 Location: Netherlands
|
Posted: Tue Jun 04, 2002 10:11 am Post subject: |
|
|
Use reiserfs as filesystem. It's one of the reasons I use it, its extremly fast searching through direcory trees. So emerge -s will therefor go faster. Tried ext3 / Xfs but reiserfs beats them all with a length difference especially in this task. Just make a partition with a lot of files and do a find you'll be amazed.... _________________ "Remember there's a big difference between kneeling down and bending over.", Frank Zappa |
|
Back to top |
|
|
klieber Bodhisattva
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Tue Jun 04, 2002 11:38 am Post subject: |
|
|
athlonkmf wrote: | What if there was an option to create a list of ebuilds emerge can search through? |
As I mentioned earlier in this thread, this should happen with portage 2.0 -- my understanding is they're moving to a real database backend (berkely DB?) so searching should be much, much faster.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
|
craftyc Guru
Joined: 23 May 2002 Posts: 443 Location: Behind You.
|
Posted: Tue Jun 04, 2002 12:47 pm Post subject: |
|
|
klieber wrote: | athlonkmf wrote: | What if there was an option to create a list of ebuilds emerge can search through? |
As I mentioned earlier in this thread, this should happen with portage 2.0 -- my understanding is they're moving to a real database backend (berkely DB?) so searching should be much, much faster.
--kurt |
That would be brilliant. My system takes forever to do an emerge -s even though I have a P4 1.6Ghz with 256 Mb SDRAM (I think, can't remember the spec now).
I will be reinstalling Gentoo soon (so that I can change the USE variables etc. and get better performance out of my system) so I will also try XFS or maybe ReiserFS. _________________ Postcount ++ |
|
Back to top |
|
|
neuron Advocate
Joined: 28 May 2002 Posts: 2371
|
Posted: Tue Jun 04, 2002 1:44 pm Post subject: |
|
|
tweaking your harddrive settings with hdparm? , that'd make it go faster.
/sbin/hdparm /dev/hdX (or -i to see settings, and -t r -Tt to test), if your getting around 2-4mb/sec you should DEFENATLY change your settings. |
|
Back to top |
|
|
Absolon n00b
Joined: 29 Apr 2002 Posts: 1
|
Posted: Tue Jun 04, 2002 2:17 pm Post subject: Hardware problems? Not likely! |
|
|
I run Gentoo on a Dual Athlon MP 2000+ with 1GB of mem and Ultra 160 SCSI disks and 'emerge -s' still takes a _LONG_ time, so I doubt that it is hardware related. I would guess that the search code is slow and could need a bit of update
/Paul |
|
Back to top |
|
|
klieber Bodhisattva
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Tue Jun 04, 2002 2:31 pm Post subject: Re: Hardware problems? Not likely! |
|
|
Absolon wrote: | I would guess that the search code is slow and could need a bit of update |
The problem is portage is using a flat file system, rather than a database. This makes things like index lookups, etc., impossible or at least exceedingly slow.
You can tweak things till the cows come home, but until portage moves to a real database backend, it's still going to be slow.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
|
arkane l33t
Joined: 30 Apr 2002 Posts: 918 Location: Phoenix, AZ
|
Posted: Tue Jun 04, 2002 3:19 pm Post subject: |
|
|
Target wrote: | Requiring any kind of scripting engine also makes it impossible to shoehorn Gentoo into some applications, which may be the reason for all the hubaloo. |
I'm sure it's about the same as calling a perl script from any other application...
just an exec call.
--
Dan |
|
Back to top |
|
|
arkane l33t
Joined: 30 Apr 2002 Posts: 918 Location: Phoenix, AZ
|
Posted: Tue Jun 04, 2002 3:24 pm Post subject: Re: Hardware problems? Not likely! |
|
|
Quote: |
The problem is portage is using a flat file system, rather than a database. This makes things like index lookups, etc., impossible or at least exceedingly slow.
You can tweak things till the cows come home, but until portage moves to a real database backend, it's still going to be slow.
--kurt |
That would definately explain people having varying hardware and not having any better performance. If your using a flatfile database, it's all memory bandwidth and doesn't have much to do with how much ram you have. (cpu probably has something to do with it, but not alot)
Screw it, lets just hack it to use a Mysql database backend and call it a night LOL
*JOKING* |
|
Back to top |
|
|
Target Apprentice
Joined: 25 Apr 2002 Posts: 200
|
Posted: Tue Jun 04, 2002 11:56 pm Post subject: |
|
|
Quote: | I'm sure it's about the same as calling a perl script from any other application... |
Yes, but you still need perl or python installed. They ain't free disk (or flash rom) space-wise. |
|
Back to top |
|
|
|