Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Can I make emerge -s go faster?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
bobby
Guest





PostPosted: Sat May 18, 2002 7:50 pm    Post subject: Can I make emerge -s go faster? Reply with quote

Is there a way to make emerge -s go faster? I want to search for things, but I hate to wait.


If not, ideas on strategies for making it faster? A cache, perhaps?



Shanks,

Bobby

quicker faster better
Back to top
rommel
Veteran
Veteran


Joined: 19 Apr 2002
Posts: 1140
Location: Williamsburg Virginia

PostPosted: Sat May 18, 2002 7:56 pm    Post subject: Reply with quote

more ram faster cpu ....raid?

i just did it after reading your thread and it took like a second to see the version of portage that i had and the latest out....so post something more specific about your setup...internet connection cpu ddr or sdram...how much of it...cpu speed...blah blah blanh

ciao
Back to top
View user's profile Send private message
Guest






PostPosted: Sun May 19, 2002 1:18 am    Post subject: Reply with quote

honestly I hate waiting too ... my system is a 333 K6-2 with 128 megs of ram
it takes long enough for the blasted emege --pretend statments and what
not. a cache tied into the rsync command would be nice ... something ala
locate and updatedb

emege --cache --clean rsync
or something like that ... so that you can use it if you want
but arn't forced to
Back to top
BradN
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2379
Location: Wisconsin (USA)

PostPosted: Sun May 19, 2002 3:06 am    Post subject: Reply with quote

I'm on a P3 515mhz with 448 megs of ram... emerge takes a long time to load it's package list (or whatever it may be) - about 10 seconds or so roughly, but only takes a couple seconds the next time you run it because it's all cached in the filesystem. A cache would probably help a lot here.
Back to top
View user's profile Send private message
coshaw
n00b
n00b


Joined: 12 May 2002
Posts: 7

PostPosted: Sun May 19, 2002 3:49 am    Post subject: Reply with quote

I am not sure, but it seems if you use XFS, emerge -s is faster, especially compared with ext3. I didnt try reiserfs. :)
Back to top
View user's profile Send private message
WhyteWolf
Apprentice
Apprentice


Joined: 01 May 2002
Posts: 168
Location: Las Vegas

PostPosted: Sun May 19, 2002 4:27 am    Post subject: Reply with quote

with a proper cacheing system it would be even faster ... sence it wouldn't
hafta read every file just the cache ...

dependency checking would take a matter of nanoseconds
and searching could be much cleaner
Back to top
View user's profile Send private message
TheWart
Guru
Guru


Joined: 10 May 2002
Posts: 432
Location: Nashville,TN - USA

PostPosted: Sun May 19, 2002 4:48 am    Post subject: Reply with quote

IIRC, they are working on some sort of quicker cache thing fro portage 2.0....
_________________
Face it, we are all noobs.

On the box it said it was designed for Win XP or better, so why won't it work with Linux?
Back to top
View user's profile Send private message
AutoBot
l33t
l33t


Joined: 22 Apr 2002
Posts: 968
Location: Usually Out

PostPosted: Sun May 19, 2002 4:54 am    Post subject: Reply with quote

coshaw wrote:
I am not sure, but it seems if you use XFS, emerge -s is faster, especially compared with ext3. I didnt try reiserfs. :)

It doesn't seem to be faster on XFS compared to ext3 it is, when I ran gentoo on XFS emerge rsync and emerge -s <foopackage> ran almost instantanious. But now on ext3 I get major lag when running either, enough to *almost* warrant a back up and a change over to XFS again but not until the preemp gets worked out properly.

My Gentoo Box For Reference:
10gig 5200rpm HD --- hdparm Cached: 94.38 MB/sec Buffered: 25.45 MB/sec
40gig USB 2.0 Buslink HD
533 Celeron
192 Meg Ram
GeForce II MX 400
Cable: 300kbs Down / 50kbs Up
_________________
This message self destructed a long time ago.
Back to top
View user's profile Send private message
sibn
n00b
n00b


Joined: 15 May 2002
Posts: 36

PostPosted: Sun May 19, 2002 4:18 pm    Post subject: Reply with quote

rm -rf /usr/portage should do the trick. Of course, you should not do this because it will break portage (and you'll have to have some kind of backup plan for restoring the ebuild tree), and every search you do will come up dry. But each search will take less than a second. :twisted:

You have been warned: Don't do this.
_________________
Hey um, this is my signature.
Back to top
View user's profile Send private message
rommel
Veteran
Veteran


Joined: 19 Apr 2002
Posts: 1140
Location: Williamsburg Virginia

PostPosted: Sun May 19, 2002 7:37 pm    Post subject: Reply with quote

i did notice a big difference with lsr after i had gentoo the first time installed on just a normal partition then the seccond install i created a raid 0 array so the /mnt/gentoo dir was created adn mounted on /dev/md0 ....from that there was a big improvement on anything related to emerge....read transfers went from 38mb/s on a normal ibm 7200 60gxp to 72mb/s with using two of these drives and the lsr.

try it you'll like it

ciao
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Sun May 19, 2002 9:42 pm    Post subject: Reply with quote

TheWart wrote:
IIRC, they are working on some sort of quicker cache thing fro portage 2.0....


Well, they're working on a real database for a portage backend, as opposed to trying to parse through a bunch of text files. This will make things like 'emerge -s' much faster.

You can learn a bit more about portage 2.0 here, though be warned that at least some of that info is out of date. (for instance, last I heard, portage 2.0 was still going to be python-based, not C++ based -- definitely a sore subject among some members of the gentoo dev team.) So, take what you read with a grain of salt.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
bobby
Guest





PostPosted: Sun May 19, 2002 11:08 pm    Post subject: Reply with quote

Ok, thanks, all.

To sum up (so far), it sounds like hardware bottlenecks (memory) and faster (for this app) filesystems are the best options. The issue is mostly tied to the scanning of many file in the filesystem -- so disk and filesystem issues are the main optimizers for the file scanning stuff.

And, there's talk of software improvements in portage 2.0.
Back to top
Jyrinx
Tux's lil' helper
Tux's lil' helper


Joined: 03 May 2002
Posts: 92
Location: Carleton College - Northfield, MN

PostPosted: Mon May 20, 2002 6:09 am    Post subject: Reply with quote

(off-topic) klieber: You sure they've switched back to Python? That would make me quite happy ... I'm still trying to figure out how they went from the practical, pragmatic Python to their ultra-anal, ideology-driven "let's use 90% of our brainpower making sure we're following the style guide by using GNU syntax for stack allocated variables and ours for others" style of C++ ... I knew I would never contribute if I had to do that.

Jyrinx
jyrinx@mindspring.com
Back to top
View user's profile Send private message
Target
Apprentice
Apprentice


Joined: 25 Apr 2002
Posts: 199

PostPosted: Mon May 20, 2002 8:19 am    Post subject: Reply with quote

Requiring any kind of scripting engine also makes it impossible to shoehorn Gentoo into some applications, which may be the reason for all the hubaloo.
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon May 20, 2002 12:15 pm    Post subject: Reply with quote

Jyrinx wrote:
klieber: You sure they've switched back to Python?


Reasonably sure, yes. This interview with Daniel Robbins seems to confirm it.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
athlonkmf
n00b
n00b


Joined: 02 Jun 2002
Posts: 37

PostPosted: Tue Jun 04, 2002 9:53 am    Post subject: Reply with quote

What if there was an option to create a list of ebuilds emerge can search through? much like debian. It seems that emerge now simply scans the complete portage-dir to find stuff.
Back to top
View user's profile Send private message
lx
Veteran
Veteran


Joined: 28 May 2002
Posts: 1012
Location: Netherlands

PostPosted: Tue Jun 04, 2002 10:11 am    Post subject: Reply with quote

Quote:
REISERFS

Use reiserfs as filesystem. It's one of the reasons I use it, its extremly fast searching through direcory trees. So emerge -s will therefor go faster. Tried ext3 / Xfs but reiserfs beats them all with a length difference especially in this task. Just make a partition with a lot of files and do a find you'll be amazed.... 8O
_________________
"Remember there's a big difference between kneeling down and bending over.", Frank Zappa
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Tue Jun 04, 2002 11:38 am    Post subject: Reply with quote

athlonkmf wrote:
What if there was an option to create a list of ebuilds emerge can search through?


As I mentioned earlier in this thread, this should happen with portage 2.0 -- my understanding is they're moving to a real database backend (berkely DB?) so searching should be much, much faster.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
craftyc
Guru
Guru


Joined: 23 May 2002
Posts: 443
Location: Behind You.

PostPosted: Tue Jun 04, 2002 12:47 pm    Post subject: Reply with quote

klieber wrote:
athlonkmf wrote:
What if there was an option to create a list of ebuilds emerge can search through?


As I mentioned earlier in this thread, this should happen with portage 2.0 -- my understanding is they're moving to a real database backend (berkely DB?) so searching should be much, much faster.

--kurt


That would be brilliant. My system takes forever to do an emerge -s even though I have a P4 1.6Ghz with 256 Mb SDRAM (I think, can't remember the spec now).

I will be reinstalling Gentoo soon (so that I can change the USE variables etc. and get better performance out of my system) so I will also try XFS or maybe ReiserFS.
_________________
Postcount ++
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Tue Jun 04, 2002 1:44 pm    Post subject: Reply with quote

tweaking your harddrive settings with hdparm? :), that'd make it go faster.

/sbin/hdparm /dev/hdX (or -i to see settings, and -t r -Tt to test), if your getting around 2-4mb/sec you should DEFENATLY change your settings.
Back to top
View user's profile Send private message
Absolon
n00b
n00b


Joined: 29 Apr 2002
Posts: 1

PostPosted: Tue Jun 04, 2002 2:17 pm    Post subject: Hardware problems? Not likely! Reply with quote

I run Gentoo on a Dual Athlon MP 2000+ with 1GB of mem and Ultra 160 SCSI disks and 'emerge -s' still takes a _LONG_ time, so I doubt that it is hardware related. I would guess that the search code is slow and could need a bit of update :)


/Paul
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Tue Jun 04, 2002 2:31 pm    Post subject: Re: Hardware problems? Not likely! Reply with quote

Absolon wrote:
I would guess that the search code is slow and could need a bit of update :)


The problem is portage is using a flat file system, rather than a database. This makes things like index lookups, etc., impossible or at least exceedingly slow.

You can tweak things till the cows come home, but until portage moves to a real database backend, it's still going to be slow.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
arkane
l33t
l33t


Joined: 30 Apr 2002
Posts: 918
Location: Phoenix, AZ

PostPosted: Tue Jun 04, 2002 3:19 pm    Post subject: Reply with quote

Target wrote:
Requiring any kind of scripting engine also makes it impossible to shoehorn Gentoo into some applications, which may be the reason for all the hubaloo.


I'm sure it's about the same as calling a perl script from any other application...

just an exec call.

--
Dan
Back to top
View user's profile Send private message
arkane
l33t
l33t


Joined: 30 Apr 2002
Posts: 918
Location: Phoenix, AZ

PostPosted: Tue Jun 04, 2002 3:24 pm    Post subject: Re: Hardware problems? Not likely! Reply with quote

Quote:

The problem is portage is using a flat file system, rather than a database. This makes things like index lookups, etc., impossible or at least exceedingly slow.

You can tweak things till the cows come home, but until portage moves to a real database backend, it's still going to be slow.

--kurt


That would definately explain people having varying hardware and not having any better performance. If your using a flatfile database, it's all memory bandwidth and doesn't have much to do with how much ram you have. (cpu probably has something to do with it, but not alot)

Screw it, lets just hack it to use a Mysql database backend and call it a night LOL

*JOKING*
Back to top
View user's profile Send private message
Target
Apprentice
Apprentice


Joined: 25 Apr 2002
Posts: 199

PostPosted: Tue Jun 04, 2002 11:56 pm    Post subject: Reply with quote

Quote:
I'm sure it's about the same as calling a perl script from any other application...


Yes, but you still need perl or python installed. They ain't free disk (or flash rom) space-wise. :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum