Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
emerge search slowness...
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
heim
n00b
n00b


Joined: 31 May 2002
Posts: 5

PostPosted: Sun Jun 02, 2002 5:41 am    Post subject: emerge search slowness... Reply with quote

Im beginning to wonder why emerge is written in python; it seems to be awfully slow.
this evening, I set out to make a tool to search portage a little quicker.

here are my results:
Code:
[root@pdq ~]# time ./searchportage.sh mozilla
[ Results for search key :  mozilla ]
[ Applications Found :   1 ]

    net-www/mozilla
     Latest version Available: 1.0_rc3-r50
     Latest version Installed: 1.0_rc1-r1
     Homepage: http://www.mozilla.org
     Description:
     The Mozilla Web Browser

real    0m0.562s
user    0m0.120s
sys     0m0.350s

[root@pdq ~]# time emerge -s mozilla
[ Results for search key : mozilla ]
[ Applications found : 1 ]
 
*  net-www/mozilla
      Latest version Available: 1.0_rc2-r1
      Latest version Installed: 1.0_rc1-r1
      Homepage: http://www.mozilla.org
      Description:
      The Mozilla Web Browser
 

real    0m9.283s
user    0m7.250s
sys     0m0.420s

these times are from a 300mhz G3. roughly 8x faster.

even more frightening are the results from a 150Mhz 604e, which follow:
Code:

[root@tnt ~]# time ./searchportage.sh mozilla
...
real    0m1.192s
user    0m0.369s
sys     0m0.785s
[root@tnt ~]# time emerge -s mozilla 
...
real    0m39.448s
user    0m37.428s
sys     0m1.925s


With that, here is the script:

Code:

#!/bin/bash

#get PORTDIR
. /etc/make.globals
cd ${PORTDIR}

#set some vars
searchkey=${1}
searchkey_num_results=`find -type d -maxdepth 2 -path ./*${searchkey}*|wc -l`
#define some escape sequences for BOLD text.
B="  "
NB=""

echo '[ Results for search key : '${B}${searchkey}${NB}' ]'
echo '[ Applications Found : '${B}${searchkey_num_results}${NB}' ]'
for DIR in `find -type d -maxdepth 2 -path ./*${searchkey}*` ; do
   echo
   cd ${PORTDIR}/${DIR}
   LATEST_AVAILABLE=`ls *.ebuild|sort -d|tail -n1`
   DESCRIPTION=`grep ^DESCRIPTION ${LATEST_AVAILABLE}| \
      sed -e 's/^DESCRIPTION[ ]*=[ ]*"//' -e 's/"$//'`
   HOMEPAGE=`grep ^HOMEPAGE ${LATEST_AVAILABLE}| \
      sed -e 's/^HOMEPAGE[ ]*=[ ]*"//' -e 's/"$//'`
   PORT_NAME=`echo ${DIR}|sed -e 's/^\.\/[-a-zA-Z0-9.]*\///'`
   CATEGORY_NAME=`echo $DIR|sed -e 's|^./||' -e 's|/[-a-zA-Z0-9.]*$||'`
   LATEST_VERSION=`echo ${LATEST_AVAILABLE}|sed -e 's/.ebuild$//' -e 's/'${PORT_NAME}'-//'`
   cd /var/db/pkg/${CATEGORY_NAME}
   LATEST_INSTALLED=`ls|grep ${PORT_NAME}|sort -d|tail -n1|sed -e 's/'${PORT_NAME}'-//'`
   if [ "${LATEST_INSTALLED}" = '' ]
   then
      LATEST_INSTALLED="[ Not Installed ]"
   fi

echo "  "${B}${CATEGORY_NAME}/${PORT_NAME}${NB}
echo "      "Latest version Available: ${LATEST_VERSION}
echo "      "Latest version Installed:  ${LATEST_INSTALLED}
echo "      "Homepage: ${HOMEPAGE}
echo "      "Description:
echo "      "${DESCRIPTION}
done

Back to top
View user's profile Send private message
nzadrozny
n00b
n00b


Joined: 22 May 2002
Posts: 11
Location: San Diego

PostPosted: Tue Jun 04, 2002 3:56 am    Post subject: Reply with quote

Well, actually, emerge *is* written in Python =)

I too have noticed a significant lag in any situation that emerge needs to search for a package. Not just for "emerge -s package" but for installation, or pretending to install. With slowness of that magnitude, I strongly suspect that their package name resolution algorithm needs some overhauling.
Back to top
View user's profile Send private message
proxy
Apprentice
Apprentice


Joined: 20 Apr 2002
Posts: 260
Location: Chantilly, VA

PostPosted: Tue Jun 04, 2002 5:31 am    Post subject: Reply with quote

you also have to keep in mind that there emerge -s option does more than simply search the /usr/portage directory.

it also checks those files found vs the package.mask file. Ideally the search algo would load this mask into mem in an easy to traverse datastructure...but if they re-parse it redundantly, it could get ugly :(

my question is, why isnt portage using some pre-made database.

it would ROCK if i could qury for packages uses a SQL like syntax ;)

well, I can dream can't I?

proxy
Back to top
View user's profile Send private message
Sivar
Apprentice
Apprentice


Joined: 25 May 2002
Posts: 266
Location: USA

PostPosted: Wed Jun 05, 2002 12:34 am    Post subject: Reply with quote

proxy wrote:
you also have to keep in mind that there emerge -s option does more than simply search the /usr/portage directory.

it also checks those files found vs the package.mask file. Ideally the search algo would load this mask into mem in an easy to traverse datastructure...but if they re-parse it redundantly, it could get ugly :(

my question is, why isnt portage using some pre-made database.

it would ROCK if i could qury for packages uses a SQL like syntax ;)


While I think that operating systems are rapidly approaching the point in which a powerful, integrated DB is needed, I don't think that requiring the installation (and setup?) of MySQL would be a great idea.
_________________
The greatest deeds are still undone, the greatest songs are still unsung...
Back to top
View user's profile Send private message
dennis
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jun 2002
Posts: 84

PostPosted: Thu Jun 06, 2002 4:00 am    Post subject: Reply with quote

It doesnt need to be a SQL, but some Database system is needed, BADLY!

well the windows registry didnt work out to well, that needs to be rethought ... :lol:

but there is a lot of stuff that could (and should!) be replaced by a database, just take the config files ... :twisted:

Dennis
Back to top
View user's profile Send private message
lx
Veteran
Veteran


Joined: 28 May 2002
Posts: 1012
Location: Netherlands

PostPosted: Fri Jun 07, 2002 10:28 am    Post subject: Reply with quote

CHECK THIS OUT
Code:

time emerge -s gtk     time emerge -s notfound

Code:

gives
real   0m1.563.s       0m1.287.s
user   0m1.240s        0m1.220s
sys    0m0.100s        0m0.060s


rougly 1.5 seconds, why? ReiserFS, Its one of the main reasons I use it found out portage works so much faster, also find and other things, just searching the three works like a charm. It's got a database like file-structure, so...... Using a database looks like a good option to me, but I don't know what the consequences are,.... I have no complaint about portage with reiserfs but with another filesystem and a slow harddrive it can be anoying, I guess.

Furthermore I have a Quantum Fireball 30Gb/AS
which gets about 34Mb/s outer and 29Mb/s inner.
Athlon 1.2 (133fsb) ... 256MB RAM.

Cya lX.
_________________
"Remember there's a big difference between kneeling down and bending over.", Frank Zappa
Back to top
View user's profile Send private message
lx
Veteran
Veteran


Joined: 28 May 2002
Posts: 1012
Location: Netherlands

PostPosted: Fri Jun 07, 2002 10:38 am    Post subject: Reply with quote

dennis wrote:

but there is a lot of stuff that could (and should!) be replaced by a database, just take the config files ... :twisted:
Dennis

No way, think ReiserFS is onto something here, it can incorperate small files into the file tree, which is structured like a database, so /etc and things work very fast and don't take up lots of space. Well my previous mail proves this.
Database sucks major ass, well take windows (even the thought alone), I know what the problem is, but how am I gonna change the registry without the tools??????? One thing I like about linux is that it's easy to repair / change things even without special tools. Besides startup times aren't that long compared to Windows, and using Linux I even start a lot more (usefull) services. I am in control, not Window$.
Also every file has its own configuration file, and doesn't mess with the configuration file of other apps. When I still used Window$, I reinstalled it each month, but when I fixed other computers the registry was huge, bloated, So maybe introduce some simple databases, but don't put everything in 1 giant database, aarrrggghhh.

Interresting note, Windows is also developing a database like file-structure which incorperates the files.
_________________
"Remember there's a big difference between kneeling down and bending over.", Frank Zappa
Back to top
View user's profile Send private message
craftyc
Guru
Guru


Joined: 23 May 2002
Posts: 443
Location: Behind You.

PostPosted: Fri Jun 07, 2002 11:17 am    Post subject: Reply with quote

Take a look at these search times:
Code:
craftyc-ftp root # time emerge -s mozilla
[ Results for search key : mozilla ]
[ Applications found : 1 ]

*  net-www/mozilla
      Latest version Available: 1.0-r1
      Latest version Installed: 1.0-r1
      Homepage: http://www.mozilla.org
      Description:
      The Mozilla Web Browser


real    0m2.363s
user    0m1.941s
sys     0m0.307s


This is on an xfs partition. Pretty funky hay. :)
_________________
Postcount ++
Back to top
View user's profile Send private message
dennis
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jun 2002
Posts: 84

PostPosted: Fri Jun 07, 2002 8:04 pm    Post subject: Reply with quote

LX:
i didnt know reiserFS could do that ... i dont have it installed, cuz they said the RFS support wasnt to good at the time i set up gentoo ...

to the windows prob you can just write .reg files ... (also you can downlod many from the net) also i know many people that use the scriptinghost to write litle scripts that change stuff in the registry ...

Dennis
Back to top
View user's profile Send private message
lx
Veteran
Veteran


Joined: 28 May 2002
Posts: 1012
Location: Netherlands

PostPosted: Fri Jun 07, 2002 11:46 pm    Post subject: Reply with quote

dennis wrote:
to the windows prob you can just write .reg files ... (also you can downlod many from the net) also i know many people that use the scriptinghost to write litle scripts that change stuff in the registry ...
Dennis


I know you can export / write reg files, but can you add them from DOS, and can you browse the registry in order to look for a mistake / error. What's easier then just browse your harddisk, besides the windows registry ain't organized, it's own big pile, maybe the should have opt for multiple registries. Strange that I can find things on Linux, but in the windows registry everybody dumps their load somewhere else, its chaos :twisted:
_________________
"Remember there's a big difference between kneeling down and bending over.", Frank Zappa
Back to top
View user's profile Send private message
AutoBot
l33t
l33t


Joined: 22 Apr 2002
Posts: 968
Location: Usually Out

PostPosted: Tue Jun 11, 2002 7:19 pm    Post subject: Reply with quote

Portage was destined for c++ running in conjuction with a firebird database, but in the end the decision was made to stick with python for the time being.
_________________
This message self destructed a long time ago.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum