View previous topic :: View next topic |
Author |
Message |
heim n00b
Joined: 31 May 2002 Posts: 5
|
Posted: Sun Jun 02, 2002 5:41 am Post subject: emerge search slowness... |
|
|
Im beginning to wonder why emerge is written in python; it seems to be awfully slow.
this evening, I set out to make a tool to search portage a little quicker.
here are my results:
Code: | [root@pdq ~]# time ./searchportage.sh mozilla
[ Results for search key : mozilla ]
[ Applications Found : 1 ]
net-www/mozilla
Latest version Available: 1.0_rc3-r50
Latest version Installed: 1.0_rc1-r1
Homepage: http://www.mozilla.org
Description:
The Mozilla Web Browser
real 0m0.562s
user 0m0.120s
sys 0m0.350s
[root@pdq ~]# time emerge -s mozilla
[ Results for search key : mozilla ]
[ Applications found : 1 ]
* net-www/mozilla
Latest version Available: 1.0_rc2-r1
Latest version Installed: 1.0_rc1-r1
Homepage: http://www.mozilla.org
Description:
The Mozilla Web Browser
real 0m9.283s
user 0m7.250s
sys 0m0.420s
|
these times are from a 300mhz G3. roughly 8x faster.
even more frightening are the results from a 150Mhz 604e, which follow:
Code: |
[root@tnt ~]# time ./searchportage.sh mozilla
...
real 0m1.192s
user 0m0.369s
sys 0m0.785s
[root@tnt ~]# time emerge -s mozilla
...
real 0m39.448s
user 0m37.428s
sys 0m1.925s
|
With that, here is the script:
Code: |
#!/bin/bash
#get PORTDIR
. /etc/make.globals
cd ${PORTDIR}
#set some vars
searchkey=${1}
searchkey_num_results=`find -type d -maxdepth 2 -path ./*${searchkey}*|wc -l`
#define some escape sequences for BOLD text.
B=" [0;5m"
NB="[0m"
echo '[ Results for search key : '${B}${searchkey}${NB}' ]'
echo '[ Applications Found : '${B}${searchkey_num_results}${NB}' ]'
for DIR in `find -type d -maxdepth 2 -path ./*${searchkey}*` ; do
echo
cd ${PORTDIR}/${DIR}
LATEST_AVAILABLE=`ls *.ebuild|sort -d|tail -n1`
DESCRIPTION=`grep ^DESCRIPTION ${LATEST_AVAILABLE}| \
sed -e 's/^DESCRIPTION[ ]*=[ ]*"//' -e 's/"$//'`
HOMEPAGE=`grep ^HOMEPAGE ${LATEST_AVAILABLE}| \
sed -e 's/^HOMEPAGE[ ]*=[ ]*"//' -e 's/"$//'`
PORT_NAME=`echo ${DIR}|sed -e 's/^\.\/[-a-zA-Z0-9.]*\///'`
CATEGORY_NAME=`echo $DIR|sed -e 's|^./||' -e 's|/[-a-zA-Z0-9.]*$||'`
LATEST_VERSION=`echo ${LATEST_AVAILABLE}|sed -e 's/.ebuild$//' -e 's/'${PORT_NAME}'-//'`
cd /var/db/pkg/${CATEGORY_NAME}
LATEST_INSTALLED=`ls|grep ${PORT_NAME}|sort -d|tail -n1|sed -e 's/'${PORT_NAME}'-//'`
if [ "${LATEST_INSTALLED}" = '' ]
then
LATEST_INSTALLED="[ Not Installed ]"
fi
echo " "${B}${CATEGORY_NAME}/${PORT_NAME}${NB}
echo " "Latest version Available: ${LATEST_VERSION}
echo " "Latest version Installed: ${LATEST_INSTALLED}
echo " "Homepage: ${HOMEPAGE}
echo " "Description:
echo " "${DESCRIPTION}
done
|
|
|
Back to top |
|
|
nzadrozny n00b
Joined: 22 May 2002 Posts: 11 Location: San Diego
|
Posted: Tue Jun 04, 2002 3:56 am Post subject: |
|
|
Well, actually, emerge *is* written in Python =)
I too have noticed a significant lag in any situation that emerge needs to search for a package. Not just for "emerge -s package" but for installation, or pretending to install. With slowness of that magnitude, I strongly suspect that their package name resolution algorithm needs some overhauling. |
|
Back to top |
|
|
proxy Apprentice
Joined: 20 Apr 2002 Posts: 260 Location: Chantilly, VA
|
Posted: Tue Jun 04, 2002 5:31 am Post subject: |
|
|
you also have to keep in mind that there emerge -s option does more than simply search the /usr/portage directory.
it also checks those files found vs the package.mask file. Ideally the search algo would load this mask into mem in an easy to traverse datastructure...but if they re-parse it redundantly, it could get ugly
my question is, why isnt portage using some pre-made database.
it would ROCK if i could qury for packages uses a SQL like syntax
well, I can dream can't I?
proxy |
|
Back to top |
|
|
Sivar Apprentice
Joined: 25 May 2002 Posts: 266 Location: USA
|
Posted: Wed Jun 05, 2002 12:34 am Post subject: |
|
|
proxy wrote: | you also have to keep in mind that there emerge -s option does more than simply search the /usr/portage directory.
it also checks those files found vs the package.mask file. Ideally the search algo would load this mask into mem in an easy to traverse datastructure...but if they re-parse it redundantly, it could get ugly
my question is, why isnt portage using some pre-made database.
it would ROCK if i could qury for packages uses a SQL like syntax
|
While I think that operating systems are rapidly approaching the point in which a powerful, integrated DB is needed, I don't think that requiring the installation (and setup?) of MySQL would be a great idea. _________________ The greatest deeds are still undone, the greatest songs are still unsung... |
|
Back to top |
|
|
dennis Tux's lil' helper
Joined: 06 Jun 2002 Posts: 84
|
Posted: Thu Jun 06, 2002 4:00 am Post subject: |
|
|
It doesnt need to be a SQL, but some Database system is needed, BADLY!
well the windows registry didnt work out to well, that needs to be rethought ...
but there is a lot of stuff that could (and should!) be replaced by a database, just take the config files ...
Dennis |
|
Back to top |
|
|
lx Veteran
Joined: 28 May 2002 Posts: 1012 Location: Netherlands
|
Posted: Fri Jun 07, 2002 10:28 am Post subject: |
|
|
CHECK THIS OUT
Code: |
time emerge -s gtk time emerge -s notfound
|
Code: |
gives
real 0m1.563.s 0m1.287.s
user 0m1.240s 0m1.220s
sys 0m0.100s 0m0.060s
|
rougly 1.5 seconds, why? ReiserFS, Its one of the main reasons I use it found out portage works so much faster, also find and other things, just searching the three works like a charm. It's got a database like file-structure, so...... Using a database looks like a good option to me, but I don't know what the consequences are,.... I have no complaint about portage with reiserfs but with another filesystem and a slow harddrive it can be anoying, I guess.
Furthermore I have a Quantum Fireball 30Gb/AS
which gets about 34Mb/s outer and 29Mb/s inner.
Athlon 1.2 (133fsb) ... 256MB RAM.
Cya lX. _________________ "Remember there's a big difference between kneeling down and bending over.", Frank Zappa |
|
Back to top |
|
|
lx Veteran
Joined: 28 May 2002 Posts: 1012 Location: Netherlands
|
Posted: Fri Jun 07, 2002 10:38 am Post subject: |
|
|
dennis wrote: |
but there is a lot of stuff that could (and should!) be replaced by a database, just take the config files ...
Dennis |
No way, think ReiserFS is onto something here, it can incorperate small files into the file tree, which is structured like a database, so /etc and things work very fast and don't take up lots of space. Well my previous mail proves this.
Database sucks major ass, well take windows (even the thought alone), I know what the problem is, but how am I gonna change the registry without the tools??????? One thing I like about linux is that it's easy to repair / change things even without special tools. Besides startup times aren't that long compared to Windows, and using Linux I even start a lot more (usefull) services. I am in control, not Window$.
Also every file has its own configuration file, and doesn't mess with the configuration file of other apps. When I still used Window$, I reinstalled it each month, but when I fixed other computers the registry was huge, bloated, So maybe introduce some simple databases, but don't put everything in 1 giant database, aarrrggghhh.
Interresting note, Windows is also developing a database like file-structure which incorperates the files. _________________ "Remember there's a big difference between kneeling down and bending over.", Frank Zappa |
|
Back to top |
|
|
craftyc Guru
Joined: 23 May 2002 Posts: 443 Location: Behind You.
|
Posted: Fri Jun 07, 2002 11:17 am Post subject: |
|
|
Take a look at these search times:
Code: | craftyc-ftp root # time emerge -s mozilla
[ Results for search key : mozilla ]
[ Applications found : 1 ]
* net-www/mozilla
Latest version Available: 1.0-r1
Latest version Installed: 1.0-r1
Homepage: http://www.mozilla.org
Description:
The Mozilla Web Browser
real 0m2.363s
user 0m1.941s
sys 0m0.307s
|
This is on an xfs partition. Pretty funky hay. _________________ Postcount ++ |
|
Back to top |
|
|
dennis Tux's lil' helper
Joined: 06 Jun 2002 Posts: 84
|
Posted: Fri Jun 07, 2002 8:04 pm Post subject: |
|
|
LX:
i didnt know reiserFS could do that ... i dont have it installed, cuz they said the RFS support wasnt to good at the time i set up gentoo ...
to the windows prob you can just write .reg files ... (also you can downlod many from the net) also i know many people that use the scriptinghost to write litle scripts that change stuff in the registry ...
Dennis |
|
Back to top |
|
|
lx Veteran
Joined: 28 May 2002 Posts: 1012 Location: Netherlands
|
Posted: Fri Jun 07, 2002 11:46 pm Post subject: |
|
|
dennis wrote: | to the windows prob you can just write .reg files ... (also you can downlod many from the net) also i know many people that use the scriptinghost to write litle scripts that change stuff in the registry ...
Dennis |
I know you can export / write reg files, but can you add them from DOS, and can you browse the registry in order to look for a mistake / error. What's easier then just browse your harddisk, besides the windows registry ain't organized, it's own big pile, maybe the should have opt for multiple registries. Strange that I can find things on Linux, but in the windows registry everybody dumps their load somewhere else, its chaos _________________ "Remember there's a big difference between kneeling down and bending over.", Frank Zappa |
|
Back to top |
|
|
AutoBot l33t
Joined: 22 Apr 2002 Posts: 968 Location: Usually Out
|
Posted: Tue Jun 11, 2002 7:19 pm Post subject: |
|
|
Portage was destined for c++ running in conjuction with a firebird database, but in the end the decision was made to stick with python for the time being. _________________ This message self destructed a long time ago. |
|
Back to top |
|
|
|