| View previous topic :: View next topic |
| Author |
Message |
Nic-E
n00b
Joined: 08 Nov 2002
Posts: 6
Location: South Africa
|
 Posted: Mon Aug 25, 2003 2:49 pm Post subject: Portage getting too big and too slow |
 |
|
I have been noticing a steady decline in emerge/portage perfomance. First i thought it was just the crappy box i am using at work. But i have noticed other Gentoo installations as the same.
It takes forever just to search for a app. Can anyone tell me why this is happening?? Is there some way of speeding up emerge? I have heard of ppl porting emerge from python to C. Why is portage not following this yet?
_________________
--
Nic-E
-[^_^]- |
|
| Back to top |
|
 |
scriptkiddie
l33t
Joined: 30 Mar 2003
Posts: 955
|
| Posted: Mon Aug 25, 2003 2:53 pm Post subject: |
|
|
The Dux Linux team is taking Portage and converting it from Python to C and also adding the MySQL support for the database instead of text files.
It will run much faster and should be easier to update..
Maybe you could try cleaning up your files and such in distfiles and packages to make it run faster |
|
| Back to top |
|
|
maxmc
Guru
Joined: 14 Oct 2002
Posts: 480
Location: Linköping, Sweden
|
| Posted: Mon Aug 25, 2003 6:08 pm Post subject: |
|
|
| Is there a link available? |
|
| Back to top |
|
|
scriptkiddie
l33t
Joined: 30 Mar 2003
Posts: 955
|
| Posted: Mon Aug 25, 2003 6:10 pm Post subject: |
|
|
we have a non informative web site up right now at:
http://dux.sunsite.dk
The website that actually contains information will be set up this weekend at:
www.dux-linux.org |
|
| Back to top |
|
|
BitJam
Advocate
Joined: 12 Aug 2003
Posts: 2508
Location: Silver City, NM
|
| Posted: Mon Aug 25, 2003 6:53 pm Post subject: |
|
|
There are at least two ebuilds floating around that speed up emerge searches so much as to make them painless. I use esearch. It is in portage but masked as experimental. | Code: | ACCEPT_KEYWORDS="~x86" emerge esearch
eupdatedb
esearch whatever
esearch -S keyword |
Or do a search on these fora for esearch to find the other package that does just about the same thing.
The downside of these packages is that they take a minute or two to build up their databases. I just ran eupdatedb and it took 90 secs. Even so, I love the thing and highly recommend it. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9530
Location: beyond the rim
|
| Posted: Mon Aug 25, 2003 9:38 pm Post subject: |
|
|
The other downside of these scripts is that the results are static, so version numbers and (less likely) other fields might be wrong/outdated.
BTW, I wrote the other script, fastsearch, but I think esearch is better for everyday use as it uses the same output format as emerge -s (fastsearch is rather limited at the moment and I don't know if I'm going to change that) |
|
| Back to top |
|
|
smiler.se
Tux's lil' helper
Joined: 18 Aug 2003
Posts: 115
Location: Sweden - Europe - Earth
|
| Posted: Mon Aug 25, 2003 9:52 pm Post subject: |
|
|
See this thread for some more info and discussions on this subject.
_________________
Christian
Sig out of date. Please upgrade to a newer one. |
|
| Back to top |
|
|
puddpunk
l33t
Joined: 20 Jul 2002
Posts: 681
Location: New Zealand
|
| Posted: Mon Aug 25, 2003 11:38 pm Post subject: |
|
|
quite simply, C isn't that much faster than Python. Especially when the program is waiting on data from the disk (i.e. emerge -S app).
Python was chosen for portage because it's faster than most interpreted languages (like BASH), it's object orientated and _very_ easy to pick up, which means rapid development, which is essential for something so important to an ever changing distro.
A MySQL backend is a very good idea, but thats about the largest speed increase you'll see out of hacking portage. Perhaps some speed hacking on some dirty portage code could speed it up, but it's normally pretty good. |
|
| Back to top |
|
|
quixoticsycophant
n00b
Joined: 10 Jan 2003
Posts: 16
|
| Posted: Tue Aug 26, 2003 1:41 am Post subject: |
|
|
| Quote: | | A MySQL backend is a very good idea, |
I think this would create difficulties with regard to dealing with a malfunctioning system (very important!). What if there is problem and you need to start up mysql to view the data, but you can't start up mysql because you have a problem. It also smells a little like the Windows Registry (and I'll allow everyone to have their own opinion about that beast).
Or did you mean an *optional* MySQL database which redundantly mirrors the current /var/db/pkg setup? I'd be all for that.
| Quote: | | but thats about the largest speed increase you'll see out of hacking portage. Perhaps some speed hacking on some dirty portage code could speed it up, but it's normally pretty good. |
Actually that isn't true. With /usr/db/pkg cached (i.e. NO disk hits), the following perl script does the same thing as emerge -S but 18 times faster (1.35 seconds vs 24.22 seconds).
| Code: | #!/usr/bin/perl
die "Usage: $0 <regular-expression>\n" unless @ARGV == 1 ;
use warnings ;
use strict ;
use File::Find () ;
my $tree = "/usr/portage" ;
my $search = shift ;
my $ebuilds = {} ;
sub each_ebuild
{
return unless m!\.ebuild$!o ;
my($cat,$pkg) = m!^$tree/(.*?)/(.*?)/!o ;
return unless $cat ;
return unless $pkg ;
return if exists $ebuilds->{$cat} and exists $ebuilds->{$cat}->{$pkg} ;
local @ARGV = ($_) ;
while(<>)
{
my($desc) = m!DESCRIPTION\s*?\=\s*\"(.*?)\"!o ;
if( $desc and ($desc =~ m!$search!i or $pkg =~ m!$search!i) )
{
$ebuilds->{$cat}->{$pkg} = $desc ;
last ;
}
}
}
File::Find::find({ no_chdir => 1, wanted =>\&each_ebuild }, $tree) ;
for my $cat (sort keys %$ebuilds)
{
for my $pkg (sort keys %{$ebuilds->{$cat}})
{
print "$cat/$pkg: $ebuilds->{$cat}->{$pkg}\n" ;
}
}
|
The point is that simply optimizing disk I/O is not enough. The whole kit and kaboodle needs to be overhauled. Still, it is true that disk I/O should be first on the agenda.
-qs |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Tue Aug 26, 2003 9:36 am Post subject: |
|
|
| putting it in mysql would require mysql to be working... that's far from always the case, and I really wouldn't like that aproach... would be much better to cache stuff. |
|
| Back to top |
|
|
scriptkiddie
l33t
Joined: 30 Mar 2003
Posts: 955
|
| Posted: Tue Aug 26, 2003 12:02 pm Post subject: |
|
|
it would have to be an optional thing for Gentoo.. as for Dux, we can consider it default over the standard way..
Why would you say that MySQL does not work? I have never had a single problem with it and I use it for some heavy stuff |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Tue Aug 26, 2003 1:53 pm Post subject: |
|
|
| I use it a lot myself, and it's great, but I have experienced problems between versions. Either way, using huge package lik mysql rather than storing stuff on the file system, isn't such a great idea imho. |
|
| Back to top |
|
|
SNo0py
Apprentice
Joined: 12 Jul 2002
Posts: 270
Location: Vienna, Austria
|
| Posted: Tue Aug 26, 2003 2:25 pm Post subject: |
|
|
KISS - keep it simple and stupid.
What if somebody doesn't want to have mysql on his computer? Does rsync work with a mysql database?
I think it should be left as it is, but there should be a fast caching...
_________________
Sex is like hacking. You get in, you get out, and you hope you didnt leave something behind that can be traced back to you. |
|
| Back to top |
|
|
|
Portage & Programming
