| View previous topic :: View next topic |
| Author |
Message |
alinv
Guru
Joined: 19 Nov 2002
Posts: 395
Location: Bucharest
|
 Posted: Thu Oct 20, 2005 12:50 am Post subject: |
 |
|
Did anybody else tried the anydbm backend for portage?
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B. |
|
| Back to top |
|
 |
marsclic
n00b
Joined: 02 Mar 2003
Posts: 71
|
| Posted: Thu Oct 20, 2005 6:35 am Post subject: |
|
|
| Yes Salivian, you are right. However, subsequent emerge syncs are 10x faster than before. It seems as if the cache was always being flushed at emerge syncs, before the python-updater ran. On the other hand, I observed that after deleting the cache "emerge sync" stuck at 52% but with a lot of disk activity, as where before it would just burn cpu cycles. Whatever it was, is gone now. |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Thu Oct 20, 2005 6:40 am Post subject: |
|
|
| Quote: | and the KDE monolithic ebuilds will continue until 4.0...  |
Have you actually tried using the split ebuilds? last time I installed xmms it took about 20x longer than it should have. Mainly because they assume noone wants any of the plugins that SHIP WITH xmms, instead the gentoo devs in all of thier infinite wisdom split the plugins off into seperate ebuilds, but kept them all in the same "monolithic" tar ball. So now instead of decompressing and configuring ONCE, you get to do it once of each part of xmms. For large packages like KDE, I can just immagine how long it'll take then. Honestly I even filed a bug report on how much longer it takes to install the split xmms, and All I got back was a "Who cares what you think, we're keeping it this way!". Also in that report I suggested they split up the xmms package to facilitate the new split structure, but THEY REFUSED! So I happily suggested they contact XMMS and ask them really nicely to package XMMS in a way compatable with the new split ebuild crap.
If they get rid of the monolithic ebuilds and they use the same structure they used for the xmms split ebuilds, I'll likely jump ship. It'll bump KDE's install time up by 2 times or more. I can't see how that a _good_ thing.
I dont know, being unsympathetic to the users seems to be a prerequsite for being a gentoo dev (someone PLEASE prove me wrong). |
|
| Back to top |
|
|
alinv
Guru
Joined: 19 Nov 2002
Posts: 395
Location: Bucharest
|
| Posted: Thu Oct 20, 2005 6:58 am Post subject: |
|
|
I traced the emerge --metadata process and indeed it seems it enters the kde-base directory at 50%.
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B. |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Thu Oct 20, 2005 7:13 am Post subject: |
|
|
| I'm going to have to add all of the split kde packages (theres gotta be hundreds of em) to the exclude list.. see if it helps. |
|
| Back to top |
|
|
alinv
Guru
Joined: 19 Nov 2002
Posts: 395
Location: Bucharest
|
| Posted: Thu Oct 20, 2005 7:20 am Post subject: |
|
|
| Tomasu wrote: | | I'm going to have to add all of the split kde packages (theres gotta be hundreds of em) to the exclude list.. see if it helps. |
Somebody else mentioned it on the previous page and there were only a couple of lines.
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B. |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Thu Oct 20, 2005 7:25 am Post subject: |
|
|
Not quite, that blocked all of the KDE packages afaics. but I think this: | Code: | | ls kde-*/*/*.ebuild | xargs grep kde-meta | cut -d'/' -f1,2 | uniq |
should pick up all of the split ebuild packages  |
|
| Back to top |
|
|
alinv
Guru
Joined: 19 Nov 2002
Posts: 395
Location: Bucharest
|
| Posted: Thu Oct 20, 2005 8:36 am Post subject: |
|
|
I was looking at /var/cache/edb/dep/usr/portage/kde-base-eclass.cpickle while emerge sync was between 50% and 52%. The size of the file changed from 0 to ~700kb and back to 0 about a hundred times (didn't count them, but they were far too many). Needless to say this doesn't seem quite right.
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B. |
|
| Back to top |
|
|
nxsty
Veteran
Joined: 23 Jun 2004
Posts: 1556
Location: .se
|
| Posted: Thu Oct 20, 2005 8:45 am Post subject: |
|
|
| Tomasu wrote: | Have you actually tried using the split ebuilds? last time I installed xmms it took about 20x longer than it should have. Mainly because they assume noone wants any of the plugins that SHIP WITH xmms, instead the gentoo devs in all of thier infinite wisdom split the plugins off into seperate ebuilds, but kept them all in the same "monolithic" tar ball. So now instead of decompressing and configuring ONCE, you get to do it once of each part of xmms. For large packages like KDE, I can just immagine how long it'll take then. Honestly I even filed a bug report on how much longer it takes to install the split xmms, and All I got back was a "Who cares what you think, we're keeping it this way!". Also in that report I suggested they split up the xmms package to facilitate the new split structure, but THEY REFUSED! So I happily suggested they contact XMMS and ask them really nicely to package XMMS in a way compatable with the new split ebuild crap.
If they get rid of the monolithic ebuilds and they use the same structure they used for the xmms split ebuilds, I'll likely jump ship. It'll bump KDE's install time up by 2 times or more. I can't see how that a _good_ thing.
I dont know, being unsympathetic to the users seems to be a prerequsite for being a gentoo dev (someone PLEASE prove me wrong). |
If you want fast installations then gentoo is obviously not the distro for you. I like the flexibility the split KDE ebuild gives and no other distro currently offers that. And you don't have to install all split ebuilds for a package but only those you need, cutting buildtime and disk-usage a lot. |
|
| Back to top |
|
|
lost+found
Guru
Joined: 15 Nov 2004
Posts: 310
Location: North~Sea~Coa~s~~t~~~
|
| Posted: Thu Oct 20, 2005 8:47 am Post subject: |
|
|
| lost+found wrote: | Gnome people:
/etc/make.conf
RSYNC_EXCLUDEFROM="/etc/portage/rsync_excludes"
/etc/portage/rsync_excludes
- kde-*/
- metadata/cache/kde-*/ |
| aethyr wrote: | | nevermind, excluding KDE from rsync updates didn't fix it. |
Hmm, I forgot to write it, but to prevent emerge --metadata to calculate the cache again with the outdated stuff: remove the existing /usr/portage/kde-*/ and /usr/portage/metadata/cache/kde-*/ manually. Maybe then it works... It's not a good idea to keep ebuilds that won't be updated in the Portage tree anyway. I cut my /usr/portage by 93%, but included KDE (monolithic): even then my syncs don't hang at 50%. I'm sure the more you exclude, the faster Portage gets. My updating cache takes a few seconds, syncs less than a minute.
Last edited by lost+found on Thu Oct 20, 2005 9:03 am; edited 1 time in total |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Thu Oct 20, 2005 8:50 am Post subject: |
|
|
| Quote: | | If you want fast installations then gentoo is obviously not the distro for you. |
I'm not looking for fast. I'm looking for not slower than what I percived as humanly possible. I happen to like to install most of the base kde packages, because its convienient. As is emerge, when it doesn't take 30 seconds just to load up python and all of its deps, and then a minute to find the package you gave it in portage... Honestly.
edit:
| Quote: | | And you don't have to install all split ebuilds for a package but only those you need, cutting buildtime and disk-usage a lot. |
I also have no problem with the idea of split ebuilds. Had it been implemented correctly, I'd love it. It'd save time, Diskspace, sanity, etc. |
|
| Back to top |
|
|
wyv3rn
Apprentice
Joined: 18 Aug 2005
Posts: 154
Location: USA
|
| Posted: Thu Oct 20, 2005 3:42 pm Post subject: |
|
|
| Tomasu wrote: | | Quote: | | and the KDE monolithic ebuilds will continue until 4.0... |
Have you actually tried using the split ebuilds? last time I installed xmms it took about 20x longer than it should have. Mainly because they assume noone wants any of the plugins that SHIP WITH xmms, instead the gentoo devs in all of thier infinite wisdom split the plugins off into seperate ebuilds, but kept them all in the same "monolithic" tar ball. So now instead of decompressing and configuring ONCE, you get to do it once of each part of xmms. For large packages like KDE, I can just immagine how long it'll take then. Honestly I even filed a bug report on how much longer it takes to install the split xmms, and All I got back was a "Who cares what you think, we're keeping it this way!". Also in that report I suggested they split up the xmms package to facilitate the new split structure, but THEY REFUSED! So I happily suggested they contact XMMS and ask them really nicely to package XMMS in a way compatable with the new split ebuild crap.
If they get rid of the monolithic ebuilds and they use the same structure they used for the xmms split ebuilds, I'll likely jump ship. It'll bump KDE's install time up by 2 times or more. I can't see how that a _good_ thing.
I dont know, being unsympathetic to the users seems to be a prerequsite for being a gentoo dev (someone PLEASE prove me wrong). |
This is way off topic. We're not here to debate the merits of KDE split ebuilds. They are only mentioned here as they pertain to the topic. Debate this elsewhere.
The split ebuilds offer excellent flexibility and can actually decrease build time if you don't want to install everything. You might also want to look into distcc (if you have multiple boxen) and ccache (multiple boxen not necessary) to speed up configure and compile times. |
|
| Back to top |
|
|
wyv3rn
Apprentice
Joined: 18 Aug 2005
Posts: 154
Location: USA
|
| Posted: Thu Oct 20, 2005 3:44 pm Post subject: |
|
|
| Tomasu wrote: | | Quote: | | If you want fast installations then gentoo is obviously not the distro for you. |
I'm not looking for fast. I'm looking for not slower than what I percived as humanly possible. I happen to like to install most of the base kde packages, because its convienient. As is emerge, when it doesn't take 30 seconds just to load up python and all of its deps, and then a minute to find the package you gave it in portage... Honestly.
edit:
| Quote: | | And you don't have to install all split ebuilds for a package but only those you need, cutting buildtime and disk-usage a lot. |
I also have no problem with the idea of split ebuilds. Had it been implemented correctly, I'd love it. It'd save time, Diskspace, sanity, etc. |
If it can be done so much better put your developer hat on, get to work and show all these inferior devs whos boss. |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Thu Oct 20, 2005 9:44 pm Post subject: |
|
|
| Quote: | | If it can be done so much better put your developer hat on, get to work and show all these inferior devs whos boss. |
I'd love to. Only they don't like my ideas, and its not the users responsibility to fix the mistakes of the developers.
I however did get rather frustrated with emerge taking so long to load and search for packages, so I wrote the following:
| Code: |
#!/usr/bin/perl
$| = 1;
require 5.006_000;
use strict;
use warnings;
use IO::File;
use Getopt::Long;
use File::Spec::Functions;
use Time::HiRes qw/setitimer getitimer ITIMER_REAL ITIMER_VIRTUAL ITIMER_PROF ITIMER_REALPROF/;
our $itwiddle = 0;
our @twiddle = qw(- \\ | - / );
$SIG{ALRM} = sub { local $| = 1; $itwiddle = 0 if $itwiddle >= scalar @twiddle; print $twiddle[$itwiddle++] . "\010"; };
$SIG{VTALRM} = sub { local $| = 1; $itwiddle = 0 if $itwiddle >= scalar @twiddle; print $twiddle[$itwiddle++] . "\010"; };
setitimer(ITIMER_REAL, 0.175, 0.175);
require "sys/ioctl.ph";
our $verbose = 0;
our $installed = 0;
our $spacer = " ";
our $html = 0;
our $onlymasked = 0;
our $listsize = 0;
our $quiet = 0;
our $pattern = 0;
GetOptions(
"html" => \$html,
"s|spacer:s" => \$spacer,
"v|verbose+" => \$verbose,
"i|inst" => \$installed,
"m|masked" => \$onlymasked,
"z|size" => \$listsize,
"h|?|help" => \&help,
"q|quiet" => \$quiet,
"p|pattern" => \$pattern,
);
$spacer = " " if $spacer eq "";
$spacer = substr $spacer, 0, 1;
sub help {
print "usage: $0 [-v] [-m] [-s ' '] <pattern>\n $0 [-v] [-m] [-s ' '] -i\n\tpattern ex: 'kde-*' sys-libs/glibc 'x11-base/*' ...\n";
exit 0;
}
help() if !scalar @ARGV && !$installed;
our $groups = `groups`;
our $whoami = `whoami`;
die "you must be root inorder to run this script :(\n" unless ($groups =~ /root/ || $whoami eq "root");
our $group = $ARGV[0] if scalar @ARGV;
our $pkgdir = '/var/db/pkg';
our %pkgs;
our %masks;
our $max_name_len = 0;
our $masked_color = $html ? "<span class='masked'>" : "\e[1;30m";
our $installed_color = $html ? "<span class='installed'>" : "\e[1;37m";
our $color_end = $html ? "</span>" : "\e[m";
our $total_size = 0;
my $winsize;
my ($row, $col, $xpixel, $ypixel);
if(ioctl(STDOUT, &TIOCGWINSZ, $winsize='')) {
($row, $col, $xpixel, $ypixel) = unpack('S4', $winsize);
}
chdir "/usr/portage";
load_packages_mask();
if($pattern) {
if($group !~ /\*/) {
if($group =~ m|/|) {
my ($g,$p) = ($group =~ s|(.*)/(.*)||);
$group = $g . '/' . $p . '*';
} else {
$group = '*/*' . $group . '*';
}
}
}
if($installed) {
# my $f = new IO::File('/var/lib/portage/world') or die "failed to read world file: $!\n";
# my @pkgs = <$f>;
$group = '*' unless defined $group;
our $pat = ($group =~ m|/|) ? "$group" : "$group/*";
chdir "/var/db/pkg";
scan_pkgdb(glob $pat);
} else {
our $pat = ($group =~ m|/|) ? "$group" : "$group/*";
scan_ebuilds(glob $pat);
}
for (sort keys %pkgs) {
my @list = reverse sort cmp_pkg_version @{$pkgs{$_}{'avail'}};
my $llen = scalar @list;
my $line_len = $max_name_len+1;
if($onlymasked) {
my $ok = 0;
for (my $i = 0; $i < scalar @list; ++$i) {
my $a = $list[$i];
if(masked("${_}-$a") || hard_masked("${_}-$a")) {
$ok = 1;
}
}
next unless $ok;
}
print $_;
my $space = $spacer x ($max_name_len - length($_) + 1);
print $space;
if($quiet) {
print "\n";
next;
}
if($verbose == 2) {
$space .= $spacer x length $_;
}
if($verbose == 2) {
print "\n\t";
print $pkgs{$_}{'desc'};
}
for (my $i = 0; $i < $llen; ++$i) {
next unless $list[$i];
my $a = $list[$i];
my $na = $list[$i+1] if $i+1 < $llen;
my $ul = $_;
my $calcsize = sub {
my $size = package_size($ul . "-" . $a);
$total_size += $size;
return byte_converter($size);
};
if(masked("${_}-$a") || hard_masked("${_}-$a")) {
local $masked_color = ((exists $pkgs{$_}{'inst'}{$a}) ? "[m]" . $installed_color : $masked_color);
if($verbose == 2) {
print "\n$space${masked_color}$a$color_end ";
if(exists $pkgs{$_}{'inst'}{$a} && $listsize) {
my $s = "(" . $calcsize->() . ") " ;
print $s;
$line_len += length $s;
} else { print " "; $line_len++; }
if(!masked("${_}-$na")) {
print("\n$space" . $_) for split("\n",$masks{$_}{'text'});
print "\n";
}
} else {
if($line_len + length($a) + 1 > $col - length($a)) {
print "\n";
print " " x ($max_name_len+1);
$line_len = $max_name_len+1;
} else {
$line_len += length($a) + 1;
}
print "$masked_color$a$color_end ";
if(exists $pkgs{$_}{'inst'}{$a} && $listsize) {
my $s = "(" . $calcsize->() . ") " ;
print $s;
$line_len += length $s;
} else { print " "; $line_len++; }
}
} else {
if($verbose == 2) {
print $installed_color if exists $pkgs{$_}{'inst'}{$a} && $listsize;
print "\n$space" . $a;
if(exists $pkgs{$_}{'inst'}{$a} && $listsize) {
print $color_end;
my $s = " (" . $calcsize->() . ")" ;
print $s;
$line_len += length $s;
} else { print $color_end." "; $line_len++; }
} else {
if($line_len + length($a) + 1 > $col - length($a)) {
print "\n";
print " " x ($max_name_len+1);
$line_len = $max_name_len+1;
} else {
$line_len += length($a) + 1;
}
print $installed_color if exists $pkgs{$_}{'inst'}{$a};
print $a;
if(exists $pkgs{$_}{'inst'}{$a} && $listsize) {
print $color_end;
my $s = " (" . $calcsize->() . ")" ;
print $s;
$line_len += length $s;
} else { print $color_end." "; $line_len++; }
print " " if $line_len + 1 < $col-length($a);
}
}
}
if($verbose == 1) {
print "\n\t";
print($pkgs{$_}{'desc'} . "\n") if exists $pkgs{$_}{'desc'};
print("\t".$pkgs{$_}{'page'} . "\n") if exists $pkgs{$_}{'page'};
print "\n";
} else {
print "\n";
}
}
if($installed) {
my $word = ($onlymasked ? "Listed" : "Installed");
print "Total $word Package Size: " . byte_converter($total_size) . "\n" if !$quiet;
}
sub package_size {
my $pkg = shift;
my $contents = catfile($pkgdir, $pkg, "CONTENTS");
# print "cont: '$contents'\n";
my $f = new IO::File($contents) || return 0;
my $final_size = 0;
for(<$f>) {
/^obj\s+([^\s]+)\s+/ or next;
next if ! -e $1;
my ($size) = (stat(_))[12] * 512; #(stat(_))[7];
$final_size += $size;
}
return $final_size;
}
sub scan_pkgdb {
for my $i (@_) {
chomp $i;
next if !-d $i;
my $qi = quotemeta $i;
my ($g,$p) = split '/', $i;
my ($name,$v) = ($p =~ /^(\w[-\+\/\w]*?)-(\d[.\d]*(?:[-_]?.*)?)$/);
my $qname = quotemeta $name;
# print "group: $g, pkg: $name, v: $v\n";
my $pkg = { 'desc' => '' };
# print "glob: $i/$p.ebuild\n";
my @l = glob("$g/$name-[0-9]*/$name-*.ebuild");
# shift @l while /$qname-\w+/;
my $febuild = $l[0];
my $desc;
my $f = new IO::File($febuild);
my $last = 0;
if($f) {
for(<$f>) {
if(/DESCRIPTION="(.*)"/) {
$pkg->{'desc'} = $1; $last++;
} elsif(/HOMEPAGE="(.*)"/) {
$pkg->{'page'} = $1; $last++;
}
last if $last > 1;
}
}
for(@l) {
s|\.ebuild||;
my @a = split '/';
my ($name,$v) = ($a[2] =~ /^(\w[-\+\/\w]*?)-(\d[.\d]*(?:[-_]?.*)?)$/);
# print "$v\n";
$pkg->{'inst'}{$v} = 1;
$_ = $v;
}
$pkg->{'avail'} = \@l;
my $rname = "$g/$name";
my $len = length $rname;
$max_name_len = $len > $max_name_len ? $len : $max_name_len;
if($quiet) {
$pkgs{$rname."-".$v} = $pkg;
} else {
$pkgs{$rname} = $pkg;
}
}
}
sub scan_ebuilds {
for my $i (@_) {
chomp $i;
next if !-d $i;
my $qi = quotemeta $i;
my ($g,$p) = split '/', $i;
my $qp = quotemeta $p;
my $len = length $i;
$max_name_len = $len > $max_name_len ? $len : $max_name_len;
my $pkg = { 'desc' => '' };
my @l = glob("$i/$p-*.ebuild");
my $febuild = $l[0];
my $desc;
my $f = new IO::File($febuild);
my $last = 0;
if($f) {
for(<$f>) {
if(/DESCRIPTION="(.*)"/) {
$pkg->{'desc'} = $1; $last++;
} elsif(/HOMEPAGE="(.*)"/) {
$pkg->{'page'} = $1; $last++;
}
last if $last > 1;
}
}
s|$qi/$qp-(.*).ebuild|$1| for @l;
$pkg->{'avail'} = \@l;
for(<$pkgdir/$g>) {
if(-d $_) {
my @l = glob("$pkgdir/$i-*");
for (@l) {
/$pkgdir\/$qi-(.*)/ or next;
$pkg->{'inst'}{$1} = 1;
}
}
}
if($quiet) {
$pkgs{$p} = $pkg;
} else {
$pkgs{$i} = $pkg;
}
}
}
#echo -e "\e[1;37m"$i
sub load_packages_mask {
my $f = new IO::File("/usr/portage/profiles/package.mask");
my $text = '';
my $erase_text = 0;
my $lt ='';
my $head = 1;
for my $line (<$f>) {
chomp $line;
if($line =~ /End example/) {
$head = 0;
next;
}
next if $head;
next if $line =~ /^\s*$/;
if($line =~ /^#/) {
$text = '' if $erase_text == 1;
$erase_text = 0;
$text .= $line . "\n";
next;
}
$line =~ s/^(.+)\s*#.*/$1/;
if($line =~ /^([=<>]+)([-\/\w]+)-(.*)$/) {
my @ops = split '', $1;
$masks{$2} = {} if ! exists $masks{$2};
$masks{$2}{$_} = $3 for @ops;
if($text ne '') {
$masks{$2}{'text'} = $text;
$erase_text = 1;
}
} elsif($line =~ /^([-\w]+)-(.*)$/) {
$masks{$2} = {} if ! exists $masks{$2};
$masks{$2}{'all'} = 1;
if($text ne '') {
$masks{$2}{'text'} = $text ;
$erase_text = 1;
}
}
}
}
sub hard_masked {
my $package = shift;
local *F;
$package =~ /^([-\w]+?)\/(([-\+\w]*?)-\d[.\d]*(?:[-_]?.*)?)$/;
#print "1: $1, 2: $2, 3: $3, 4: $4, 5: $5\n";
#print "$1/$3/$2.ebuild\n";
open F, "$1/$3/$2.ebuild" or return 0;
for(<F>) {
if(/^KEYWORDS="-\*"/) {
close F;
return 1;
} elsif(/^KEYWORDS="(.*)"/) {
my $l = readlink "/etc/make.profile";
my @a = split ' ', $1;
for(@a) {
if(/~(.*)/ && $l =~ /$1/) {
close F;
return 1;
}
}
}
}
close F;
return 0;
}
sub masked {
my $package = shift;
my ($name,$v) = ($package =~ /^(\w[-\+\/\w]*?)-(\d[.\d]*(?:[-_]?.*)?)$/);
return 0 if !exists $masks{$name};
return 1 if exists $masks{$name}{'all'} && $masks{$name}{'all'} == 1;
return 1 if exists $masks{$name}{'<'} && cmp_pkg_version2($v, $masks{$name}{'<'}) < 0;
return 1 if exists $masks{$name}{'='} && cmp_pkg_version2($v, $masks{$name}{'='}) == 0;
return 1 if exists $masks{$name}{'>'} && cmp_pkg_version2($v, $masks{$name}{'>'}) > 0;
return 0;
}
sub cmp_pkg_version {
my $pa = $a;
my $pb = $b;
my ($pa_ver,$pa_svern,$pa_sverd) = ($pa =~ m|^([\d.]+)(?:[-_](\w+)(\d+))?|);
my ($pb_ver,$pb_svern,$pb_sverd) = ($pb =~ m|^([\d.]+)(?:[-_](\w+)(\d+))?|);
$pa_svern ||= 'a';
$pb_svern ||= 'a';
$pa_sverd ||= 0;
$pb_sverd ||= 0;
return (
((eval "v$pa_ver cmp v$pb_ver") ||
(($pa_svern cmp $pb_svern) ||
(($pa_sverd <=> $pb_sverd))))
);
}
sub cmp_pkg_version2 {
my $pa = shift;
my $pb = shift;
my ($pa_ver,$pa_svern,$pa_sverd) = ($pa =~ m|^([\d.]+)(?:[-_](\w+)(\d+))?|);
my ($pb_ver,$pb_svern,$pb_sverd) = ($pb =~ m|^([\d.]+)(?:[-_](\w+)(\d+))?|);
$pa_svern ||= 'a';
$pb_svern ||= 'a';
$pa_sverd ||= 0;
$pb_sverd ||= 0;
return (
((eval "v$pa_ver cmp v$pb_ver") ||
(($pa_svern cmp $pb_svern) ||
(($pa_sverd <=> $pb_sverd))))
);
}
sub byte_converter {
my $line = shift(@_);
my $totalmegsize2 = $line;
my $begin;
my $end;
if ($totalmegsize2 > 1125899906842624.0) {
$totalmegsize2 = sprintf "%f", $totalmegsize2 / 1125899906842624.0;
$totalmegsize2 =~ /(.*)\.(.*)/;
$begin = $1;
$end = (substr ($2, 0, 9));
$totalmegsize2 = sprintf("%.2f %s", "$begin.$end", " Petabytes");
} elsif ($totalmegsize2 > 1099511627776.0) {
$totalmegsize2 = sprintf "%f", $totalmegsize2 / 1099511627776.0;
$totalmegsize2 =~ /(.*)\.(.*)/;
$begin = $1;
$end = (substr ($2, 0, 9));
$totalmegsize2 = sprintf("%.2f %s", "$begin.$end", "TB");
} elsif ($totalmegsize2 > 1073741824.0) {
$totalmegsize2 = sprintf "%f", $totalmegsize2 / 1073741824.0;
$totalmegsize2 =~ /(.*)\.(.*)/;
$begin = $1;
$end = (substr ($2, 0, 9));
$totalmegsize2 = sprintf("%.2f %s","$begin.$end","GB");
} elsif ($totalmegsize2 > 1048576.0) {
$totalmegsize2 = sprintf "%f", $totalmegsize2 / 1048576.0;
$totalmegsize2 =~ /(.*)\.(.*)/;
$begin = $1;
$end = (substr ($2, 0, 9));
$totalmegsize2 = sprintf("%.2f %s", "$begin.$end", "MB");
} elsif ($totalmegsize2 > 1024.0) {
$totalmegsize2 = sprintf "%f", $totalmegsize2 / 1024.0;
$totalmegsize2 =~ /(.*)\.(.*)/;
$begin = $1;
$end = (substr ($2, 0, 9));
if(!$begin) {
$totalmegsize2 = sprintf("%.2f %s", "$totalmegsize2", "KB");
} else {
$totalmegsize2 = sprintf("%.2f %s", "$begin.$end", "KB");
}
} else {
$totalmegsize2 = sprintf("%.2f %s", $totalmegsize2, "B");
}
return $totalmegsize2;
}
|
Its far from perfect, and far from being finished, it doesnt respect any of the /etc/portage files etc.
You pass it a package name pattern, in the form of 'category/packagename', which can include wildcards like so: 'category/foo*' or you can just pass a category, and it'll list all of the packages in the category.
Passing -i filters the result through for which packages are installed, instead of which are available, 1 -v gives package info, 2 -v s give package info and possibly the reason why a package is masked (if it can find that info), -m filters the list so it only shows packages which have a masked version, -z tells it to calculate the on disk size of each installed package, and the -p arg turns the passed arg into 'foo/bar*' if you passed 'foo/bar' and 'foo' into '*/*foo*'. |
|
| Back to top |
|
|
aethyr
Veteran
Joined: 06 Apr 2003
Posts: 1085
Location: NYC
|
| Posted: Thu Oct 20, 2005 10:22 pm Post subject: |
|
|
| lost+found wrote: | | lost+found wrote: | Gnome people:
/etc/make.conf
RSYNC_EXCLUDEFROM="/etc/portage/rsync_excludes"
/etc/portage/rsync_excludes
- kde-*/
- metadata/cache/kde-*/ |
| aethyr wrote: | | nevermind, excluding KDE from rsync updates didn't fix it. |
Hmm, I forgot to write it, but to prevent emerge --metadata to calculate the cache again with the outdated stuff: remove the existing /usr/portage/kde-*/ and /usr/portage/metadata/cache/kde-*/ manually. Maybe then it works... It's not a good idea to keep ebuilds that won't be updated in the Portage tree anyway. I cut my /usr/portage by 93%, but included KDE (monolithic): even then my syncs don't hang at 50%. I'm sure the more you exclude, the faster Portage gets. My updating cache takes a few seconds, syncs less than a minute. |
Hmm, it would appear that adding that to rsync_excludes, and then deleting those two directories did fix my problem. Thanks :D |
|
| Back to top |
|
|
stettler
Tux's lil' helper
Joined: 04 Mar 2004
Posts: 113
Location: Germany
|
| Posted: Thu Oct 20, 2005 10:59 pm Post subject: |
|
|
| kev009 wrote: | | Firstly, I wiped /usr/portage and /var/cache/edb. I then reinstalled a portage snapshot and synced but the sync improved only marginally (maybe a minute or two). [...] Next I proceeded to update Python to 2.4.2 and run python-cleaner. My syncs were instantly an order of magnitude faster! |
I already did python-updater before, but wiping out /usr/portage and /var/cache/edb fixed it for me. My sync time also decreased from ~30 min to ~3 min. Hopefully permanently...
Best regards,
stettler
_________________
Claiming that your operating system is the best in the world because more people use it, is like saying McDonalds makes the best food in the world. |
|
| Back to top |
|
|
Gergan Penkov
Veteran
Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)
|
| Posted: Thu Oct 20, 2005 11:30 pm Post subject: |
|
|
| Quote: | | I already did python-updater before, but wiping out /usr/portage and /var/cache/edb fixed it for me. My sync time also decreased from ~30 min to ~3 min. Hopefully permanently... |
But 3 minutes are also too much, a year ago this was also faster.
I don't know what the portage does but import portage takes ages and 11000 records with max 5-6 deps should not be a problem for python.
That's why I always open a terminal with ipython and simply use import portage and let it be simply there,,. after that all subsequent emerges and emerge sync are way faster.
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack |
|
| Back to top |
|
|
stock
n00b
Joined: 22 Nov 2003
Posts: 14
|
| Posted: Fri Oct 21, 2005 11:11 am Post subject: |
|
|
just thought i'd share my experience.
i deleted /var/cache/edb and 'emerge metadata' went from i have no idea how long, to 4m2.494s, which was a huge improvement. out of curiosity, i tried the suggestion of making a 'portage-file', formatted it with reiser 4, symlinked /usr/portage to it (the rest system is using reiser 3.6.x currenly), and now 'emerge metadata' is done in less than 30 seconds. after having not done 'emerge sync' for about 24 hours, i just did it and the total time was 0m57.144s.
_________________
stock @ efnet irc -- http://muffin.homeunix.net |
|
| Back to top |
|
|
Matteo Azzali
Retired Dev
Joined: 23 Sep 2004
Posts: 1133
|
| Posted: Fri Oct 21, 2005 12:40 pm Post subject: |
|
|
I'm running on reiserfs filesystem, runned python-updater,
deleted /var/cache/edb/* and /usr/portage/* , still sync seems more than 4 times slower
than a month ago. Doh!
_________________
Every day a new distro comes to birth. Every day a distro "eats" another.
If you're born distro, no matter what, start to run.
---- http://www.linuxprinting.org/ ---- http://tuxmobil.org/ |
|
| Back to top |
|
|
Decibels
Veteran
Joined: 16 Aug 2002
Posts: 1542
Location: U.S.A.
|
| Posted: Sat Oct 22, 2005 4:28 am Post subject: |
|
|
Well, I have the problem also, the whole emerge --sync took about 10mins, but most of that time was around 50%.
This is after, switching to python 2.4, running python-updater and using /etc/portage/rsync_excludes to eliminate a bunch of packages not using. I was testing the rsync_excludes list by rsync'ing /usr/portage over to my home folder. The list made it drastically shorter in time to archive /usr/portage over to home folder. Thought it might help with emerge --sync, even though it isn't the same exactly as archiving on the local harddrive. But took 10 mins for emerge --sync before doing the exclude list and next day, still took 10 mins.
_________________
http://webpages.charter.net/decibelshelp/
Don't assume malice for what stupidity can explain.
Always remember you're unique. Just like everyone else.
Everybody lies........but it doesn't matter since nobody listens. |
|
| Back to top |
|
|
saturday
Apprentice
Joined: 20 Dec 2004
Posts: 246
Location: de/munich/home
|
|
| Back to top |
|
|
kev009
n00b
Joined: 17 Oct 2005
Posts: 43
Location: Tempe, Arizona
|
| Posted: Sat Oct 22, 2005 8:04 am Post subject: |
|
|
As we all know, the rsync is very expediant. The delays we all experiance come from the portage rebuild of the cache. Question: WHY does portage rebuild EVERYTHING every sync? Why not simply the difference?
_________________
http://www.kev009.com |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Sat Oct 22, 2005 8:07 am Post subject: |
|
|
It helps a little with searching, but the metedata generation gets slower, and python's extesion system seems to be rather unstable. Using anything but the standard file, anydbm or cpickle modules makes emerge rather unstable. Oh, and don't attempt the pysco "tip". It'll crash doing the simplest things.
It makes me wonder why there hasn't been a single decent portage implementation created yet. The C++, C and python versions are all incomplete in some way or another.. MAybe its not possible to create a good implementation? I know its rather hard to deal with hudreds of thousands of files in any case. That also makes me wonder, who decided that the current design was even workable? Did they spend any time at all testing it? Or even think about where certian bottle-necks in the system might lay?
edit:
| Quote: | | As we all know, the rsync is very expediant. |
It takes alot longer to scan and download whatever updates I may need, than to download and unpack an entirely new portage snapshot.
Hardly expedient. And having dead servers in the rotation doesn't help any. |
|
| Back to top |
|
|
alinv
Guru
Joined: 19 Nov 2002
Posts: 395
Location: Bucharest
|
| Posted: Sat Oct 22, 2005 8:27 am Post subject: |
|
|
| Tomasu wrote: | It helps a little with searching, but the metedata generation gets slower, and python's extesion system seems to be rather unstable. Using anything but the standard file, anydbm or cpickle modules makes emerge rather unstable. Oh, and don't attempt the pysco "tip". It'll crash doing the simplest things.
|
Looking at /var/cache/edb/dep/usr/portage, portage is using cpickle for its cache by default.
I've been using anydbm for the last week, and I didn't have any problem with it. As I already said, the sync part is as quick as it should be (about 3 min), but the first emerge -pu world takes 10 minutes. This makes me think that the main issue is in portage and it's not related to the filesystem or other external factors. Of course, I may be wrong, as I didn't look any deeper into this.
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B. |
|
| Back to top |
|
|
Tomasu
n00b
Joined: 01 Jun 2005
Posts: 34
|
| Posted: Sat Oct 22, 2005 9:50 am Post subject: |
|
|
3 minutes to updates say 5 files? Seems a bit much. But then, I've tried several things, and my syncs (with metadata regen) still take a good 10-20 minutes. That is inexcusable. And as far as I can see, nothing has been done about it, nor is anyone even thinking about doing anything about it. (prove me wrong devs, by fixing it) Really, a regular "emerge --help" command takes a good 30 odd seconds to load, that is WAY too much. Especially on a 2ghz machine! it even worse if I decide I want to install something, then it takes a good minute to load the tree and search it..
| Quote: | | I've been using anydbm for the last week, and I didn't have any problem with it. |
Right, I didn't say there was a problem with cpickle or anydbm, infact, if you re read my last post, you'll see those are the ones you won't find a problem with.
Oh, and asking the devs to include a faster module, will fail, they'll eitherflat out say "we aint doin nothin!", or blame it on python (which is where I got my oppinion of python's extension system). |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
