Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
New prelinking approach (typically more effective for IA-32)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Desktop Environments
View previous topic :: View next topic  
Author Message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Wed Sep 06, 2006 11:43 pm    Post subject: New prelinking approach (typically more effective for IA-32) Reply with quote

(You are reading version 1.6 of this guide.)

After the unexpectedly successful attempt to speed up 64 bit Gentoo installations using a modified approach to prelinking (see http://forums.gentoo.org/viewtopic-t-494955-highlight-.html for more details) where multple cases of considerable speed gain have been reported, I was eagerly searching for a similar solution for 32 bit systems.

I did this although it was not clear to me whether such a solution could possibly exist at all. The problem here was the limited virtual address space of the IA-32 architecture.

However, after some experimentation I found a satisfactory result: My script was able to prelink all of my shared libraries to different, unique base addresses.

I wrote that script as a wrapper around the prelink command. The script checks whether my new scheme actually can be applied, and applies it if possible.

If the new scheme does not work for a system, the script includes a fallback to the standard Gentoo prelinking method, which always works but is less efficient. It also remembers that fallback for future runs of the tool, in order to avoid futile reattempts (wasting your time). Once the fallback occurred, my script will be just another way to perform the standard Gentoo prelinking commands.

Which means you can use it anyway: Either it works better than the standard prelink approach, or it does the same.

And even in fallback mode there is a little advantage compared to using the prelink binary directly: Usually you can run the script without any arguments at all (except perhaps for -v / --verbose), because the script will use the right switches for the prelink command automatically.

Using the script, I was able to prelink successfully all shared libraries to unique base addresses on one of my boxes. On this box I use IceWM as a window manager, so there are way fewer SOs installed than on a fully-blown KDE system.

On the other (heavily bloated) box the new approach failed, and so it performed a fallback to the standard Gentoo method.

Interestingly, the new approach actually seemed to work on the bloated box also. But when I checked /etc/prelink.conf, I noticed that for some reason the kde-3.5 libraries were not among the set of search paths used by prelink!

So I added /usr/kde/3.5/lib to PRELINK_PATH in /etc/env.d/99local manually, ran env-update to propagate my changes into /etc/prelink.conf, and eventually re-ran my tool. And only then it failed.

So it seems the Gentoo devs have excluded the KDE libraries from prelinking in the standard configuration.

That is, as long as you do not manually add the KDE libraries to the prelink search path (as I did), my new prelinking scheme will typically also work with KDE systems, despite of the fact that KDE is too large for a 32-bit-processor to be prelinked: KDE will just be skipped when prelinking, as it always has been using the standard Gentoo prelinking guide.

If you actually want to prelink KDE, there are 2 choices for you:
  • Purchase a 64 bit processor and recompile your system in 64 bit native mode, yielding a fully-blown 64-bit Gentoo with multi-exabyte virtual addressing space for every single process. Then you can easily add the KDE-paths to /etc/env.d/99local, and prelink will certainly never run ouf of virtual addresses.
  • Or just stay with 32 bit and also add the KDE-paths to /etc/env.d/99local. But expect fallback to the rather inefficient Gentoo standard method then. On the other hand, a KDE with partial prelinking is still better than a KDE without any prelinking at all. It can never be as effective as on a 64-bit box, but a moderate speed gain can still be expected.


Before finally presenting the latest version of my guide, a big "thank you" goes to all the people in this thread who have contributed to this guide by testing or suggesting valuable additions.

And here's how to use my script:


  • First follow the Gentoo prelink guide (http://www.gentoo.org/doc/en/prelink-howto.xml) except for the Code listing 3.1 in section "Prelink usage".
  • Add the following lines to your /etc/env.d/99local script:
    Code:
    PRELINK_PATH="/usr/local/bin:/usr/local/sbin:/usr/local/opt:/opt"
    PRELINK_PATH_MASK="/opt/doc:/opt/include:/opt/info:/opt/man\
    :/usr/local/opt/doc:/usr/local/opt/include:/usr/local/opt/info\
    :/usr/local/opt/man"

  • If you have a 64-bit system and are also using KDE, add even more paths to the colon-separated list of prelink search paths in /etc/env.d/99local. If you are using kde-3.5, add the following path to PRELINK_PATH in that file: /usr/kde/3.5/lib. (Substitute 3.5 by your KDE version.) Or even better, just do an ls /usr/kde and you can see which KDE versions are actually installed. Add the lib paths for all of them to PRELINK_PATH.
  • Run
    Code:
    env-update
    .
  • Run my script prelink-system (You can find it at the end of this article). Hint: Run the script with the --help option for learning more about its options.
  • Run my script again as
    Code:
    prelink --verbose
    each time your libraries might have been updated, such as after an emerge. This will prelink any new/updated libraries also.


I have also written a second script show-writable-code-segments which lists all writable code segments which currently exist in memory.

Based on the naive assumption that relocated libraries might remain writable after they have been relocated, this would then print a list of loaded libraries where prelinking was not effective.

Which means, the shorter that list is, the better.

However, I am not sure how correct this naive assumtion is. If the dynamic loader resets the memory protection for relocated code pages back to read-only after relocation, this won't work. If you know more about this, please let me know.

If you want to revert what my script has done, run the following command:
Code:
prelink -au && prelink -amR

This should restore the default behaviour of the Gentoo prelink guide.

Nevertheless, here are both scripts:

  • Listing "/usr/local/sbin/prelink-system":
    Code:

    #!/bin/sh
    #
    # Prelinks system or updates existing prelinks.
    #
    # $HeadURL: /caches/xsvn/trunk/usr/local/sbin/prelink-system $
    # $Author: root $
    # $Date: 2006-09-08T20:44:13.264408Z $
    # $Revision: 363 $
    #
    # Written by Guenther Brunthaler in 2006.


    # Change this to anything you like.
    STATE_FILE="$HOME/.prelink-system.state"


    usage() {
       cat <<- "."
       prelink-system - apply prelinking to the local system
       
       Usage: prelink-system [ options ]
       
       options:
       --full, -f: Process all objects. SLOW.
       --incremental, -i: Process only new/updated objects.
       --help, -h: Display this help text.
       --verbose, -v: Verbose operation.
       --dry-run, -n: Simulated run. Shows what would be done.
       --quiet, -q, --silent:
           Don't output anything.
           Only the return code will indicate success or failure.
           No warnings will be displayed at all.
          
       prelink-system is a wrapper script for the "prelink" command.
       If applies the best prelink options and parameters for your system.
       
       Prelinking can make your applications start considerably faster,
       especially if heavy-weight desktop environments like KDE are used.
       
       prelink-system can operate in two different modes, "full" and
       "incremental", as specified using the command line options. The
       default mode is "incremental", unless if run for the first time,
       when "full" will be the default.
       
       In full mode, each and every library and executable within prelink's
       search path will be processed. This will take a long time, but will
       will be done thoroughly. Because of its long running time, full mode
       should only be used once in a month or so, or after reaching some sort
       of "system installation milestone", such as recompiling large parts of
       the system.
       
       In incremental mode, only new or updated libraries and executables
       within prelink's search path will be examined, and thus there may be a
       very small chance for doing things not as optimal as in full mode.
       
       However, incremental mode is much faster than full mode, and can
       therefore be run after every "emerge" installation operation or on a
       daily basis without performance concerns.
       
       Note: The effect of prelinking will only be as good as the settings of
       PRELINK_PATH and PRELINK_PATH_MASK which can be customized in your
       /etc/env.d/99local configuration file.
       
       I suggest adding the following lines to that configuration files,
       unless you know what you are doing and see a stringent reason not to
       do so:

       ---cut here---   
       PRELINK_PATH="/usr/local/bin:/usr/local/sbin:/usr/local/opt:/opt"
       PRELINK_PATH_MASK="/opt/doc:/opt/include:/opt/info:/opt/man\
       :/usr/local/opt/doc:/usr/local/opt/include:/usr/local/opt/info\
       :/usr/local/opt/man"
       ---cut here---

       Don't forget to run env-update after any change to 99local, or
       otherwise your changes won't have any effect.
       
       Version 1.1
       Written by Guenther Brunthaler in 2006.
    .
    }


    die() {
       {
          echo "ERROR: $*"
          echo "Use $0 --help for help."
       } >& 2
       exit 1
    }


    warn() {
       echo "WARNING: $*" >& 2
    }


    seh_setup_check() {
       local ME; ME="$0.$$"
       test "$SEH_INIT" = "$ME" && return
       SEH_INIT="$ME";   SEH_DTORS=
       trap seh_process 0
    }


    seh_process() {
       local DTOR TAIL
       while [ -n "$SEH_DTORS" ]; do
          TAIL="${SEH_DTORS#*:}";   DTOR="${SEH_DTORS%$TAIL}"
          SEH_DTORS="$TAIL"; DTOR="${DTOR%:}"
          "$DTOR" || warn "Clean-up function $DTOR() failed!"
       done
    }


    finally() {
       seh_setup_check; SEH_DTORS="$1:$SEH_DTORS"
    }


    load_state() {
       if [ ! -e "$STATE_FILE" ]; then
          STATE="initial"
          return
       fi
       STATE=
       read STATE < "$STATE_FILE" 2> /dev/null
       test -n "$STATE" || die "Failed reading state from '$STATE_FILE'!"
       LAST_STATE="$STATE"
    }


    save_state() {
       test "$STATE" = "$LAST_STATE" && return
       echo "$STATE" > "$STATE_FILE" || \
          die "Could not write state file '$STATE_FILE'!"
    }


    inform() {
       test -z "$VERBOSE" && return
       echo $*
    }


    do_prelink() {
       if [ -n "$DRY" ]; then
          echo "# prelink $*"
          return
       fi
       prelink "$@" || die "Could not prelink $*"
    }


    out_of_addresses() {
       echo "Out of virtual address space. Full prelinking impossible."
       echo "(Seems it's time for you to purchase some 64 bit processor?)"
       echo "Reverting to inefficient standard prelinking scheme."
       echo "(Still better than no prelinking at all.)"
       inform "Undoing current prelinkage..."
       STATE=conservative
       do_prelink --undo --all
       inform "Prelinking the conservative standard way..."
       inform "(This will be remembered for future sessions.)"
       do_prelink --force --all --random --conserve-memory
    }


    FULL=
    VERBOSE=
    QUIET=
    DRY=
    load_state
    if [ "$STATE" = "initial" ]; then
       FULL=1;   VERBOSE=1
    fi
    COPTS=
    while true; do
       if [ -z "$COPTS" ]; then
          case "$1" in
             -?*) COPTS="$1"; shift;;
             *) break;;
          esac
       fi
       if [ "${COPTS#--}" = "$COPTS" ]; then
          TAIL="${COPTS#-?}"; # Switch custering.
          COPT="${COPTS%$TAIL}"; COPTS="${TAIL:+-}$TAIL"
       else
          COPT="$COPTS"; COPTS=
       fi
       case "$COPT" in
          --) break;;
          --help | -h) usage; exit;;
          --version)
             # Sychronize this with the usage text!
             echo "Version 1.1"
             exit;;
          --verbose | -v) VERBOSE=1;;
          --dry-run | -n) DRY=1;;
          --quiet | --silent | -q) QUIET=1;;
          --full | -f) FULL=1;;
          --incremental | -i) FULL='';;
          *) die "Unknown option '$COPT'!";;
       esac
    done
    test $# = 0 || die "Unexpected excess arguments: $*"
    if [ -n "$QUIET" -a -z "$DRY" ]; then
       CMD="${FULL:+--full}"
       exec "$0" "${CMD:---incremental}" > /dev/null 2>& 1
    fi
    test -z "$DRY" && finally save_state
    if [ -n "$FULL" ]; then
       inform "Full system prelink phase 1 - undoing old prelinkage..."
       do_prelink --undo --all
       inform "Full system prelink phase 2 - prelinking binaries..."
       do_prelink --force --all --conserve-memory
       inform "Full system prelink phase 3 - prelinking libraries..."
       if prelink --force --libs-only --all --random; then
          STATE=flat
       else
          out_of_addresses
       fi
    else
       inform "\"Prelinking\" (relocating) new/updated libraries..."
       if [ $STATE == conservative ]; then
          inform "(Using low-efficiency conservative standard mode.)"
          do_prelink --all --random --conserve-memory
       else
          inform "(Using full-efficiency mode.)"
          if prelink --libs-only --all --quick --random; then
             STATE=flat
          else
             out_of_addresses
          fi
       fi
    fi
    inform "Prelinking complete."

  • Listing "/usr/local/bin/show-writable-code-segments":
    Code:

    #! /usr/bin/perl -w
    #
    # $HeadURL: /caches/xsvn/trunk/usr/local/bin/show-writable-code-segments $
    # $Author: root $
    # $Date: 2006-09-06T19:21:09.348653Z $
    # $Revision: 356 $
    #
    # Written in 2006 by Guenther Brunthaler.


    use strict;


    {
       package File;
       
       
       # Create a new file handle container object.
       sub new {
          my $self= shift;
          $self= bless {} if !ref $self || ref $self eq __PACKAGE__;
          return $self;
       }
       
       
       # Close any open file.
       DESTROY {
          my $self= shift;
          my $fh= $self->{fh};
          return unless $fh;
          undef $self->{fh};
          close $fh or die "Could not close file '$self->{filename}': $!";
       }
       

       # Open a new file for reading.
       # If already in use, the old handle will be closed first.
       # Returns the new file handle.
       sub open {
          my($self, $filename)= @_;
          DESTROY;
          local *FH;
          open FH, '<', $filename or die "Cannot open '$filename': $!";
          $self->{filename}= $filename;
          return $self->{fh}= *FH{IO};
       }
    }


    my($pid, $fh, $cmd);
    my $fobj= new File;
    opendir PROC, '/proc' or die $!;
    while (defined($_= readdir PROC)) {
       next unless /^(\d+)$/;
       $pid= $1;
       $fh= $fobj->open("/proc/$pid/maps");
       while (defined($_= <$fh>)) {
          next unless /
             ^ [[:xdigit:]]{4,} - [[:xdigit:]]{4,} \s+
             [-r] ( [-w] [-x] ) [sp] \s+ [[:xdigit:]]{4,} \s+
             [[:xdigit:]]{2,} : [[:xdigit:]]{2,} \s+ \d+ \s+
             (.+?) \s* $
          /x;
          $cmd= readlink("/proc/$pid/exe") || "(process $pid)";
          chomp;
          print "$_ PID($pid) $cmd\n" if $1 eq 'wx';
       }
    }
    closedir PROC or die;



And finally, a few words about my the basic ideas behind my script:

  • In order for prelinking to be effective, all binaries and libraries should be assigned a different, unique address range.
  • This will allow the loader to just map the code segments of those binaries and libraries into the address space of a process and running them without any changes. That is, no relocations are necessary.
  • It will also allow sharing those mapped pages between processes, because it's always the same data, mapped at the same virtual address within all address spaces.
  • This will enable you, for instance, to start 100 copies of your favorite KDE word processor, but there will only be a single instance of the word processor's code pages in memory, which will be shared among all those 100 instances.
  • Relocation is never an issue for binaries (I am using "binaries" in this context when I am referring to executables other than shared libraries) which use SOs, just for SOs themselves. This is because all binaries are typically loaded at the same address anyway. Which will not be a problem, because initially each binary is "alone" in the address space of its process, so no address conflicts can occur at all before SOs are to be loaded.
  • In contrary to 64 bit processors where it is not a problem to assign each and every binary and library in the system its unique virtual address space range, 32 bit processors have a rather limited overall virtual address space size. So it's not possible to map each binary and library to its own, unique memory range, because we will be running out of virtual addresses then.
  • However, as you may have noted from what I wrote first, it is not actually necessary to assign each binary and library its own address range: It it sufficient if only all the libraries are assigned different address ranges. It does not hurt if the binaries use conflicting address ranges (among all of the binaries), as long as the do not conflict with any of the SOs. (It will not hurt because normally there is only a single binary in every address space. But there will multiple SOs be mapped into the same address space.)
  • So, the first important thing my script does is relinking all objects, binaries as well as libraries, using the --conserve-memory switch of prelink. This may be suboptimal for libraries, but is good enough for binaries and will allow the binaries to share the same address space ranges. I also disable address randomization in this phase to make it more likely for each binary to get the same base address.
  • After that, a second prelinking pass is performed. This time, only the libraries are relocated, and now without the --conserve-memory switch. This should assign a unique address range to each library, but leave the binaries relocated as the are. Address randomization is also enabled, because it's a nice security feature and won't hurt in this phase neither.
  • If we are lucky, the total code requirements of all SOs will still fit into the 2 or 3 gigabyte address range available to 32 bit processes, without wasting precious virtual address space for also assigning unique address ranges to binaries.
  • The successfulness of this approach is therefore subject to the question, whether the combined code segment sizes of all SOs in the system exceeds 2 or three gigabytes, or not.
  • This might not be the case on my box, so it seems to works.
  • But more testing on different systems is needed in oder to see whether this also works as well on other installations. Which is why testing is required.


Last edited by Guenther Brunthaler on Mon Sep 11, 2006 2:27 pm; edited 6 times in total
Back to top
View user's profile Send private message
zxy
Veteran
Veteran


Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer

PostPosted: Fri Sep 08, 2006 3:32 pm    Post subject: Reply with quote

Interesitng. I'm gonna check it on my laptop as soon as possible.

I got this (3rd phase)
Code:

...
prelink: /usr/local/bin/bjfilteri255: Could not find one of the dependencies
prelink: /opt/cxoffice/bin/wineserver: Could not find one of the dependencies
prelink: Could not find virtual address slot for /usr/lib/openoffice/program/libsfx680li.so
ERROR: Could not prelink --force --libs-only --all --random
Use /usr/local/sbin/prelink-system --help for help.


on pentium-m 512Mb RAM, Kde, mozilla-*, openoffice
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 08, 2006 7:12 pm    Post subject: Reply with quote

zxy wrote:
prelink: Could not find virtual address slot for /usr/lib/openoffice/program/libsfx680li.so


Thank you very much! This was exactly what I was waiting for: Now I know that prelink does not silently fail if it is running out of virtual addresses, but actually gives an error.

With that knowledge, I was able to add fallback code to my script, which reverts to the conventional

Code:
prelink -avm


method of the Gentoo Prelink guide in such cases. My article now contains the updated script.

Unfortunately for you, that also means my script cannot help you: The combined size of all your shared objects alone (not counting the remaining binaries) exceeds the maximum virtual address size provided by Linux to 32 bit processes; so there is no way of assigning a different address range to each and every shared library.

In order to revert your system to the previous prelinking stage, execute the following commands:

Code:
prelink -au && prelink -afvm


This should restore the prelinkage state of your system to that of the standard Gentoo prelinking guide.

It seems, that you are actually one of those Gentoo users who actually needs a 64 bit processor which its large virtual address space. Then prelinking will no more be a problem. (As it seems, prelinking is the killer argument for purchasing a 64 bit processor. Forget about it's wider integers. Forget about it being faster. The mere fact that it allows unrestricted prelinking can start up apps like KDE faster by a factor of 2, and also save a lot of RAM because code pages can be shared to a much higher degree between processes.)

However, there may perhaps still exist a solution to your 32-bit box: I know the Linux kernel provides some choices about the memory layout, which directly affects how much virtual addressing space is available to applications.

So, if you are lucky and use one of the other memory models, the increased virtual address space might be large enough for all your libraries to be prelinked with my script without having to return to the rather inefficient standard settings.

Anyway, thank you for your help. I will update my script in such a way that it reverts to the standard settings automatically if an error occurs like you have encountered one.
Back to top
View user's profile Send private message
zxy
Veteran
Veteran


Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer

PostPosted: Fri Sep 08, 2006 9:40 pm    Post subject: Reply with quote

After the error, I didn't automaticly revert to old prelink, but I tried how what has been done works. Result is much better than before. It at least fels faster.

This machine really needs a cleanup. So when it is done new prelink might still work.

I'm gonna try the script on another 32bit machine soon. That one is just brought up, so there is not much unnececary apps there. (Just wait for OO to compile)
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu
Back to top
View user's profile Send private message
zxy
Veteran
Veteran


Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer

PostPosted: Sat Sep 09, 2006 3:46 pm    Post subject: Reply with quote

On the other machine with kde, oo, firefox, and thunderbird prelink went ok. The difference is that this machine has less bloat. I hope to get the first one to prelink too, just a cleanup is needed.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Sep 09, 2006 9:47 pm    Post subject: Reply with quote

zxy wrote:
After the error, I didn't automaticly revert to old prelink, but I tried how what has been done works. Result is much better than before. It at least fels faster.


The best approach would certainly be prioritizing shared libraries, so prelink should assign those libraries unique addresses first.

That way the important libraries would never have to be relocated, while the less important ones may occasionally encounter address conflicts leading to relocation.

The question is only: How to gather that information? There are so many SOs on a typical system; it's too much effort specifying it manually.

I think the best solution would be some sort of prelink-monitor.

For instance, the entry points of the dynamic loader could be hooked in order to gather statistics, which SOs are loaded how frequently, and how long they remain to be loaded.

This could then be used as a basis for some statistical evaluation.

Unfortunately is also looks like a rather large amount of work to do, and as I'm rather lazy I assume I'd prefer purchasing a 64 bit processor instead... :wink:
Back to top
View user's profile Send private message
Small_Penguin
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2005
Posts: 119

PostPosted: Tue Feb 20, 2007 6:32 pm    Post subject: Reply with quote

Thanks for the script, it worked very nice for a long time. However, lately it switched to low-efficient mode. My question: Is it possible to completely revert prelinking and restore full-efficiency mode (after removing some dirs or packages)?

Btw: I noticed that some time ago during a prelink update a prelink script was added to my cron jobs. Does this interfere with your prelink-system script or doesn't it matter at all? I've removed it but think it has already been executed once or twice. Fortunately, I didn't experience any negative effects.

I enjoy your work, recompile-entire-system was very helpful too!
Back to top
View user's profile Send private message
Small_Penguin
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2005
Posts: 119

PostPosted: Tue Feb 20, 2007 6:37 pm    Post subject: Reply with quote

Quote:
So, if you are lucky and use one of the other memory models, the increased virtual address space might be large enough for all your libraries to be prelinked with my script without having to return to the rather inefficient standard settings.


Maybe I'll try this too. You're talking about the High Memory support option (1GB/4GB/64GB), don't you? My config is currently set at 4GB, I could try 64GB and see if it improves anything.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Thu Feb 22, 2007 7:55 pm    Post subject: Reply with quote

Small_Penguin wrote:
Is it possible to completely revert prelinking and restore full-efficiency mode (after removing some dirs or packages)?


Yes. Just run

Code:
prelink-system --full


This will undo any previous prelinkage in its first phase, and then retry full-efficiency mode.

Small_Penguin wrote:
Btw: I noticed that some time ago during a prelink update a prelink script was added to my cron jobs.


Yes, I noted this too and just modified my copy of that newly installed cron file to do just nothing. (Of course I could just have deleted it, but making it a no-op script instead has the advantage that future reinstallations won't recreate the original cron file, thanks to Gentoo's configuration file protection.)

Small_Penguin wrote:
Does this interfere with your prelink-system script or doesn't it matter at all?


While there are cannot be any catastrophic consequences when using that cron job together with my script, I do not recommend actually doing this.

The problem are the different strategies of the approaches:

The cron job of the prelink package always runs prelink with command line switches which conform to my "conservative" (low-efficiency) mode which uses shared .so base addresses.

While this is a safe approach, it will not have the advantages of my script's full-efficiency mode with its unique .so base addresses.

Regarding the usage of prelinking in cron jobs from a conceptual point of view, I prefer running prelink-system manually after emerging some package, because there is little use updating .so file base addresses when no new .so files have been installed at all...

But if a cron job is desired, it should not be a problem running my script from within a cron job instead.

However, care should be taken to avoid running multiple instances of my script or the underlying "prelink"-command at the same time, as this might result in race conditions in the prelink algorithm. (I have not checked whether this is actually true. Perhaps "prelink" uses lock files internally and is thus safe to be run at any time. But I would not count on it.)
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Thu Feb 22, 2007 8:20 pm    Post subject: Reply with quote

Small_Penguin wrote:
You're talking about the High Memory support option (1GB/4GB/64GB)


Uhm, 64GB mode... cool. Did not actually know this!

But I'm afraid, a 64GB mode won't help prelink on a 32 bit processor, because the 32 bit limit the virtual address space of any process to at most 4 GB. No matter how much RAM is actually installed and is available as a total to different processes.

And as prelink operates on virtual addresses, no memory model beyond 4 GB can actually help prelink.

What I actually wanted to refer to in my original posting were the different kernel options how to partition the kernel - user space address assignment. That's the "Memory model" options in the "Processor type and features" section of the kernel configuration.
Back to top
View user's profile Send private message
ideler
n00b
n00b


Joined: 23 Feb 2007
Posts: 1

PostPosted: Fri Feb 23, 2007 3:29 am    Post subject: Reply with quote

I have got a largish desktop box (gnome + kde + misc other stuff) that would fail with prelink -aR.
But ever since I compiled the 2.6.19 kernel with "64 bit Memory and IO resources (EXPERIMENTAL)" option turned on, prelink succeeds with these parameters.
Now "High Memory support" set to 64GB forces that same CONFIG_RESOURCES_64BIT flag to be set, so it should have the same effect plus enabling Intel PAE switching.

Now it is true I did not test is syntetically, and rather on the live desktop system, but I doubt my libs shrunk suddenly to below 4gb.

I am using farly recent 32 bit dual core pentium 4 chip, so it must support PAE. Although setting CONFIG_RESOURCES_64BIT seems sufficient.

Also konqueror goes down from 0.1 second real load time to 0.01s. It's 10 times faster when prelinked.
Back to top
View user's profile Send private message
Small_Penguin
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2005
Posts: 119

PostPosted: Sat Feb 24, 2007 1:51 pm    Post subject: Reply with quote

Quote:
prelink-system --full

Thanks, that worked fantastically.

Quote:
Yes, I noted this too and just modified my copy of that newly installed cron file to do just nothing. (Of course I could just have deleted it, but making it a no-op script instead has the advantage that future reinstallations won't recreate the original cron file, thanks to Gentoo's configuration file protection.)

Good idea.

Quote:
But ever since I compiled the 2.6.19 kernel with "64 bit Memory and IO resources (EXPERIMENTAL)" option turned on, prelink succeeds with these parameters.
Now "High Memory support" set to 64GB forces that same CONFIG_RESOURCES_64BIT flag to be set, so it should have the same effect plus enabling Intel PAE switching.

Confirmed. I've enabled the experimental 64bit memory and IO resources option, and prelink-system now runs in full efficient mode. Thanks.
Back to top
View user's profile Send private message
drizzt
Guru
Guru


Joined: 21 Jul 2002
Posts: 337

PostPosted: Wed Mar 07, 2007 4:42 pm    Post subject: Reply with quote

To answer a question about virtual-address space on 64Bit:

Code:
address sizes   : 40 bits physical, 48 bits virtual


This output from
Code:
cat /proc/cpuinfo
should say everything which was unanswered before.
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...
Back to top
View user's profile Send private message
VoVaN
l33t
l33t


Joined: 02 Jul 2003
Posts: 666
Location: The Netherlands

PostPosted: Thu Mar 15, 2007 8:33 am    Post subject: Reply with quote

I'm using 32bits Gento and CONFIG_RESOURCES_64BIT is not set. Below is my prelink.conf file:
Code:
cat prelink.conf
# prelink.conf autogenerated by env-update; make all changes to
# contents of /etc/env.d directory
-l /bin
-l /sbin
-l /usr/bin
-l /usr/sbin
-l /lib
-l /usr/lib
-h /usr/local/lib/
-h //usr//lib/opengl/nvidia/lib/
-h /usr/i686-pc-linux-gnu/lib/
-h /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/
-h /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/
-h /usr/lib/nspr/
-h /usr/kde/3.5/lib/
-h /usr/qt/3/lib/
-h /usr/games/lib/
-h /usr/lib/libstdc++-v3/
-h /opt/bin/
-h /usr/i686-pc-linux-gnu/gcc-bin/4.1.2/
-h /usr/kde/3.5/bin/
-h /usr/qt/3/bin/
-h /usr/games/bin/
-b /lib/modules
-b /usr/lib/locale
-b /usr/lib/wine
-b /usr/lib/valgrind
-b *.la
-b *.png
-b *.py
-b *.pl
-b *.pm
-b *.sh
-b *.xml
-b *.xslt
-b *.a
-b *.js

When I'm doing 'prelink -au && prelink -afR' and not getting any errors. Does that mean it works fine for me? How I can check that everything is fine. Normally in order to check if binary is prelinked I use
Code:
LD_DEBUG=statistics kate --help
      9085:
      9085:     runtime linker statistics:
      9085:       total startup time in dynamic loader: 3832766 clock cycles
      9085:                 time needed for relocation: 442827 clock cycles (11.5%)
      9085:                      number of relocations: 0
      9085:           number of relocations from cache: 454
      9085:             number of relative relocations: 0
      9085:                time needed to load objects: 3034150 clock cycles (79.1%)
      9085:
      9085:     runtime linker statistics:
      9085:                final number of relocations: 0
      9085:     final number of relocations from cache: 454

According this output it works for binaries on my machine just fine, but how I can check libraries and shoud I do that?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Desktop Environments All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum