Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to get preload works with gentoo
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 3:45 pm    Post subject: How to get preload works with gentoo Reply with quote

http://en.opensuse.org/SUPER_preloading
http://download.opensuse.org/repositories/home:/dirkmueller:/playground:/gcc43/gcc43_Factory/src/preload-0.2-109.7.src.rpm
http://download.opensuse.org/repositories/home:/dirkmueller:/playground:/gcc43/gcc43_Factory/i586/preload-0.2-109.7.i586.rpm

I want this things in gentoo
Is there any chance?
I wanna only faster start application.(1st time, because next time are ok)
Back to top
View user's profile Send private message
pussi
l33t
l33t


Joined: 08 May 2004
Posts: 727
Location: Finland

PostPosted: Sat Sep 29, 2007 4:07 pm    Post subject: Reply with quote

there's readahead-list in portage which is quite similar
http://gentoo-wiki.com/HOWTO_prefetch_files_on_boot
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 5:29 pm    Post subject: Reply with quote

readhead doesnt optimise e.g firefox start :P
Back to top
View user's profile Send private message
pussi
l33t
l33t


Joined: 08 May 2004
Posts: 727
Location: Finland

PostPosted: Sat Sep 29, 2007 6:11 pm    Post subject: Reply with quote

sorry, I actually thought it could be configured to do that.
you could simply try download the preloading source and compile it manually
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 6:59 pm    Post subject: Reply with quote

preloading source? what is it?
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, overlooking Dublin's docklands

PostPosted: Sat Sep 29, 2007 8:22 pm    Post subject: Reply with quote

seems like nothing more than a shell script, you should find it on the suse ftp... or maybe it's linked from somewhere in there...
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
pussi
l33t
l33t


Joined: 08 May 2004
Posts: 727
Location: Finland

PostPosted: Sat Sep 29, 2007 8:25 pm    Post subject: Reply with quote

matiqing wrote:
preloading source? what is it?

source code for that preload application
probably this:
http://download.opensuse.org/repositories/home:/dirkmueller:/playground:/gcc43/gcc43_Factory/src/preload-0.2-109.7.src.rpm
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 8:47 pm    Post subject: Reply with quote

yes i know that but how to use that :/ ?
Back to top
View user's profile Send private message
jlh
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2007
Posts: 144
Location: Switzerland::Zürich

PostPosted: Sat Sep 29, 2007 8:56 pm    Post subject: Reply with quote

There are multiple ways to improve load times. One is to just keep portions of programs in memory, waiting to be used. OpenOffice.org quickstarter probably does that and similar things are available for firefox and probably other apps. I consider this approach to be bad, because for the entire time you do not use those applications, it just uses up precious RAM that could be used for regular file caching. While it speeds up some things, it slows down others. If at all, I think on-demand preloading is the way to go, so whenever a process starts, something will immediately start loading files that are likely to be requested by that application soon; so when the actually request comes, they're already ready. Windows XP does that and it's a pretty cool thing.

I once tried my luck to write something like that, based on the preload idea, but I didn't get anywhere. After various approaches, everything was starting slower, not faster; the bottom line for me was that the preload-trick is just not powerful enough and has no way to catch all file accesses. However, if I remember correctly there is some work being done at kernel level for linux (and that's probably the only sane level to do this at), but I completely forgot its name or its webpage. I have some vague association with google, so maybe it was a GSoC, or it's hosted by google. Anyone knows what that is?
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 9:00 pm    Post subject: Reply with quote

jlh
But on openSUSE it is working pretty good.
Firefox is starting 2s!!...
i like that!
Please somebody help me with it...
Back to top
View user's profile Send private message
jlh
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2007
Posts: 144
Location: Switzerland::Zürich

PostPosted: Sat Sep 29, 2007 9:01 pm    Post subject: Reply with quote

Yes, but I bet you used static preloading to get this result. I don't like that: It makes firefox faster and everything else slightly slower.

Of course it makes a lot of sense if you open (and close) firefox a lot.
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 9:08 pm    Post subject: Reply with quote

jlh
You say about openSuse's method?
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, overlooking Dublin's docklands

PostPosted: Sat Sep 29, 2007 9:12 pm    Post subject: Reply with quote

jlh: yeah something like "kernel based readahead"?

it shouldn't make anything slower, only if it's actually reading while other things would need to read something. from my understanding it's just pre-filling the disk cache with things that are likely to be loaded, probably just using ldd to find out which libs will be used by firefox and then using readahead to tell the kernel to load those files.

it shouldn't actually hurt performance of other apps: if they really do need the ram, then the disk cache will just get smaller and some stuff that was read ahead will be dropped. usually the linux disk cache will simply keep growing anyway, but it's empty at first; telling it what could be needed in advance is nothing bad and shouldn't hurt performance...
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sat Sep 29, 2007 9:30 pm    Post subject: Reply with quote

any concrects?
jlh
You say about openSuse's method?
Quote:
keep portions of programs in memory, waiting to be used.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sun Sep 30, 2007 6:35 am    Post subject: Reply with quote

The openSuse method of getting running files for me broke quite horribly, using strace like that is.. a tad bit abusive (and put all my processes in HALT state). So I thought I'd do it a cleaner way, by hooking open and looking at all the files actually opened.

Saw someone had already done that (opentrace.c from http://gentoo-wiki.com/HOWTO_prefetch_files_on_boot ), although that broke for me as well, as init wasn't called on all processes. So after slight modification this should work:

Code:

/* relevant includes, the define is needed on Linux
 * but doesn't break it for FreeBSD
 */

#define _GNU_SOURCE
#include <dlfcn.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdarg.h>
#include <stdlib.h>
#include <stdio.h>

/* mark init() to be executed on library load
 */
static void init (void) __attribute__ ((constructor));
static void onExit (void) __attribute__ ((destructor));

/* pointer to the real write call */
static int (*old_open)() = 0;

static int oFD;
static int init_called = 0;
/* any initialization code here (reading configfiles, whatever)
 */
static void init (void)
{
      if (init_called == 1)
         return;
      init_called = 1;
      
        char* oFile;
        /* find the 'real' write, we might need it later */
        old_open= (int (*)()) dlsym(RTLD_NEXT, "open");
        if (old_open== NULL)
        {
                fprintf(stderr, "dlsym: %s\n", dlerror());
                exit(1);
        }

        oFile = "/tmp/open_trace.log";

        if (oFile)
                oFD = old_open (oFile, O_CREAT|O_WRONLY|O_APPEND, 0622);
        else
                oFD = 2;
}

static void onExit (void)
{
        close (oFD);

}

/* grab the open() library call
 * hijacking a function is as easy as creating one with the
 * same name, since we're the last library loaded
 */
int open (__const char *file, int oflag, ...)
{
      init();
        va_list ap;

        va_start (ap, oflag);

        /* do funky stuff */
        write (oFD, file, strlen(file));
        write (oFD, "\n", 1);
        fsync(oFD);

        /* call the real open function */
        if (oflag & O_CREAT)
                return old_open(file, oflag, va_arg(ap,mode_t));
        else
                return old_open(file, oflag);
        va_end(ap);
}


gcc -fpic -nodefaultlibs -shared opentrace.c -ldl -o opentrace.so && LD_PRELOAD=/full/path/to/opentrace.so firefox

That should put all firefox'es open's in /tmp/open_trace.log, look at the file, do some grep magic (rip out "." ".." "/dev/*" and stuff like that, and you should have a file you can put in the readahead lists. And you should get firefox preloaded during boot.

I might toy with making a program that runs as init (and for example dumps file on kill signal), should be a very accurate way of doing it.
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sun Sep 30, 2007 7:10 am    Post subject: Reply with quote

ok
But how to made it like in suse? :P
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Sun Sep 30, 2007 3:20 pm    Post subject: Reply with quote

is there any chance to do it like in openSUSE!!?
Back to top
View user's profile Send private message
Naib
Advocate
Advocate


Joined: 21 May 2004
Posts: 4200
Location: Removed by Neddy

PostPosted: Sun Sep 30, 2007 9:04 pm    Post subject: Reply with quote

matiqing wrote:
is there any chance to do it like in openSUSE!!?


readahead *can* do what you are after
If you are basically after FF starting in 2sec then edit the readahead list to add firefox-bin and the libs it will need
This will then make FF cached in RAM when you boot to give you yr 2sec first-boot experience
_________________
A free press is the unsleeping guardian of every other right that free men prize; it is the most dangerous foe of tyranny. Where men have the habit of liberty, the Press will continue to be the vigilant guardian of the rights of the ordinary citizen.
Back to top
View user's profile Send private message
jlh
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2007
Posts: 144
Location: Switzerland::Zürich

PostPosted: Sun Sep 30, 2007 9:22 pm    Post subject: Reply with quote

I've tried the strace approach and I've tried the LD_PRELOAD technique and they're both not-so-good. The reason for this is that those methods only allow you to catch library calls, like open(2), read(2), etc. This is not sufficient because *a lot* of binaries and libraries use mmap(2) to map files into their address space and accessing their content without any further library calls. And you cannot possibly track what is being read from those files using any of the above methods. You still know which files are being accessed (via open(2)), but you cannot find out which parts and how much of the file is being loaded (if any at all). So either you don't preload those files at all, and you partially miss the purpose of this entire idea, or you preload the entire file, potentially loading a lot of unnecessary data, which slows things down. That's why I say that the kernel is the only sane place to do things like this, because it's the only instance that can really know what a process is doing.

And I also have some doubt that all this is worth the trouble if you don't combine it with some sort of block re-ordering on the hard disk. The thing that's slow while starting an application isn't that the system is waiting for the application to make I/O requests; and merely knowing those requests in advance won't make anything faster. The thing that's slow is the reading process itself because the hard disk's head is jumping over the entire disk. If you merely preload the files without having them re-ordered on your disk, then you cause that same jumping around with readahead(2) instead of read(2) and you get no improvement. If you re-order your blocks, the hard disk can read continuous blocks and *then* you get an improvement. (Or alternatively, you don't re-order, but create an additional cache file that contains the same data in continuous blocks, saving the burden of writing block re-ordering code. I think WinXP does it like that, but I'm not sure. This alternative approach has advantages and disadvantages.)
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, overlooking Dublin's docklands

PostPosted: Sun Sep 30, 2007 11:42 pm    Post subject: Reply with quote

matiqing wrote:
is there any chance to do it like in openSUSE!!?
i'm quite sure the readahead thing IS the way it's done in opensuse, just like naib suggested.
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Mon Oct 01, 2007 5:00 am    Post subject: Reply with quote

jlh wrote:
And I also have some doubt that all this is worth the trouble if you don't combine it with some sort of block re-ordering on the hard disk. The thing that's slow while starting an application isn't that the system is waiting for the application to make I/O requests; and merely knowing those requests in advance won't make anything faster. The thing that's slow is the reading process itself because the hard disk's head is jumping over the entire disk. If you merely preload the files without having them re-ordered on your disk, then you cause that same jumping around with readahead(2) instead of read(2) and you get no improvement. If you re-order your blocks, the hard disk can read continuous blocks and *then* you get an improvement. (Or alternatively, you don't re-order, but create an additional cache file that contains the same data in continuous blocks, saving the burden of writing block re-ordering code. I think WinXP does it like that, but I'm not sure. This alternative approach has advantages and disadvantages.)


While the suse approach doesn't do block reordering on the disk, it does analyze the disk order of the files, and attempts to read them in the correct order.

I'm toying with making something similar in C now, as a preloaded executable loading on boot from init and on. It does work, but I need something very high performance to deliver the data to a single process, instead of just dumping it to a file.
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, overlooking Dublin's docklands

PostPosted: Mon Oct 01, 2007 1:59 pm    Post subject: Reply with quote

neuron wrote:
I'm toying with making something similar in C now, as a preloaded executable loading on boot from init and on. It does work, but I need something very high performance to deliver the data to a single process, instead of just dumping it to a file.
what about a pipe() or a socketpair()? ;)
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Mon Oct 01, 2007 2:27 pm    Post subject: Reply with quote

mdeininger wrote:
neuron wrote:
I'm toying with making something similar in C now, as a preloaded executable loading on boot from init and on. It does work, but I need something very high performance to deliver the data to a single process, instead of just dumping it to a file.
what about a pipe() or a socketpair()? ;)


I'm loading this using preloading, meaning it's starting with init. So filesystems and loopback are not up yet (so pipe and socketpair is a bit of a pain).

I actually solved it using message queue's, and I've got a little preloaded executable that can load with init and log all open files (and readahead stuff from previus boot info), then dump them to a file after the boot has been completed. Wrote a little python file to sort through the file removing invalid stuff (cache/tmp, resolve symlinks), and it can preload from that list as well (using the readahead function).

Just not entirely sure how I'm gonna... stop it :P, cleanly that is.

I'm thinking something that very easily removes /etc/ld.so.preload after it's been loaded, but I dont like the idea of removing that file when something else might use it.
Back to top
View user's profile Send private message
matiit
Guru
Guru


Joined: 05 Aug 2007
Posts: 506

PostPosted: Mon Oct 01, 2007 5:41 pm    Post subject: Reply with quote

somebode tell me:
add it to rc.local:
find /sciezka/do/firefox -exec cat '{}' >/dev/null \; - genious!!
(of course in gentoo i dont see rc.local but it can be of course any script which is run with system)
Back to top
View user's profile Send private message
pussi
l33t
l33t


Joined: 08 May 2004
Posts: 727
Location: Finland

PostPosted: Mon Oct 01, 2007 7:51 pm    Post subject: Reply with quote

matiqing wrote:
(of course in gentoo i dont see rc.local but it can be of course any script which is run with system)
you could simply add that in /etc/conf.d/local.start
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum