| View previous topic :: View next topic |
| Author |
Message |
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
|
| Back to top |
|
 |
pussi
l33t
Joined: 08 May 2004
Posts: 727
Location: Finland
|
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
 Posted: Sat Sep 29, 2007 5:29 pm Post subject: |
 |
|
readhead doesnt optimise e.g firefox start  |
|
| Back to top |
|
|
pussi
l33t
Joined: 08 May 2004
Posts: 727
Location: Finland
|
| Posted: Sat Sep 29, 2007 6:11 pm Post subject: |
|
|
sorry, I actually thought it could be configured to do that.
you could simply try download the preloading source and compile it manually |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sat Sep 29, 2007 6:59 pm Post subject: |
|
|
| preloading source? what is it? |
|
| Back to top |
|
|
mdeininger
Veteran
Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands
|
| Posted: Sat Sep 29, 2007 8:22 pm Post subject: |
|
|
seems like nothing more than a shell script, you should find it on the suse ftp... or maybe it's linked from somewhere in there...
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland
( Twitter | Blog | GitHub ) |
|
| Back to top |
|
|
pussi
l33t
Joined: 08 May 2004
Posts: 727
Location: Finland
|
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sat Sep 29, 2007 8:47 pm Post subject: |
|
|
| yes i know that but how to use that :/ ? |
|
| Back to top |
|
|
jlh
Tux's lil' helper
Joined: 06 May 2007
Posts: 145
Location: Switzerland::Zürich
|
| Posted: Sat Sep 29, 2007 8:56 pm Post subject: |
|
|
There are multiple ways to improve load times. One is to just keep portions of programs in memory, waiting to be used. OpenOffice.org quickstarter probably does that and similar things are available for firefox and probably other apps. I consider this approach to be bad, because for the entire time you do not use those applications, it just uses up precious RAM that could be used for regular file caching. While it speeds up some things, it slows down others. If at all, I think on-demand preloading is the way to go, so whenever a process starts, something will immediately start loading files that are likely to be requested by that application soon; so when the actually request comes, they're already ready. Windows XP does that and it's a pretty cool thing.
I once tried my luck to write something like that, based on the preload idea, but I didn't get anywhere. After various approaches, everything was starting slower, not faster; the bottom line for me was that the preload-trick is just not powerful enough and has no way to catch all file accesses. However, if I remember correctly there is some work being done at kernel level for linux (and that's probably the only sane level to do this at), but I completely forgot its name or its webpage. I have some vague association with google, so maybe it was a GSoC, or it's hosted by google. Anyone knows what that is? |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sat Sep 29, 2007 9:00 pm Post subject: |
|
|
jlh
But on openSUSE it is working pretty good.
Firefox is starting 2s!!...
i like that!
Please somebody help me with it... |
|
| Back to top |
|
|
jlh
Tux's lil' helper
Joined: 06 May 2007
Posts: 145
Location: Switzerland::Zürich
|
| Posted: Sat Sep 29, 2007 9:01 pm Post subject: |
|
|
Yes, but I bet you used static preloading to get this result. I don't like that: It makes firefox faster and everything else slightly slower.
Of course it makes a lot of sense if you open (and close) firefox a lot. |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sat Sep 29, 2007 9:08 pm Post subject: |
|
|
jlh
You say about openSuse's method? |
|
| Back to top |
|
|
mdeininger
Veteran
Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands
|
| Posted: Sat Sep 29, 2007 9:12 pm Post subject: |
|
|
jlh: yeah something like "kernel based readahead"?
it shouldn't make anything slower, only if it's actually reading while other things would need to read something. from my understanding it's just pre-filling the disk cache with things that are likely to be loaded, probably just using ldd to find out which libs will be used by firefox and then using readahead to tell the kernel to load those files.
it shouldn't actually hurt performance of other apps: if they really do need the ram, then the disk cache will just get smaller and some stuff that was read ahead will be dropped. usually the linux disk cache will simply keep growing anyway, but it's empty at first; telling it what could be needed in advance is nothing bad and shouldn't hurt performance...
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland
( Twitter | Blog | GitHub ) |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sat Sep 29, 2007 9:30 pm Post subject: |
|
|
any concrects?
jlh
You say about openSuse's method? | Quote: | | keep portions of programs in memory, waiting to be used. |
|
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Sun Sep 30, 2007 6:35 am Post subject: |
|
|
The openSuse method of getting running files for me broke quite horribly, using strace like that is.. a tad bit abusive (and put all my processes in HALT state). So I thought I'd do it a cleaner way, by hooking open and looking at all the files actually opened.
Saw someone had already done that (opentrace.c from http://gentoo-wiki.com/HOWTO_prefetch_files_on_boot ), although that broke for me as well, as init wasn't called on all processes. So after slight modification this should work:
| Code: |
/* relevant includes, the define is needed on Linux
* but doesn't break it for FreeBSD
*/
#define _GNU_SOURCE
#include <dlfcn.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdarg.h>
#include <stdlib.h>
#include <stdio.h>
/* mark init() to be executed on library load
*/
static void init (void) __attribute__ ((constructor));
static void onExit (void) __attribute__ ((destructor));
/* pointer to the real write call */
static int (*old_open)() = 0;
static int oFD;
static int init_called = 0;
/* any initialization code here (reading configfiles, whatever)
*/
static void init (void)
{
if (init_called == 1)
return;
init_called = 1;
char* oFile;
/* find the 'real' write, we might need it later */
old_open= (int (*)()) dlsym(RTLD_NEXT, "open");
if (old_open== NULL)
{
fprintf(stderr, "dlsym: %s\n", dlerror());
exit(1);
}
oFile = "/tmp/open_trace.log";
if (oFile)
oFD = old_open (oFile, O_CREAT|O_WRONLY|O_APPEND, 0622);
else
oFD = 2;
}
static void onExit (void)
{
close (oFD);
}
/* grab the open() library call
* hijacking a function is as easy as creating one with the
* same name, since we're the last library loaded
*/
int open (__const char *file, int oflag, ...)
{
init();
va_list ap;
va_start (ap, oflag);
/* do funky stuff */
write (oFD, file, strlen(file));
write (oFD, "\n", 1);
fsync(oFD);
/* call the real open function */
if (oflag & O_CREAT)
return old_open(file, oflag, va_arg(ap,mode_t));
else
return old_open(file, oflag);
va_end(ap);
}
|
gcc -fpic -nodefaultlibs -shared opentrace.c -ldl -o opentrace.so && LD_PRELOAD=/full/path/to/opentrace.so firefox
That should put all firefox'es open's in /tmp/open_trace.log, look at the file, do some grep magic (rip out "." ".." "/dev/*" and stuff like that, and you should have a file you can put in the readahead lists. And you should get firefox preloaded during boot.
I might toy with making a program that runs as init (and for example dumps file on kill signal), should be a very accurate way of doing it. |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sun Sep 30, 2007 7:10 am Post subject: |
|
|
ok
But how to made it like in suse? |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Sun Sep 30, 2007 3:20 pm Post subject: |
|
|
| is there any chance to do it like in openSUSE!!? |
|
| Back to top |
|
|
Naib
Watchman
Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy
|
| Posted: Sun Sep 30, 2007 9:04 pm Post subject: |
|
|
| matiqing wrote: | | is there any chance to do it like in openSUSE!!? |
readahead *can* do what you are after
If you are basically after FF starting in 2sec then edit the readahead list to add firefox-bin and the libs it will need
This will then make FF cached in RAM when you boot to give you yr 2sec first-boot experience
_________________
| Quote: | | Removed by Chiitoo |
|
|
| Back to top |
|
|
jlh
Tux's lil' helper
Joined: 06 May 2007
Posts: 145
Location: Switzerland::Zürich
|
| Posted: Sun Sep 30, 2007 9:22 pm Post subject: |
|
|
I've tried the strace approach and I've tried the LD_PRELOAD technique and they're both not-so-good. The reason for this is that those methods only allow you to catch library calls, like open(2), read(2), etc. This is not sufficient because *a lot* of binaries and libraries use mmap(2) to map files into their address space and accessing their content without any further library calls. And you cannot possibly track what is being read from those files using any of the above methods. You still know which files are being accessed (via open(2)), but you cannot find out which parts and how much of the file is being loaded (if any at all). So either you don't preload those files at all, and you partially miss the purpose of this entire idea, or you preload the entire file, potentially loading a lot of unnecessary data, which slows things down. That's why I say that the kernel is the only sane place to do things like this, because it's the only instance that can really know what a process is doing.
And I also have some doubt that all this is worth the trouble if you don't combine it with some sort of block re-ordering on the hard disk. The thing that's slow while starting an application isn't that the system is waiting for the application to make I/O requests; and merely knowing those requests in advance won't make anything faster. The thing that's slow is the reading process itself because the hard disk's head is jumping over the entire disk. If you merely preload the files without having them re-ordered on your disk, then you cause that same jumping around with readahead(2) instead of read(2) and you get no improvement. If you re-order your blocks, the hard disk can read continuous blocks and *then* you get an improvement. (Or alternatively, you don't re-order, but create an additional cache file that contains the same data in continuous blocks, saving the burden of writing block re-ordering code. I think WinXP does it like that, but I'm not sure. This alternative approach has advantages and disadvantages.) |
|
| Back to top |
|
|
mdeininger
Veteran
Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands
|
| Posted: Sun Sep 30, 2007 11:42 pm Post subject: |
|
|
| matiqing wrote: | | is there any chance to do it like in openSUSE!!? |
i'm quite sure the readahead thing IS the way it's done in opensuse, just like naib suggested.
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland
( Twitter | Blog | GitHub ) |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Mon Oct 01, 2007 5:00 am Post subject: |
|
|
| jlh wrote: | | And I also have some doubt that all this is worth the trouble if you don't combine it with some sort of block re-ordering on the hard disk. The thing that's slow while starting an application isn't that the system is waiting for the application to make I/O requests; and merely knowing those requests in advance won't make anything faster. The thing that's slow is the reading process itself because the hard disk's head is jumping over the entire disk. If you merely preload the files without having them re-ordered on your disk, then you cause that same jumping around with readahead(2) instead of read(2) and you get no improvement. If you re-order your blocks, the hard disk can read continuous blocks and *then* you get an improvement. (Or alternatively, you don't re-order, but create an additional cache file that contains the same data in continuous blocks, saving the burden of writing block re-ordering code. I think WinXP does it like that, but I'm not sure. This alternative approach has advantages and disadvantages.) |
While the suse approach doesn't do block reordering on the disk, it does analyze the disk order of the files, and attempts to read them in the correct order.
I'm toying with making something similar in C now, as a preloaded executable loading on boot from init and on. It does work, but I need something very high performance to deliver the data to a single process, instead of just dumping it to a file. |
|
| Back to top |
|
|
mdeininger
Veteran
Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands
|
| Posted: Mon Oct 01, 2007 1:59 pm Post subject: |
|
|
| neuron wrote: | | I'm toying with making something similar in C now, as a preloaded executable loading on boot from init and on. It does work, but I need something very high performance to deliver the data to a single process, instead of just dumping it to a file. |
what about a pipe() or a socketpair()? 
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland
( Twitter | Blog | GitHub ) |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Mon Oct 01, 2007 2:27 pm Post subject: |
|
|
| mdeininger wrote: | | neuron wrote: | | I'm toying with making something similar in C now, as a preloaded executable loading on boot from init and on. It does work, but I need something very high performance to deliver the data to a single process, instead of just dumping it to a file. |
what about a pipe() or a socketpair()? |
I'm loading this using preloading, meaning it's starting with init. So filesystems and loopback are not up yet (so pipe and socketpair is a bit of a pain).
I actually solved it using message queue's, and I've got a little preloaded executable that can load with init and log all open files (and readahead stuff from previus boot info), then dump them to a file after the boot has been completed. Wrote a little python file to sort through the file removing invalid stuff (cache/tmp, resolve symlinks), and it can preload from that list as well (using the readahead function).
Just not entirely sure how I'm gonna... stop it , cleanly that is.
I'm thinking something that very easily removes /etc/ld.so.preload after it's been loaded, but I dont like the idea of removing that file when something else might use it. |
|
| Back to top |
|
|
matiit
Guru
Joined: 05 Aug 2007
Posts: 506
|
| Posted: Mon Oct 01, 2007 5:41 pm Post subject: |
|
|
somebode tell me:
add it to rc.local:
find /sciezka/do/firefox -exec cat '{}' >/dev/null \; - genious!!
(of course in gentoo i dont see rc.local but it can be of course any script which is run with system) |
|
| Back to top |
|
|
pussi
l33t
Joined: 08 May 2004
Posts: 727
Location: Finland
|
| Posted: Mon Oct 01, 2007 7:51 pm Post subject: |
|
|
| matiqing wrote: | | (of course in gentoo i dont see rc.local but it can be of course any script which is run with system) |
you could simply add that in /etc/conf.d/local.start |
|
| Back to top |
|
|
|
Unsupported Software
