View previous topic :: View next topic |
Author |
Message |
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Sun Jul 09, 2006 6:32 am Post subject: [HOWTO] Best boot speedup yet!! |
|
|
The idea is to preload the useful files into kernel buffer cache before they are needed. Since I/O can happen parallel to the other CPU intensive tasks during bootup, this leads to faster bootup and faster startup time for your chosen applications. Other projects use static lists to preload the files. But I have automated the process of generating the "useful" prefetch database.
Have lsof installed before you install it and test it out.
This is /etc/init.d/my-readahead.
Code: |
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
depend() {
before logger
}
start() {
ebegin "Starting readahead ROYALE"
cat /etc/conf.d/* > /dev/null 2>&1 &
cat `cat /etc/conf.d/sofile-list.load` > /dev/null 2>&1 &
if [ -f /forcesampler ];then
/usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
rm -f /forcesampler
fi
eend 0
}
stop() {
eend 0
}
|
The executable shell script /usr/sbin/sample-init-process:
Code: |
#!/bin/bash
final_db="$1"
# total number of lists to keep in /etc/conf.d/
total_to_keep=4
# total number of samples to make in /forcesampler boot.
# It will take twice this amount of seconds to finish sampling.
total_samples_per_boot=200
# touch empty file if not there
[ ! -f "$final_db" ] && touch "$final_db"
let i=0
while [ "$i" -ne "$total_to_keep" ] ; do
if [ ! -f "$final_db$i" ] ; then
break;
else
j=$((i+1))
if [ "$j" = "$total_to_keep" ] ;then
j=0
fi
if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then
i=$((j))
break;
fi
fi
i=$((i+1))
done
if [ "$i" = "$total_to_keep" ] ;then
i=0
fi
slot="$i"
cp "$final_db" "$final_db$i"
function collect_sample_old()
{
lsof |awk '{print $9}'|grep \.so|sort|uniq > $1
lsof |awk '{print $9}'|grep \.conf|sort|uniq >> $1
lsof |awk '{print $9}'|grep /bin/|sort|uniq >> $1
lsof |awk '{print $9}'|grep /sbin/|sort|uniq >> $1
lsof |awk '{print $9}'|grep ttf|sort|uniq >> $1
lsof |awk '{print $9}'|grep /libexec|sort|uniq >> $1
}
function collect_sample()
{
/usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | grep -v "^$" | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" > $1
}
collect_sample /tmp/init_sample1
i=0
while [ "$i" -ne "$total_samples_per_boot" ] ; do
collect_sample /tmp/init_sample2
cat /tmp/init_sample1 /tmp/init_sample2 | /usr/bin/uniquer > /tmp/init_sample3
mv /tmp/init_sample3 /tmp/init_sample1
sleep 0.5
i=$((i+1))
done
cat /tmp/init_sample1 "$final_db$slot" | /usr/bin/uniquer > "$final_db"
rm -f /tmp/init_sample[123]
|
And this is /usr/bin/uniquer:
Code: |
#!/bin/sh
/usr/bin/awk '{
if ($0 in stored_lines)
x=1
else
print
stored_lines[$0]=1
}' $1
|
And then do this:
Code: |
chmod +x /usr/sbin/sample-init-process /usr/bin/uniquer /etc/init.d/my-readahead
rc-update add my-readahead default
touch /forcesampler
|
What you have done is asked my-readahead to create a sampler process which samples what files (fonts, binaries, libraries, configs) are in use, every second. It collects 200 samples (will be busy for at least 10 minutes after boot) and keeps updating the database with a list of useful files.
Once you boot after this the first time, you will probably see a slow down because its collecting samples for use on the next and subsequent boots. Use your system as you normally would. Open firefox, gaim, OO or whatever. Once the sampler script has finished ('ps -aef|grep -v grep|grep sample-init-process' will tell you if its running), reboot. Now this and all subsequent boots will be much faster. <update> Something has changed with baselayout recently and requires you to hit enter when the sampler starts during the once-only boot for sample collection. </update>
If it takes too long for the sample-init-process to finish, reduce the total_samples_per_boot variable in the sampler script to 100 and try again by touching /forcesampler.
In future, whenever you feel like updating the prefetch database, just 'touch /forcesampler' and database will be updated on next reboot.
Uninstall is easy:
Code: | rc-update del my-readahead
\rm /etc/init.d/my-readahead
\rm /usr/sbin/sample-init-process
\rm /etc/conf.d/sofile-list.load*
\rm /usr/bin/uniquer
|
Last edited by devsk on Sat Aug 19, 2006 2:10 am; edited 9 times in total |
|
Back to top |
|
|
johntash n00b
Joined: 30 Apr 2005 Posts: 20 Location: KS
|
Posted: Sun Jul 09, 2006 6:42 am Post subject: |
|
|
I'll give it a try =] |
|
Back to top |
|
|
suicidal_orange_II Apprentice
Joined: 04 Sep 2004 Posts: 299
|
Posted: Sun Jul 09, 2006 11:11 pm Post subject: |
|
|
Me too - the idea sounds good and anything that speeds up booting (and program loading?) must be good
Suicidal_Orange |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Mon Jul 10, 2006 1:04 am Post subject: |
|
|
I made it a howto and put more work into automating it after finding it shaved off close to 15 seconds off of my startup time. I strongly believe that if we have to preload something, it has to be something which will be used. And that's what this script tries to do.
I am down to 40 seconds for a full fledged KDE desktop with initrd and splash enabled at boot and tonnes of services like UPS, sensors, hddtemp etc. starting at default. Moreover, firefox and gaim start really fast when I get my desktop.
I am thinking this might be a better sampler, more concise and faster:
Code: |
/usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | sort | uniq | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" |
What do you guys think? |
|
Back to top |
|
|
cypherphreak n00b
Joined: 23 May 2005 Posts: 17
|
Posted: Wed Jul 26, 2006 3:05 pm Post subject: |
|
|
Mine appears to have not worked.
The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct? |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Thu Jul 27, 2006 4:58 am Post subject: |
|
|
what does Code: | wc -l /etc/conf.d/sofile-list.load | show? is there a list of files in there? |
|
Back to top |
|
|
jit_v1 n00b
Joined: 16 Dec 2004 Posts: 24
|
Posted: Fri Jul 28, 2006 9:37 am Post subject: |
|
|
Thanks for the howto. I've just tried using it but unfortunately don't seem to have gained anything from my boot time
The sofile-list.load seems to have been created correctly etc. and contains 180 files.
I noticed that the rc-script is being loaded in the default runlevel, could this be moved further up so that it starts earlier?
It's probably worth altering the rc-script so that it doesn't do the preload if /forcesampler exists - otherwise the sampler will pickup previously sampled files that may no longer be required.
I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.
jv |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Fri Jul 28, 2006 2:42 pm Post subject: |
|
|
jit_v1 wrote: |
I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.
| ok, this is a bug. Did you rebuild the database after doing this? I think it didn't capture much for you. Was it still sampling after you had your system up? For me, the list has 500 filenames, all of those files are the ones which are used during the boot and afterwards. So, it helps me a lot. |
|
Back to top |
|
|
jit_v1 n00b
Joined: 16 Dec 2004 Posts: 24
|
Posted: Fri Jul 28, 2006 3:50 pm Post subject: |
|
|
Yeah, I rebuilt the database after, got an extra 20 or so files, but still didn't seem to make much difference to the startup time. The sampler process stayed running after bootup for around five minutes. I did notice that my boot time wasn't affected that much when the sampling was going on - you mentioned a delay of 10 minutes to boot - it took just a little longer than normal. Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?
jv |
|
Back to top |
|
|
thomasvk Guru
Joined: 19 Mar 2005 Posts: 597
|
Posted: Sat Jul 29, 2006 1:37 pm Post subject: |
|
|
So if you put something in /dev/null it will be cached? |
|
Back to top |
|
|
drwook Veteran
Joined: 30 Mar 2005 Posts: 1324 Location: London
|
Posted: Sat Jul 29, 2006 1:58 pm Post subject: |
|
|
t0maz wrote: | So if you put something in /dev/null it will be cached? |
Sure, whatever it reads gets buffered/cached. so cat'ing it to /dev/null is as good a way as any. Could probably just as well grep it for the word 'wibble' and redirect the output to /dev/null or something, but that'd just look odd in a script |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Sat Jul 29, 2006 2:54 pm Post subject: |
|
|
jit_v1 wrote: | Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?
jv | yeah, your system is too fast I think, so sampling at probably 0.5 or even 0.2 seconds might give you better results. Note that reducing the sampling interface will be more intrusive while collecting the first time. Also, you might have to up the number of samples because now it will finish in half the time with 0.5 sleep instead of 1 sec sleep. You don't wanna finish too early either.
Another reason for different size database is that you are probably running a different set of services than me.
One way of finding if its working or not is: disable the service my-readahead. Reboot and login into the box. Inside X, do a top and see how much memory is used. Note it down. Enable the service, reboot and login into the box and again do a top to see the memory used. This number should be substantially higher, but close to what you will eventually use if you used your system normally. You can compare o/p of for these two scenarios and it will clearly indicate that you have loaded extra stuff (all useful because you were using that stuff when we sampled it). |
|
Back to top |
|
|
stahlsau Guru
Joined: 09 Jan 2004 Posts: 584 Location: WildWestwoods
|
Posted: Sun Jul 30, 2006 7:56 am Post subject: |
|
|
gj, thanx for the script, i like the idea
At first, i thought you'd be kidding us with cat'ing the files to /dev/null. But after a lil thinking ... |
|
Back to top |
|
|
balk n00b
Joined: 10 Apr 2002 Posts: 62 Location: Olanda
|
Posted: Sun Jul 30, 2006 6:36 pm Post subject: |
|
|
cypherphreak wrote: | Mine appears to have not worked.
The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct? |
Do you have lsof installed? I didn't and the script obviously did not work. I needed another cycle before it worked; I had zero-byte /etc/conf.d/sofile-list.load and /etc/conf.d/sofile-list.load0 files which I had to delete first.
I will now reboot and find out if there is any increase in speed.
[edit]
There seems to be ~some~ increase. I did not measure but it felt faster Thanks for the nice script! _________________ Powered by Gentoo since 1.0_rc6 |
|
Back to top |
|
|
dway n00b
Joined: 18 Jun 2006 Posts: 35
|
Posted: Sun Jul 30, 2006 11:56 pm Post subject: |
|
|
Hi, I'm new on Gentoo (and on linux in general) and i have a fully fonctionnal system now (learning from the documentations, wiki and forum).
Just a little question : does this script break something if I try to use it or is it safe for the system (by the way how do you clear it form the system ?) |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Mon Jul 31, 2006 1:34 am Post subject: |
|
|
dway, if you can't figure out what its doing, you should learn to figure that first. The script is harmless but I would recommend you read up and figure what it is doing before going installing it. You need 'lsof' installed. There are only couple of files it creates. Its easy to get rid of the whole thing by just doing:
Code: |
rc-update del my-readahead
\rm /etc/init.d/my-readahead
\rm /usr/sbin/sample-init-process
\rm /etc/conf.d/sofile-list.load* |
|
|
Back to top |
|
|
dway n00b
Joined: 18 Jun 2006 Posts: 35
|
Posted: Mon Jul 31, 2006 9:21 am Post subject: |
|
|
thx for infos, I'll try to figure it out before trying, regarding your advice (sorry for my english). |
|
Back to top |
|
|
Ic3M4n Advocate
Joined: 02 Nov 2004 Posts: 3489 Location: Bergamo.
|
Posted: Mon Jul 31, 2006 12:30 pm Post subject: |
|
|
Hi all!
I have a problem starting the init script
I have an output like this:
Code: | Could not get dependency info for "my-readahead"!
Please run:
# /sbin/depscan.sh
to try and fix this |
if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this? |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2998 Location: Bay Area, CA
|
Posted: Mon Jul 31, 2006 3:17 pm Post subject: |
|
|
Ic3M4n wrote: | Hi all!
I have a problem starting the init script
I have an output like this:
Code: | Could not get dependency info for "my-readahead"!
Please run:
# /sbin/depscan.sh
to try and fix this |
if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this? | what baselayout, sysvinit, udev versions are you using? |
|
Back to top |
|
|
cypherphreak n00b
Joined: 23 May 2005 Posts: 17
|
Posted: Tue Aug 08, 2006 6:31 pm Post subject: |
|
|
balk wrote: | Do you have lsof installed? |
Simple things like that are always overlooked by me, thanks for pointing that out, works fine now. |
|
Back to top |
|
|
thomasvk Guru
Joined: 19 Mar 2005 Posts: 597
|
Posted: Wed Aug 09, 2006 3:25 pm Post subject: |
|
|
So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now? |
|
Back to top |
|
|
prymitive Apprentice
Joined: 13 Jun 2004 Posts: 260
|
Posted: Wed Aug 09, 2006 3:30 pm Post subject: |
|
|
t0maz wrote: | So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now? |
It's not about dumping some output stuff to /dev/null but reading files from disk so they are put in buffers so when some app want to read them it gets it from buffer with is a lot faster then reading from disk. You think that /dev/null is some buffor or am I stupid? |
|
Back to top |
|
|
thomasvk Guru
Joined: 19 Mar 2005 Posts: 597
|
Posted: Wed Aug 09, 2006 3:36 pm Post subject: |
|
|
prymitive wrote: | You think that /dev/null is some buffor or am I stupid? |
You're stupid!
No, I'm sleeping... |
|
Back to top |
|
|
prymitive Apprentice
Joined: 13 Jun 2004 Posts: 260
|
Posted: Wed Aug 09, 2006 3:41 pm Post subject: |
|
|
t0maz wrote: | prymitive wrote: | You think that /dev/null is some buffor or am I stupid? |
You're stupid!
No, I'm sleeping... |
Don't get me wrong, I wasn't sure if I understood Your question, it's been long time since my english lessons . |
|
Back to top |
|
|
Dr.Dran l33t
Joined: 08 Oct 2004 Posts: 766 Location: Imola - Italy
|
Posted: Thu Aug 10, 2006 7:49 pm Post subject: |
|
|
@devsk
Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc
Thanx
P.S. I your patch work better you may submit it to Uberlod (the baselayout dev)
Cheers
Franco _________________ :: [Dr.Dran] Details ::
- Linux User # 286282
- IT FreeLance Consultant
- President of ImoLUG [Imola & Faenza Linux User Group] |
|
Back to top |
|
|
|