Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[HOWTO] Best boot speedup yet!!
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sun Jul 09, 2006 6:32 am    Post subject: [HOWTO] Best boot speedup yet!! Reply with quote

The idea is to preload the useful files into kernel buffer cache before they are needed. Since I/O can happen parallel to the other CPU intensive tasks during bootup, this leads to faster bootup and faster startup time for your chosen applications. Other projects use static lists to preload the files. But I have automated the process of generating the "useful" prefetch database.

Have lsof installed before you install it and test it out.

This is /etc/init.d/my-readahead.

Code:

#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

depend() {
        before logger
}

start() {
        ebegin "Starting readahead ROYALE"
        cat /etc/conf.d/* > /dev/null 2>&1 &
        cat `cat /etc/conf.d/sofile-list.load` > /dev/null 2>&1 &
        if [ -f /forcesampler ];then
                /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
                rm -f /forcesampler
        fi
        eend 0
}

stop() {
        eend 0
}


The executable shell script /usr/sbin/sample-init-process:

Code:

#!/bin/bash
final_db="$1"

# total number of lists to keep in /etc/conf.d/
total_to_keep=4

# total number of samples to make in /forcesampler boot.
# It will take twice this amount of seconds to finish sampling.
total_samples_per_boot=200

# touch empty file if not there
[ ! -f "$final_db" ] && touch "$final_db"

let i=0
while [ "$i" -ne "$total_to_keep" ] ; do
        if [ ! -f "$final_db$i" ] ; then
                break;
        else
                j=$((i+1))
                if [ "$j" = "$total_to_keep" ] ;then
                        j=0
                fi
                if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then
                        i=$((j))
                        break;
                fi
        fi
        i=$((i+1))
done
if [ "$i" = "$total_to_keep" ] ;then
        i=0
fi

slot="$i"
cp "$final_db" "$final_db$i"

function collect_sample_old()
{
        lsof |awk '{print $9}'|grep \.so|sort|uniq > $1
        lsof |awk '{print $9}'|grep \.conf|sort|uniq >> $1
        lsof |awk '{print $9}'|grep /bin/|sort|uniq >> $1
        lsof |awk '{print $9}'|grep /sbin/|sort|uniq >> $1
        lsof |awk '{print $9}'|grep ttf|sort|uniq >> $1
        lsof |awk '{print $9}'|grep /libexec|sort|uniq >> $1
}

function collect_sample()
{
        /usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | grep -v "^$" | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" > $1
}

collect_sample /tmp/init_sample1

i=0
while [ "$i" -ne "$total_samples_per_boot" ] ; do
        collect_sample /tmp/init_sample2
        cat /tmp/init_sample1 /tmp/init_sample2 | /usr/bin/uniquer > /tmp/init_sample3
        mv /tmp/init_sample3 /tmp/init_sample1
        sleep 0.5
        i=$((i+1))
done

cat /tmp/init_sample1 "$final_db$slot" | /usr/bin/uniquer > "$final_db"

rm -f /tmp/init_sample[123]


And this is /usr/bin/uniquer:
Code:

#!/bin/sh
/usr/bin/awk '{
if ($0 in stored_lines)
        x=1
else
        print
        stored_lines[$0]=1
}' $1

And then do this:

Code:

chmod +x /usr/sbin/sample-init-process /usr/bin/uniquer /etc/init.d/my-readahead
rc-update add my-readahead default
touch /forcesampler


What you have done is asked my-readahead to create a sampler process which samples what files (fonts, binaries, libraries, configs) are in use, every second. It collects 200 samples (will be busy for at least 10 minutes after boot) and keeps updating the database with a list of useful files.

Once you boot after this the first time, you will probably see a slow down because its collecting samples for use on the next and subsequent boots. Use your system as you normally would. Open firefox, gaim, OO or whatever. Once the sampler script has finished ('ps -aef|grep -v grep|grep sample-init-process' will tell you if its running), reboot. Now this and all subsequent boots will be much faster. <update> Something has changed with baselayout recently and requires you to hit enter when the sampler starts during the once-only boot for sample collection. </update>

If it takes too long for the sample-init-process to finish, reduce the total_samples_per_boot variable in the sampler script to 100 and try again by touching /forcesampler.

In future, whenever you feel like updating the prefetch database, just 'touch /forcesampler' and database will be updated on next reboot.

Uninstall is easy:

Code:
rc-update del my-readahead
\rm /etc/init.d/my-readahead
\rm /usr/sbin/sample-init-process
\rm /etc/conf.d/sofile-list.load*
\rm /usr/bin/uniquer


Last edited by devsk on Sat Aug 19, 2006 2:10 am; edited 9 times in total
Back to top
View user's profile Send private message
johntash
n00b
n00b


Joined: 30 Apr 2005
Posts: 20
Location: KS

PostPosted: Sun Jul 09, 2006 6:42 am    Post subject: Reply with quote

I'll give it a try =]
Back to top
View user's profile Send private message
suicidal_orange_II
Apprentice
Apprentice


Joined: 04 Sep 2004
Posts: 299

PostPosted: Sun Jul 09, 2006 11:11 pm    Post subject: Reply with quote

Me too - the idea sounds good and anything that speeds up booting (and program loading?) must be good :D


Suicidal_Orange
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Mon Jul 10, 2006 1:04 am    Post subject: Reply with quote

I made it a howto and put more work into automating it after finding it shaved off close to 15 seconds off of my startup time. I strongly believe that if we have to preload something, it has to be something which will be used. And that's what this script tries to do.

I am down to 40 seconds for a full fledged KDE desktop with initrd and splash enabled at boot and tonnes of services like UPS, sensors, hddtemp etc. starting at default. Moreover, firefox and gaim start really fast when I get my desktop.

I am thinking this might be a better sampler, more concise and faster:
Code:

/usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | sort | uniq | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]"


What do you guys think?
Back to top
View user's profile Send private message
cypherphreak
n00b
n00b


Joined: 23 May 2005
Posts: 17

PostPosted: Wed Jul 26, 2006 3:05 pm    Post subject: Reply with quote

Mine appears to have not worked.
The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Jul 27, 2006 4:58 am    Post subject: Reply with quote

what does
Code:
wc -l /etc/conf.d/sofile-list.load
show? is there a list of files in there?
Back to top
View user's profile Send private message
jit_v1
n00b
n00b


Joined: 16 Dec 2004
Posts: 24

PostPosted: Fri Jul 28, 2006 9:37 am    Post subject: Reply with quote

Thanks for the howto. I've just tried using it but unfortunately don't seem to have gained anything from my boot time :-(

The sofile-list.load seems to have been created correctly etc. and contains 180 files.

I noticed that the rc-script is being loaded in the default runlevel, could this be moved further up so that it starts earlier?

It's probably worth altering the rc-script so that it doesn't do the preload if /forcesampler exists - otherwise the sampler will pickup previously sampled files that may no longer be required.

I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.

jv
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Fri Jul 28, 2006 2:42 pm    Post subject: Reply with quote

jit_v1 wrote:

I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.
ok, this is a bug. Did you rebuild the database after doing this? I think it didn't capture much for you. Was it still sampling after you had your system up? For me, the list has 500 filenames, all of those files are the ones which are used during the boot and afterwards. So, it helps me a lot.
Back to top
View user's profile Send private message
jit_v1
n00b
n00b


Joined: 16 Dec 2004
Posts: 24

PostPosted: Fri Jul 28, 2006 3:50 pm    Post subject: Reply with quote

Yeah, I rebuilt the database after, got an extra 20 or so files, but still didn't seem to make much difference to the startup time. The sampler process stayed running after bootup for around five minutes. I did notice that my boot time wasn't affected that much when the sampling was going on - you mentioned a delay of 10 minutes to boot - it took just a little longer than normal. Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?

jv
Back to top
View user's profile Send private message
thomasvk
Guru
Guru


Joined: 19 Mar 2005
Posts: 597

PostPosted: Sat Jul 29, 2006 1:37 pm    Post subject: Reply with quote

So if you put something in /dev/null it will be cached? 8O
Back to top
View user's profile Send private message
drwook
Veteran
Veteran


Joined: 30 Mar 2005
Posts: 1324
Location: London

PostPosted: Sat Jul 29, 2006 1:58 pm    Post subject: Reply with quote

t0maz wrote:
So if you put something in /dev/null it will be cached? 8O


Sure, whatever it reads gets buffered/cached. so cat'ing it to /dev/null is as good a way as any. Could probably just as well grep it for the word 'wibble' and redirect the output to /dev/null or something, but that'd just look odd in a script ;)
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sat Jul 29, 2006 2:54 pm    Post subject: Reply with quote

jit_v1 wrote:
Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?

jv
yeah, your system is too fast I think, so sampling at probably 0.5 or even 0.2 seconds might give you better results. Note that reducing the sampling interface will be more intrusive while collecting the first time. Also, you might have to up the number of samples because now it will finish in half the time with 0.5 sleep instead of 1 sec sleep. You don't wanna finish too early either.

Another reason for different size database is that you are probably running a different set of services than me.

One way of finding if its working or not is: disable the service my-readahead. Reboot and login into the box. Inside X, do a top and see how much memory is used. Note it down. Enable the service, reboot and login into the box and again do a top to see the memory used. This number should be substantially higher, but close to what you will eventually use if you used your system normally. You can compare o/p of
Code:
cat /proc/meminfo
for these two scenarios and it will clearly indicate that you have loaded extra stuff (all useful because you were using that stuff when we sampled it).
Back to top
View user's profile Send private message
stahlsau
Guru
Guru


Joined: 09 Jan 2004
Posts: 584
Location: WildWestwoods

PostPosted: Sun Jul 30, 2006 7:56 am    Post subject: Reply with quote

gj, thanx for the script, i like the idea ;)
At first, i thought you'd be kidding us with cat'ing the files to /dev/null. But after a lil thinking ... ;)
Back to top
View user's profile Send private message
balk
n00b
n00b


Joined: 10 Apr 2002
Posts: 62
Location: Olanda

PostPosted: Sun Jul 30, 2006 6:36 pm    Post subject: Reply with quote

cypherphreak wrote:
Mine appears to have not worked.
The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct?

Do you have lsof installed? I didn't and the script obviously did not work. :) I needed another cycle before it worked; I had zero-byte /etc/conf.d/sofile-list.load and /etc/conf.d/sofile-list.load0 files which I had to delete first.

I will now reboot and find out if there is any increase in speed.
[edit]
There seems to be ~some~ increase. I did not measure but it felt faster :) Thanks for the nice script!
_________________
Powered by Gentoo since 1.0_rc6
Back to top
View user's profile Send private message
dway
n00b
n00b


Joined: 18 Jun 2006
Posts: 35

PostPosted: Sun Jul 30, 2006 11:56 pm    Post subject: Reply with quote

Hi, I'm new on Gentoo (and on linux in general) and i have a fully fonctionnal system now (learning from the documentations, wiki and forum).
Just a little question : does this script break something if I try to use it or is it safe for the system (by the way how do you clear it form the system ?)
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Mon Jul 31, 2006 1:34 am    Post subject: Reply with quote

dway, if you can't figure out what its doing, you should learn to figure that first. The script is harmless but I would recommend you read up and figure what it is doing before going installing it. You need 'lsof' installed. There are only couple of files it creates. Its easy to get rid of the whole thing by just doing:
Code:

rc-update del my-readahead
\rm /etc/init.d/my-readahead
\rm /usr/sbin/sample-init-process
\rm /etc/conf.d/sofile-list.load*
Back to top
View user's profile Send private message
dway
n00b
n00b


Joined: 18 Jun 2006
Posts: 35

PostPosted: Mon Jul 31, 2006 9:21 am    Post subject: Reply with quote

thx for infos, I'll try to figure it out before trying, regarding your advice (sorry for my english).
Back to top
View user's profile Send private message
Ic3M4n
Advocate
Advocate


Joined: 02 Nov 2004
Posts: 3489
Location: Bergamo.

PostPosted: Mon Jul 31, 2006 12:30 pm    Post subject: Reply with quote

Hi all!
I have a problem starting the init script
I have an output like this:
Code:
Could not get dependency info for "my-readahead"!
Please run:
# /sbin/depscan.sh
to try and fix this


if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Mon Jul 31, 2006 3:17 pm    Post subject: Reply with quote

Ic3M4n wrote:
Hi all!
I have a problem starting the init script
I have an output like this:
Code:
Could not get dependency info for "my-readahead"!
Please run:
# /sbin/depscan.sh
to try and fix this


if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?
what baselayout, sysvinit, udev versions are you using?
Back to top
View user's profile Send private message
cypherphreak
n00b
n00b


Joined: 23 May 2005
Posts: 17

PostPosted: Tue Aug 08, 2006 6:31 pm    Post subject: Reply with quote

balk wrote:
Do you have lsof installed?

Simple things like that are always overlooked by me, thanks for pointing that out, works fine now.
Back to top
View user's profile Send private message
thomasvk
Guru
Guru


Joined: 19 Mar 2005
Posts: 597

PostPosted: Wed Aug 09, 2006 3:25 pm    Post subject: Reply with quote

So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?
Back to top
View user's profile Send private message
prymitive
Apprentice
Apprentice


Joined: 13 Jun 2004
Posts: 260

PostPosted: Wed Aug 09, 2006 3:30 pm    Post subject: Reply with quote

t0maz wrote:
So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?


It's not about dumping some output stuff to /dev/null but reading files from disk so they are put in buffers so when some app want to read them it gets it from buffer with is a lot faster then reading from disk. You think that /dev/null is some buffor or am I stupid?
Back to top
View user's profile Send private message
thomasvk
Guru
Guru


Joined: 19 Mar 2005
Posts: 597

PostPosted: Wed Aug 09, 2006 3:36 pm    Post subject: Reply with quote

prymitive wrote:
You think that /dev/null is some buffor or am I stupid?

You're stupid!

No, I'm sleeping... :oops:
Back to top
View user's profile Send private message
prymitive
Apprentice
Apprentice


Joined: 13 Jun 2004
Posts: 260

PostPosted: Wed Aug 09, 2006 3:41 pm    Post subject: Reply with quote

t0maz wrote:
prymitive wrote:
You think that /dev/null is some buffor or am I stupid?

You're stupid!

No, I'm sleeping... :oops:


Don't get me wrong, I wasn't sure if I understood Your question, it's been long time since my english lessons ;).
Back to top
View user's profile Send private message
Dr.Dran
l33t
l33t


Joined: 08 Oct 2004
Posts: 766
Location: Imola - Italy

PostPosted: Thu Aug 10, 2006 7:49 pm    Post subject: Reply with quote

@devsk
Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc

Thanx :D

P.S. I your patch work better you may submit it to Uberlod (the baselayout dev) :D

Cheers

Franco
_________________
:: [Dr.Dran] Details ::
- Linux User # 286282
- IT FreeLance Consultant
- President of ImoLUG [Imola & Faenza Linux User Group]
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum