Forums

Skip to content

Advanced search
  • Quick links
    • Unanswered topics
    • Active topics
    • Search
  • FAQ
  • Login
  • Register
  • Board index Discussion & Documentation Documentation, Tips & Tricks
  • Search

[HOWTO] Best boot speedup yet!!

Unofficial documentation for various parts of Gentoo Linux. Note: This is not a support forum.
Post Reply
Advanced search
63 posts
  • 1
  • 2
  • 3
  • Next
Author
Message
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

[HOWTO] Best boot speedup yet!!

  • Quote

Post by devsk » Sun Jul 09, 2006 6:32 am

The idea is to preload the useful files into kernel buffer cache before they are needed. Since I/O can happen parallel to the other CPU intensive tasks during bootup, this leads to faster bootup and faster startup time for your chosen applications. Other projects use static lists to preload the files. But I have automated the process of generating the "useful" prefetch database.

Have lsof installed before you install it and test it out.

This is /etc/init.d/my-readahead.

Code: Select all

#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

depend() {
        before logger
}

start() {
        ebegin "Starting readahead ROYALE"
        cat /etc/conf.d/* > /dev/null 2>&1 &
        cat `cat /etc/conf.d/sofile-list.load` > /dev/null 2>&1 &
        if [ -f /forcesampler ];then
                /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
                rm -f /forcesampler
        fi
        eend 0
}

stop() {
        eend 0
}
The executable shell script /usr/sbin/sample-init-process:

Code: Select all

#!/bin/bash
final_db="$1"

# total number of lists to keep in /etc/conf.d/
total_to_keep=4

# total number of samples to make in /forcesampler boot.
# It will take twice this amount of seconds to finish sampling.
total_samples_per_boot=200

# touch empty file if not there
[ ! -f "$final_db" ] && touch "$final_db"

let i=0
while [ "$i" -ne "$total_to_keep" ] ; do
        if [ ! -f "$final_db$i" ] ; then
                break;
        else
                j=$((i+1))
                if [ "$j" = "$total_to_keep" ] ;then
                        j=0
                fi
                if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then
                        i=$((j))
                        break;
                fi
        fi
        i=$((i+1))
done
if [ "$i" = "$total_to_keep" ] ;then
        i=0
fi

slot="$i"
cp "$final_db" "$final_db$i"

function collect_sample_old()
{
        lsof |awk '{print $9}'|grep \.so|sort|uniq > $1
        lsof |awk '{print $9}'|grep \.conf|sort|uniq >> $1
        lsof |awk '{print $9}'|grep /bin/|sort|uniq >> $1
        lsof |awk '{print $9}'|grep /sbin/|sort|uniq >> $1
        lsof |awk '{print $9}'|grep ttf|sort|uniq >> $1
        lsof |awk '{print $9}'|grep /libexec|sort|uniq >> $1
}

function collect_sample()
{
        /usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | grep -v "^$" | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" > $1
}

collect_sample /tmp/init_sample1

i=0
while [ "$i" -ne "$total_samples_per_boot" ] ; do
        collect_sample /tmp/init_sample2
        cat /tmp/init_sample1 /tmp/init_sample2 | /usr/bin/uniquer > /tmp/init_sample3
        mv /tmp/init_sample3 /tmp/init_sample1
        sleep 0.5
        i=$((i+1))
done

cat /tmp/init_sample1 "$final_db$slot" | /usr/bin/uniquer > "$final_db"

rm -f /tmp/init_sample[123]
And this is /usr/bin/uniquer:

Code: Select all

#!/bin/sh
/usr/bin/awk '{
if ($0 in stored_lines)
        x=1
else
        print
        stored_lines[$0]=1
}' $1
And then do this:

Code: Select all

chmod +x /usr/sbin/sample-init-process /usr/bin/uniquer /etc/init.d/my-readahead
rc-update add my-readahead default
touch /forcesampler

What you have done is asked my-readahead to create a sampler process which samples what files (fonts, binaries, libraries, configs) are in use, every second. It collects 200 samples (will be busy for at least 10 minutes after boot) and keeps updating the database with a list of useful files.

Once you boot after this the first time, you will probably see a slow down because its collecting samples for use on the next and subsequent boots. Use your system as you normally would. Open firefox, gaim, OO or whatever. Once the sampler script has finished ('ps -aef|grep -v grep|grep sample-init-process' will tell you if its running), reboot. Now this and all subsequent boots will be much faster. <update> Something has changed with baselayout recently and requires you to hit enter when the sampler starts during the once-only boot for sample collection. </update>

If it takes too long for the sample-init-process to finish, reduce the total_samples_per_boot variable in the sampler script to 100 and try again by touching /forcesampler.

In future, whenever you feel like updating the prefetch database, just 'touch /forcesampler' and database will be updated on next reboot.

Uninstall is easy:

Code: Select all

rc-update del my-readahead
\rm /etc/init.d/my-readahead
\rm /usr/sbin/sample-init-process
\rm /etc/conf.d/sofile-list.load*
\rm /usr/bin/uniquer
Last edited by devsk on Sat Aug 19, 2006 2:10 am, edited 9 times in total.
Top
johntash
n00b
n00b
Posts: 20
Joined: Sat Apr 30, 2005 10:47 pm
Location: KS

  • Quote

Post by johntash » Sun Jul 09, 2006 6:42 am

I'll give it a try =]
Top
suicidal_orange_II
Apprentice
Apprentice
Posts: 299
Joined: Sat Sep 04, 2004 11:39 am

  • Quote

Post by suicidal_orange_II » Sun Jul 09, 2006 11:11 pm

Me too - the idea sounds good and anything that speeds up booting (and program loading?) must be good :D


Suicidal_Orange
Top
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

  • Quote

Post by devsk » Mon Jul 10, 2006 1:04 am

I made it a howto and put more work into automating it after finding it shaved off close to 15 seconds off of my startup time. I strongly believe that if we have to preload something, it has to be something which will be used. And that's what this script tries to do.

I am down to 40 seconds for a full fledged KDE desktop with initrd and splash enabled at boot and tonnes of services like UPS, sensors, hddtemp etc. starting at default. Moreover, firefox and gaim start really fast when I get my desktop.

I am thinking this might be a better sampler, more concise and faster:

Code: Select all

/usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | sort | uniq | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]"
What do you guys think?
Top
cypherphreak
n00b
n00b
Posts: 17
Joined: Mon May 23, 2005 2:34 pm

  • Quote

Post by cypherphreak » Wed Jul 26, 2006 3:05 pm

Mine appears to have not worked.
The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct?
Top
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

  • Quote

Post by devsk » Thu Jul 27, 2006 4:58 am

what does

Code: Select all

wc -l /etc/conf.d/sofile-list.load
show? is there a list of files in there?
Top
jit_v1
n00b
n00b
Posts: 24
Joined: Thu Dec 16, 2004 2:18 pm

  • Quote

Post by jit_v1 » Fri Jul 28, 2006 9:37 am

Thanks for the howto. I've just tried using it but unfortunately don't seem to have gained anything from my boot time :-(

The sofile-list.load seems to have been created correctly etc. and contains 180 files.

I noticed that the rc-script is being loaded in the default runlevel, could this be moved further up so that it starts earlier?

It's probably worth altering the rc-script so that it doesn't do the preload if /forcesampler exists - otherwise the sampler will pickup previously sampled files that may no longer be required.

I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.

jv
Top
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

  • Quote

Post by devsk » Fri Jul 28, 2006 2:42 pm

jit_v1 wrote: I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.
ok, this is a bug. Did you rebuild the database after doing this? I think it didn't capture much for you. Was it still sampling after you had your system up? For me, the list has 500 filenames, all of those files are the ones which are used during the boot and afterwards. So, it helps me a lot.
Top
jit_v1
n00b
n00b
Posts: 24
Joined: Thu Dec 16, 2004 2:18 pm

  • Quote

Post by jit_v1 » Fri Jul 28, 2006 3:50 pm

Yeah, I rebuilt the database after, got an extra 20 or so files, but still didn't seem to make much difference to the startup time. The sampler process stayed running after bootup for around five minutes. I did notice that my boot time wasn't affected that much when the sampling was going on - you mentioned a delay of 10 minutes to boot - it took just a little longer than normal. Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?

jv
Top
thomasvk
Guru
Guru
Posts: 597
Joined: Sat Mar 19, 2005 6:53 pm

  • Quote

Post by thomasvk » Sat Jul 29, 2006 1:37 pm

So if you put something in /dev/null it will be cached? 8O
Top
drwook
Veteran
Veteran
Posts: 1324
Joined: Wed Mar 30, 2005 3:02 pm
Location: London

  • Quote

Post by drwook » Sat Jul 29, 2006 1:58 pm

t0maz wrote:So if you put something in /dev/null it will be cached? 8O
Sure, whatever it reads gets buffered/cached. so cat'ing it to /dev/null is as good a way as any. Could probably just as well grep it for the word 'wibble' and redirect the output to /dev/null or something, but that'd just look odd in a script ;)
Top
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

  • Quote

Post by devsk » Sat Jul 29, 2006 2:54 pm

jit_v1 wrote:Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?

jv
yeah, your system is too fast I think, so sampling at probably 0.5 or even 0.2 seconds might give you better results. Note that reducing the sampling interface will be more intrusive while collecting the first time. Also, you might have to up the number of samples because now it will finish in half the time with 0.5 sleep instead of 1 sec sleep. You don't wanna finish too early either.

Another reason for different size database is that you are probably running a different set of services than me.

One way of finding if its working or not is: disable the service my-readahead. Reboot and login into the box. Inside X, do a top and see how much memory is used. Note it down. Enable the service, reboot and login into the box and again do a top to see the memory used. This number should be substantially higher, but close to what you will eventually use if you used your system normally. You can compare o/p of

Code: Select all

cat /proc/meminfo
for these two scenarios and it will clearly indicate that you have loaded extra stuff (all useful because you were using that stuff when we sampled it).
Top
stahlsau
Guru
Guru
User avatar
Posts: 584
Joined: Fri Jan 09, 2004 8:16 am
Location: WildWestwoods

  • Quote

Post by stahlsau » Sun Jul 30, 2006 7:56 am

gj, thanx for the script, i like the idea ;)
At first, i thought you'd be kidding us with cat'ing the files to /dev/null. But after a lil thinking ... ;)
Top
balk
n00b
n00b
Posts: 62
Joined: Wed Apr 10, 2002 6:28 am
Location: Olanda
Contact:
Contact balk
Website

  • Quote

Post by balk » Sun Jul 30, 2006 6:36 pm

cypherphreak wrote:Mine appears to have not worked.
The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct?
Do you have lsof installed? I didn't and the script obviously did not work. :) I needed another cycle before it worked; I had zero-byte /etc/conf.d/sofile-list.load and /etc/conf.d/sofile-list.load0 files which I had to delete first.

I will now reboot and find out if there is any increase in speed.
[edit]
There seems to be ~some~ increase. I did not measure but it felt faster :) Thanks for the nice script!
Powered by Gentoo since 1.0_rc6
Top
dway
n00b
n00b
Posts: 35
Joined: Sun Jun 18, 2006 10:14 pm

  • Quote

Post by dway » Sun Jul 30, 2006 11:56 pm

Hi, I'm new on Gentoo (and on linux in general) and i have a fully fonctionnal system now (learning from the documentations, wiki and forum).
Just a little question : does this script break something if I try to use it or is it safe for the system (by the way how do you clear it form the system ?)
Top
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

  • Quote

Post by devsk » Mon Jul 31, 2006 1:34 am

dway, if you can't figure out what its doing, you should learn to figure that first. The script is harmless but I would recommend you read up and figure what it is doing before going installing it. You need 'lsof' installed. There are only couple of files it creates. Its easy to get rid of the whole thing by just doing:

Code: Select all

rc-update del my-readahead
\rm /etc/init.d/my-readahead
\rm /usr/sbin/sample-init-process
\rm /etc/conf.d/sofile-list.load*
Top
dway
n00b
n00b
Posts: 35
Joined: Sun Jun 18, 2006 10:14 pm

  • Quote

Post by dway » Mon Jul 31, 2006 9:21 am

thx for infos, I'll try to figure it out before trying, regarding your advice (sorry for my english).
Top
Ic3M4n
Advocate
Advocate
User avatar
Posts: 3489
Joined: Tue Nov 02, 2004 5:46 pm
Location: Bergamo.

  • Quote

Post by Ic3M4n » Mon Jul 31, 2006 12:30 pm

Hi all!
I have a problem starting the init script
I have an output like this:

Code: Select all

Could not get dependency info for "my-readahead"!
Please run:
# /sbin/depscan.sh
to try and fix this
if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?
Top
devsk
Advocate
Advocate
User avatar
Posts: 3039
Joined: Fri Oct 24, 2003 1:16 am
Location: Bay Area, CA

  • Quote

Post by devsk » Mon Jul 31, 2006 3:17 pm

Ic3M4n wrote:Hi all!
I have a problem starting the init script
I have an output like this:

Code: Select all

Could not get dependency info for "my-readahead"!
Please run:
# /sbin/depscan.sh
to try and fix this
if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?
what baselayout, sysvinit, udev versions are you using?
Top
cypherphreak
n00b
n00b
Posts: 17
Joined: Mon May 23, 2005 2:34 pm

  • Quote

Post by cypherphreak » Tue Aug 08, 2006 6:31 pm

balk wrote:Do you have lsof installed?
Simple things like that are always overlooked by me, thanks for pointing that out, works fine now.
Top
thomasvk
Guru
Guru
Posts: 597
Joined: Sat Mar 19, 2005 6:53 pm

  • Quote

Post by thomasvk » Wed Aug 09, 2006 3:25 pm

So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?
Top
prymitive
Apprentice
Apprentice
Posts: 260
Joined: Sun Jun 13, 2004 2:37 pm

  • Quote

Post by prymitive » Wed Aug 09, 2006 3:30 pm

t0maz wrote:So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?
It's not about dumping some output stuff to /dev/null but reading files from disk so they are put in buffers so when some app want to read them it gets it from buffer with is a lot faster then reading from disk. You think that /dev/null is some buffor or am I stupid?
Top
thomasvk
Guru
Guru
Posts: 597
Joined: Sat Mar 19, 2005 6:53 pm

  • Quote

Post by thomasvk » Wed Aug 09, 2006 3:36 pm

prymitive wrote:You think that /dev/null is some buffor or am I stupid?
You're stupid!

No, I'm sleeping... :oops:
Top
prymitive
Apprentice
Apprentice
Posts: 260
Joined: Sun Jun 13, 2004 2:37 pm

  • Quote

Post by prymitive » Wed Aug 09, 2006 3:41 pm

t0maz wrote:
prymitive wrote:You think that /dev/null is some buffor or am I stupid?
You're stupid!

No, I'm sleeping... :oops:
Don't get me wrong, I wasn't sure if I understood Your question, it's been long time since my english lessons ;).
Top
Dr.Dran
l33t
l33t
User avatar
Posts: 766
Joined: Fri Oct 08, 2004 5:21 pm
Location: Imola - Italy
Contact:
Contact Dr.Dran
Website

  • Quote

Post by Dr.Dran » Thu Aug 10, 2006 7:49 pm

@devsk
Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc

Thanx :D

P.S. I your patch work better you may submit it to Uberlod (the baselayout dev) :D

Cheers

Franco
:: [Dr.Dran] Details ::
- Linux User # 286282
- IT FreeLance Consultant
- President of ImoLUG [Imola & Faenza Linux User Group]
Top
Post Reply

63 posts
  • 1
  • 2
  • 3
  • Next

Return to “Documentation, Tips & Tricks”

Jump to
  • Assistance
  • ↳   News & Announcements
  • ↳   Frequently Asked Questions
  • ↳   Installing Gentoo
  • ↳   Multimedia
  • ↳   Desktop Environments
  • ↳   Networking & Security
  • ↳   Kernel & Hardware
  • ↳   Portage & Programming
  • ↳   Gamers & Players
  • ↳   Other Things Gentoo
  • ↳   Unsupported Software
  • Discussion & Documentation
  • ↳   Documentation, Tips & Tricks
  • ↳   Gentoo Chat
  • ↳   Gentoo Forums Feedback
  • ↳   Duplicate Threads
  • International Gentoo Users
  • ↳   中文 (Chinese)
  • ↳   Dutch
  • ↳   Finnish
  • ↳   French
  • ↳   Deutsches Forum (German)
  • ↳   Diskussionsforum
  • ↳   Deutsche Dokumentation
  • ↳   Greek
  • ↳   Forum italiano (Italian)
  • ↳   Forum di discussione italiano
  • ↳   Risorse italiane (documentazione e tools)
  • ↳   Polskie forum (Polish)
  • ↳   Instalacja i sprzęt
  • ↳   Polish OTW
  • ↳   Portuguese
  • ↳   Documentação, Ferramentas e Dicas
  • ↳   Russian
  • ↳   Scandinavian
  • ↳   Spanish
  • ↳   Other Languages
  • Architectures & Platforms
  • ↳   Gentoo on ARM
  • ↳   Gentoo on PPC
  • ↳   Gentoo on Sparc
  • ↳   Gentoo on Alternative Architectures
  • ↳   Gentoo on AMD64
  • ↳   Gentoo for Mac OS X (Portage for Mac OS X)
  • Board index
  • All times are UTC
  • Delete cookies

© 2001–2026 Gentoo Foundation, Inc.

Powered by phpBB® Forum Software © phpBB Limited

Privacy Policy

 

 

magic