in All availableThis forumThis topic
 [HOWTO] Best boot speedup yet!! View unanswered posts View posts from last 24 hours Goto page 1, 2, 3  Next
Author Message
devsk

Joined: 24 Oct 2003
Posts: 2773
Location: Bay Area, CA

Posted: Sun Jul 09, 2006 6:32 am    Post subject: [HOWTO] Best boot speedup yet!!

The idea is to preload the useful files into kernel buffer cache before they are needed. Since I/O can happen parallel to the other CPU intensive tasks during bootup, this leads to faster bootup and faster startup time for your chosen applications. Other projects use static lists to preload the files. But I have automated the process of generating the "useful" prefetch database.

Have lsof installed before you install it and test it out.

 Code: #!/sbin/runscript # Copyright 1999-2006 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header:$ depend() {         before logger } start() {         ebegin "Starting readahead ROYALE"         cat /etc/conf.d/* > /dev/null 2>&1 &         cat cat /etc/conf.d/sofile-list.load > /dev/null 2>&1 &         if [ -f /forcesampler ];then                 /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &                 rm -f /forcesampler         fi         eend 0 } stop() {         eend 0 }

The executable shell script /usr/sbin/sample-init-process:

 Code: #!/bin/bash final_db="$1" # total number of lists to keep in /etc/conf.d/ total_to_keep=4 # total number of samples to make in /forcesampler boot. # It will take twice this amount of seconds to finish sampling. total_samples_per_boot=200 # touch empty file if not there [ ! -f "$final_db" ] && touch "$final_db" let i=0 while [ "$i" -ne "$total_to_keep" ] ; do if [ ! -f "$final_db$i" ] ; then break; else j=$((i+1))                 if [ "$j" = "$total_to_keep" ] ;then                         j=0                 fi                 if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then                         i=$((j)) break; fi fi i=$((i+1)) done if [ "$i" = "$total_to_keep" ] ;then         i=0 fi slot="$i" cp "$final_db" "$final_db$i" function collect_sample_old() {         lsof |awk '{print $9}'|grep \.so|sort|uniq >$1         lsof |awk '{print $9}'|grep \.conf|sort|uniq >>$1         lsof |awk '{print $9}'|grep /bin/|sort|uniq >>$1         lsof |awk '{print $9}'|grep /sbin/|sort|uniq >>$1         lsof |awk '{print $9}'|grep ttf|sort|uniq >>$1         lsof |awk '{print $9}'|grep /libexec|sort|uniq >>$1 } function collect_sample() {         /usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | grep -v "^$" | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" >$1 } collect_sample /tmp/init_sample1 i=0 while [ "$i" -ne "$total_samples_per_boot" ] ; do         collect_sample /tmp/init_sample2         cat /tmp/init_sample1 /tmp/init_sample2 | /usr/bin/uniquer > /tmp/init_sample3         mv /tmp/init_sample3 /tmp/init_sample1         sleep 0.5         i=$((i+1)) done cat /tmp/init_sample1 "$final_db$slot" | /usr/bin/uniquer > "$final_db" rm -f /tmp/init_sample[123]

And this is /usr/bin/uniquer:
 Code: #!/bin/sh /usr/bin/awk '{ if ($0 in stored_lines) x=1 else print stored_lines[$0]=1 }' $1 And then do this:  Code: chmod +x /usr/sbin/sample-init-process /usr/bin/uniquer /etc/init.d/my-readahead rc-update add my-readahead default touch /forcesampler What you have done is asked my-readahead to create a sampler process which samples what files (fonts, binaries, libraries, configs) are in use, every second. It collects 200 samples (will be busy for at least 10 minutes after boot) and keeps updating the database with a list of useful files. Once you boot after this the first time, you will probably see a slow down because its collecting samples for use on the next and subsequent boots. Use your system as you normally would. Open firefox, gaim, OO or whatever. Once the sampler script has finished ('ps -aef|grep -v grep|grep sample-init-process' will tell you if its running), reboot. Now this and all subsequent boots will be much faster. <update> Something has changed with baselayout recently and requires you to hit enter when the sampler starts during the once-only boot for sample collection. </update> If it takes too long for the sample-init-process to finish, reduce the total_samples_per_boot variable in the sampler script to 100 and try again by touching /forcesampler. In future, whenever you feel like updating the prefetch database, just 'touch /forcesampler' and database will be updated on next reboot. Uninstall is easy:  Code: rc-update del my-readahead \rm /etc/init.d/my-readahead \rm /usr/sbin/sample-init-process \rm /etc/conf.d/sofile-list.load* \rm /usr/bin/uniquer Last edited by devsk on Sat Aug 19, 2006 2:10 am; edited 9 times in total johntash n00b Joined: 30 Apr 2005 Posts: 20 Location: KS  Posted: Sun Jul 09, 2006 6:42 am Post subject: I'll give it a try =] suicidal_orange_II Apprentice Joined: 04 Sep 2004 Posts: 299  Posted: Sun Jul 09, 2006 11:11 pm Post subject: Me too - the idea sounds good and anything that speeds up booting (and program loading?) must be good Suicidal_Orange devsk Advocate Joined: 24 Oct 2003 Posts: 2773 Location: Bay Area, CA Posted: Mon Jul 10, 2006 1:04 am Post subject: I made it a howto and put more work into automating it after finding it shaved off close to 15 seconds off of my startup time. I strongly believe that if we have to preload something, it has to be something which will be used. And that's what this script tries to do. I am down to 40 seconds for a full fledged KDE desktop with initrd and splash enabled at boot and tonnes of services like UPS, sensors, hddtemp etc. starting at default. Moreover, firefox and gaim start really fast when I get my desktop. I am thinking this might be a better sampler, more concise and faster:  Code: /usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | sort | uniq | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" What do you guys think? cypherphreak n00b Joined: 23 May 2005 Posts: 17  Posted: Wed Jul 26, 2006 3:05 pm Post subject: Mine appears to have not worked. The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load & from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the$1 value in the script which is where final_db is stored correct?
devsk

Joined: 24 Oct 2003
Posts: 2773
Location: Bay Area, CA

Posted: Thu Jul 27, 2006 4:58 am    Post subject:

what does
show? is there a list of files in there?
jit_v1
n00b

Joined: 16 Dec 2004
Posts: 24

 Posted: Fri Jul 28, 2006 9:37 am    Post subject: Thanks for the howto. I've just tried using it but unfortunately don't seem to have gained anything from my boot time The sofile-list.load seems to have been created correctly etc. and contains 180 files. I noticed that the rc-script is being loaded in the default runlevel, could this be moved further up so that it starts earlier? It's probably worth altering the rc-script so that it doesn't do the preload if /forcesampler exists - otherwise the sampler will pickup previously sampled files that may no longer be required. I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me. jv
devsk

Joined: 24 Oct 2003
Posts: 2773
Location: Bay Area, CA

Posted: Fri Jul 28, 2006 2:42 pm    Post subject:

 jit_v1 wrote: I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.
ok, this is a bug. Did you rebuild the database after doing this? I think it didn't capture much for you. Was it still sampling after you had your system up? For me, the list has 500 filenames, all of those files are the ones which are used during the boot and afterwards. So, it helps me a lot.
jit_v1
n00b

Joined: 16 Dec 2004
Posts: 24

 Posted: Fri Jul 28, 2006 3:50 pm    Post subject: Yeah, I rebuilt the database after, got an extra 20 or so files, but still didn't seem to make much difference to the startup time. The sampler process stayed running after bootup for around five minutes. I did notice that my boot time wasn't affected that much when the sampling was going on - you mentioned a delay of 10 minutes to boot - it took just a little longer than normal. Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial? jv
thomasvk
Guru

Joined: 19 Mar 2005
Posts: 597

 Posted: Sat Jul 29, 2006 1:37 pm    Post subject: So if you put something in /dev/null it will be cached?
drwook
Veteran

Joined: 30 Mar 2005
Posts: 1322
Location: London

Posted: Sat Jul 29, 2006 1:58 pm    Post subject:

 t0maz wrote: So if you put something in /dev/null it will be cached?

Sure, whatever it reads gets buffered/cached. so cat'ing it to /dev/null is as good a way as any. Could probably just as well grep it for the word 'wibble' and redirect the output to /dev/null or something, but that'd just look odd in a script
devsk

Joined: 24 Oct 2003
Posts: 2773
Location: Bay Area, CA

Posted: Sat Jul 29, 2006 2:54 pm    Post subject:

 jit_v1 wrote: Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial? jv
yeah, your system is too fast I think, so sampling at probably 0.5 or even 0.2 seconds might give you better results. Note that reducing the sampling interface will be more intrusive while collecting the first time. Also, you might have to up the number of samples because now it will finish in half the time with 0.5 sleep instead of 1 sec sleep. You don't wanna finish too early either.

Another reason for different size database is that you are probably running a different set of services than me.

One way of finding if its working or not is: disable the service my-readahead. Reboot and login into the box. Inside X, do a top and see how much memory is used. Note it down. Enable the service, reboot and login into the box and again do a top to see the memory used. This number should be substantially higher, but close to what you will eventually use if you used your system normally. You can compare o/p of
 Code: cat /proc/meminfo
for these two scenarios and it will clearly indicate that you have loaded extra stuff (all useful because you were using that stuff when we sampled it).
stahlsau
Guru

Joined: 09 Jan 2004
Posts: 565
Location: WildWestwoods

 Posted: Sun Jul 30, 2006 7:56 am    Post subject: gj, thanx for the script, i like the idea At first, i thought you'd be kidding us with cat'ing the files to /dev/null. But after a lil thinking ... _________________Join the adopt an unanswered post initiative today
balk
n00b

Joined: 10 Apr 2002
Posts: 62
Location: Olanda

Posted: Sun Jul 30, 2006 6:36 pm    Post subject:

 cypherphreak wrote: Mine appears to have not worked. The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load & from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the \$1 value in the script which is where final_db is stored correct?

Do you have lsof installed? I didn't and the script obviously did not work. I needed another cycle before it worked; I had zero-byte /etc/conf.d/sofile-list.load and /etc/conf.d/sofile-list.load0 files which I had to delete first.

I will now reboot and find out if there is any increase in speed.

There seems to be ~some~ increase. I did not measure but it felt faster Thanks for the nice script!
_________________
dway
n00b

Joined: 18 Jun 2006
Posts: 35

 Posted: Sun Jul 30, 2006 11:56 pm    Post subject: Hi, I'm new on Gentoo (and on linux in general) and i have a fully fonctionnal system now (learning from the documentations, wiki and forum). Just a little question : does this script break something if I try to use it or is it safe for the system (by the way how do you clear it form the system ?)
devsk

Joined: 24 Oct 2003
Posts: 2773
Location: Bay Area, CA

Posted: Mon Jul 31, 2006 1:34 am    Post subject:

dway, if you can't figure out what its doing, you should learn to figure that first. The script is harmless but I would recommend you read up and figure what it is doing before going installing it. You need 'lsof' installed. There are only couple of files it creates. Its easy to get rid of the whole thing by just doing:
dway
n00b

Joined: 18 Jun 2006
Posts: 35

 Posted: Mon Jul 31, 2006 9:21 am    Post subject: thx for infos, I'll try to figure it out before trying, regarding your advice (sorry for my english).
Ic3M4n

Joined: 02 Nov 2004
Posts: 3489
Location: Bergamo.

Posted: Mon Jul 31, 2006 12:30 pm    Post subject:

Hi all!
I have a problem starting the init script
I have an output like this:
 Code: Could not get dependency info for "my-readahead"! Please run: # /sbin/depscan.sh to try and fix this

if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?
devsk

Joined: 24 Oct 2003
Posts: 2773
Location: Bay Area, CA

Posted: Mon Jul 31, 2006 3:17 pm    Post subject:

Ic3M4n wrote:
Hi all!
I have a problem starting the init script
I have an output like this:
 Code: Could not get dependency info for "my-readahead"! Please run: # /sbin/depscan.sh to try and fix this

if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?
what baselayout, sysvinit, udev versions are you using?
cypherphreak
n00b

Joined: 23 May 2005
Posts: 17

Posted: Tue Aug 08, 2006 6:31 pm    Post subject:

 balk wrote: Do you have lsof installed?

Simple things like that are always overlooked by me, thanks for pointing that out, works fine now.
thomasvk
Guru

Joined: 19 Mar 2005
Posts: 597

 Posted: Wed Aug 09, 2006 3:25 pm    Post subject: So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?
prymitive
Apprentice

Joined: 13 Jun 2004
Posts: 260

Posted: Wed Aug 09, 2006 3:30 pm    Post subject:

 t0maz wrote: So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?

It's not about dumping some output stuff to /dev/null but reading files from disk so they are put in buffers so when some app want to read them it gets it from buffer with is a lot faster then reading from disk. You think that /dev/null is some buffor or am I stupid?
thomasvk
Guru

Joined: 19 Mar 2005
Posts: 597

Posted: Wed Aug 09, 2006 3:36 pm    Post subject:

 prymitive wrote: You think that /dev/null is some buffor or am I stupid?

You're stupid!

No, I'm sleeping...
prymitive
Apprentice

Joined: 13 Jun 2004
Posts: 260

Posted: Wed Aug 09, 2006 3:41 pm    Post subject:

t0maz wrote:
 prymitive wrote: You think that /dev/null is some buffor or am I stupid?

You're stupid!

No, I'm sleeping...

Don't get me wrong, I wasn't sure if I understood Your question, it's been long time since my english lessons .
Dr.Dran
l33t

Joined: 08 Oct 2004
Posts: 766
Location: Imola - Italy

 Posted: Thu Aug 10, 2006 7:49 pm    Post subject: @devsk Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc Thanx P.S. I your patch work better you may submit it to Uberlod (the baselayout dev) Cheers Franco_________________:: [Dr.Dran] Details :: - Linux User # 286282 - IT FreeLance Consultant - President of ImoLUG [Imola & Faenza Linux User Group]
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
 All times are GMTGoto page 1, 2, 3  Next Page 1 of 3

 Jump to: Select a forum Assistance----------------News & AnnouncementsFrequently Asked QuestionsInstalling GentooMultimediaDesktop EnvironmentsNetworking & SecurityKernel & HardwarePortage & ProgrammingGamers & PlayersOther Things GentooUnsupported Software Discussion & Documentation----------------Documentation, Tips & TricksGentoo ChatGentoo Forums FeedbackOff the WallDuplicate Threads International Gentoo Users----------------中文 (Chinese)DutchFinnishFrenchDeutsches Forum (German)  Diskussionsforum  Deutsche DokumentationGreekForum italiano (Italian)  Forum di discussione italiano  Risorse italiane (documentazione e tools)Polskie forum (Polish)  Instalacja i sprzęt  Polish OTWPortuguese  Documentação, Ferramentas e DicasRussianScandinavianSpanishOther Languages Architectures & Platforms----------------Gentoo on AMD64Gentoo on PPCGentoo on SparcGentoo on Alternative ArchitecturesGentoo for Mac OS X (Portage for Mac OS X)
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum