Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
GOPreload
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Mon Nov 26, 2007 11:01 am    Post subject: GOPreload Reply with quote

Description:
Got tired of the way the kernel decides what to keep cached and what not, expecially after watching a DVD or copying large files,
i decided to write two simple scripts which keep the applications i use most in the system cache by mmapping and mlocking them in memory.


Requirements:
mapandlock (precompiled i686 and amd64, as well as sourcecode included)
strace
readlink
basename
dirname
bash
superuser privileges

Installation:

Simply put GOPreload directory somewhere on your HD, but don't forget it :-p

Usage and example:
First of all, you must think on a few applications which you REALLY need to be very responsive on starting.
For my everyday use i could think to firefox:

--
I will use then 'Prepare.sh' that way:
cd GOPreload
./Prepare.sh firefox
(wait for Firefox to fully load and press ENTER in the shell)

Now Prepare.sh has created a new file called firefox.openfiles.txt in the enabled/ folder.
The generated filename includes an estimation of the MB wich will stay resident in ram.
It contains the list of files which firefox needs to start-up, the idea is to keep them in ram so that next time firefox will start,
it will not read them from disk and will start-up much, much faster.

You can reiterate Prepare.sh as much as you like, think at konqueror, nautilus, kwrite, gedit and so on, but don't do too much, because
your memory is limited.

***NOTICE:***
As soon as you update firefox or it's needed libraries, you will need to (re)Prepare it again!
--

Now that you 'prepared' your programs,you need to use 'StartPreload.sh'
cd GOPreload
sudo ./StartPreload.sh
Just sit and wait for StartPreload.sh to preload the file lists you created with Prepare.sh, next time (and forever) you will launch
the programs, you choose, you will notice a nice speed boost, even if you're short of memory.
StartPreload.sh can be launched with debug parameter:
./StartPreload.sh debug
...to see some interesting things.
--

You may have noticed that there are two folders under MOPreload:
* GOPreload/enabled/ meaning should now be clear
* GOPreload/disabled/ is empty, you can use it to move filelists
'on the fly' from enabled/ to disabled/ folder, StartPreload.sh will notice the changes and will react accordingly.


Configuration Variables:
Prepare.sh:
* MAXMB is the Maximum filesize (in MB) to preload at 'prepare' stage.
Files bigger than MAXMB will NOT be included in the list.
Small files tend to slow down things more than big ones, big files will use a lot of ram, so i suggest you to keep
it to a value not greater than 15MB.
* EXCLUDE_LIST contains a list of file extension you wan't preload.
Maybe you would like to exclude avi, mp3 and so on...
Please, refer to Prepare.sh to learn the syntax.

StartPreload.sh
* MAXMB is the Maximum filesize (in MB) to preload at 'preload' stage.
files bigger than MAXMB will NOT be preloaded, even if they are in the list created by Prepare.sh, set it to "-1" to disable the feature.
* LONG_DELAY is the time (in seconds) to check if enabled/ disabled/ content has changed, don't set it too low or leave as default.


Suggestions:
*As StartPreload.sh could be very heavy the first time it starts
(i use it to load about 150MB of files), i suggest you to launch it with nice and ionice at boot time in your rc scripts using something like that:
nice -n 19 ionice -c3 sh -c 'path/GOPreload/StartPreload.sh'

*Please note that even if you have, say, prepared konqueror which takes 20MB of ram and kwrite which takes 14, it is more likely that
if they are preloaded togheter, they will share a lot of (kde)libs, so you can expect them togheter to keep only a little more of 20MB.

*Please note that programs which you prepare can't be swapped to swapfile.

Some Benchmarks; Start Application Times
Program|WithPreload|WithoutPreload
oowriter|0m3.502s|0m11.768s
thunderbird|0m3.704s|0m12.675s
firefox|0m5.038s|0m16.160s
winecfg|0m2.559s|0m5.032s
kwrite|0m1.166s|0m3.045s

You can download it here:
http://wpage.unina.it/aorefice/GOPreload-0.5.tar.gz


Last edited by kokoko3k on Sun Jan 13, 2008 9:24 pm; edited 17 times in total
Back to top
View user's profile Send private message
BradN
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2383
Location: Wisconsin (USA)

PostPosted: Tue Nov 27, 2007 6:33 am    Post subject: Reply with quote

I wonder if it would be a better option to mmap the files and then read them once... this might be enough to discourage linux from dropping them from memory unless memory pressure gets really bad (and then, you probably don't want them to stay in memory!) You might even be able to lock the pages in memory too, but I'm not really sure how this works. It would be a good idea to periodically re-map the files in case they were replaced on-disk (linux won't delete a file if it's still used, but will happily remove it from the directory tree)

If you wanted, this would also let you re-read the whole set slowly and avoid any sudden CPU/disk usage (even for large files), although this part could be done in bash almost as easily.

Then again, you'd probably have to write it in C or something to do this, since I don't think you can mmap from a bash script...
Back to top
View user's profile Send private message
mark_alec
Bodhisattva
Bodhisattva


Joined: 11 Sep 2004
Posts: 6066
Location: Melbourne, Australia

PostPosted: Tue Nov 27, 2007 9:04 am    Post subject: Reply with quote

Moved from Desktop Environments to Unsupported Software.
_________________
www.gentoo.org.au || #gentoo-au
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Tue Nov 27, 2007 5:05 pm    Post subject: Reply with quote

BradN wrote:
I wonder if it would be a better option to mmap the files and then read them once... this might be enough to discourage linux from dropping them from memory unless memory pressure gets really bad (and then, you probably don't want them to stay in memory!) You might even be able to lock the pages in memory too, but I'm not really sure how this works. It would be a good idea to periodically re-map the files in case they were replaced on-disk (linux won't delete a file if it's still used, but will happily remove it from the directory tree)[

If you wanted, this would also let you re-read the whole set slowly and avoid any sudden CPU/disk usage (even for large files), although this part could be done in bash almost as easily.

Then again, you'd probably have to write it in C or something to do this, since I don't think you can mmap from a bash script...

I doubt pmap with no lock would help, anyway with the actual script you can read the files as slow as you want by raising the "DELAY" value, but i feel better with that set to 0 and LONG_DELAY to 60sec.
Anyway you're right, mmap() with MAP_LOCKED (because i want, say, the filemanager to be always responsive, even if memory pressure is high) would do a better job, but I've no clue about programming in C, so i will call some friends to get help.
I will then replace:
Code:

for file in $LISTAFILE;
do
   cp  -L $file /dev/null 2>/dev/null
   sleep $DELAY
done

with something like:
Code:

  keeploaded filelist.txt


Thanks!
Back to top
View user's profile Send private message
Zacho
n00b
n00b


Joined: 28 Nov 2007
Posts: 5

PostPosted: Wed Nov 28, 2007 5:40 pm    Post subject: Re: M.y O.wn way to preload. Reply with quote

kokoko3k wrote:
Got tired of the way the kernel decides what to keep cached and what not, expecially after watching a DVD, or copying large files,...


I feel exactly the same. I have 2gb of ram and the programs i use most of the time use only 200-500mb. I should have enough ram to load some of my favourite programs there and keep them there.

I tried your script and it seems to be working ok. I got some ideas and thought i share it with you.

- Would it be possible to make the script(startpreload.sh) to check memory after each cycle? If i am running out of memory i see no reason for it to try to copy the files anymore.

- I think the script works better if you run it with nice/ionice and change the priority level. Just to make sure it doesn't affect computer performance.
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Thu Nov 29, 2007 9:12 am    Post subject: Re: M.y O.wn way to preload. Reply with quote

Zacho wrote:
- Would it be possible to make the script(startpreload.sh) to check memory after each cycle? If i am running out of memory i see no reason for it to try to copy the files anymore.

Maybe it could come in only when the total free memory is under a specified amount and the sum of buffers+cached ram is lower than a threshold, but if i manage to use mmap() to map and lock those files in memory (read BradN suggestion), the entire cycle will be useless, as they will never be paged, staying in the cache forever, or until the script got killed.
No need to copy them again, just read them once and map them locked in ram.

Quote:
- I think the script works better if you run it with nice/ionice and change the priority level. Just to make sure it doesn't affect computer performance.

Personally, i use this in /etc/conf.d/local.start:
Code:
nice -n 19 sh -c '/root/scripts/KdeInRam/StartPreload.sh' 1>/dev/null 2>/dev/null &
sleep 5
ionice -c3 -p`ps -e|grep StartPreload|cut -b 0-6`

...Because i didn't find a way to use ionice and nice in the same command.
But again, this could became useless without cycles.
Back to top
View user's profile Send private message
Zacho
n00b
n00b


Joined: 28 Nov 2007
Posts: 5

PostPosted: Thu Nov 29, 2007 10:35 am    Post subject: Re: M.y O.wn way to preload. Reply with quote

kokoko3k wrote:
Maybe it could come in only when the total free memory is under a specified amount and the sum of buffers+cached ram is lower than a threshold, but if i manage to use mmap() to map and lock those files in memory (read BradN suggestion), the entire cycle will be useless, as they will never be paged, staying in the cache forever, or until the script got killed.
No need to copy them again, just read them once and map them locked in ram.


That would be really nice if somebody can actually do the program. Until that happens, lets make the script run better. :wink: I think i can do the mem check myself. I know some basic scripting.

Quote:
Personally, i use this in /etc/conf.d/local.start:
Code:
nice -n 19 sh -c '/root/scripts/KdeInRam/StartPreload.sh' 1>/dev/null 2>/dev/null &
sleep 5
ionice -c3 -p`ps -e|grep StartPreload|cut -b 0-6`

...Because i didn't find a way to use ionice and nice in the same command.
But again, this could became useless without cycles.


I just run it like this: nice ionice -n 7 startpreload.sh
Back to top
View user's profile Send private message
BradN
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2383
Location: Wisconsin (USA)

PostPosted: Fri Nov 30, 2007 2:05 am    Post subject: Reply with quote

Hmm, I'd never heard of ionice... what package is it in?
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Fri Nov 30, 2007 8:18 am    Post subject: Reply with quote

It's in sys-apps/util-linux, you need Kernel version >=2.6.13 and CFQ as io scheduler.
Back to top
View user's profile Send private message
yoyo
Bodhisattva
Bodhisattva


Joined: 04 Mar 2003
Posts: 4273
Location: Lyon - France

PostPosted: Fri Nov 30, 2007 8:21 am    Post subject: Reply with quote

Well I'm not really sure that does the same but what about : sys-apps/readahead-list included in portage ?

Is there any difference between yours ? What are they ?

Enjoy !

PS : There's an interesting topic about preload : How to get preload works with gentoo.

EDIT : the wiki page about readahead-list.
_________________
La connaissance s'accroît quand on la partage.
JCB
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Fri Nov 30, 2007 2:33 pm    Post subject: Reply with quote

EDIT
Post Removed, mmap version seems to be less effective, and i need to find why.
@yoyo: thanks for pointing me to that thread!
Back to top
View user's profile Send private message
Zacho
n00b
n00b


Joined: 28 Nov 2007
Posts: 5

PostPosted: Fri Nov 30, 2007 6:31 pm    Post subject: Reply with quote

kokoko3k wrote:
SuperPreload (from Suse):
* Never heard of it before you told me about!
* Seems to use a similar approach
* I don't know if it locks memory
* Is not easy to add new programs to preload, you'll have to do it manually.


The last one is not true.

http://en.opensuse.org/SUPER_preloading

There are a couple of pepr scripts on that page that can help you do the filelist in a similar manner that prepare.sh does it. When you execute the pepr_update.sh script it asks you to run all the programs and press enter when finished. Then it does the list. I think this method is more flexible because now i can run whatever progs/files i like or just one program. I think you should check it out.

The list it creates seems to be a little bit different. I don't think i can use the list like that with your program.
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Fri Nov 30, 2007 8:06 pm    Post subject: Reply with quote

On the other side, i think that you can't select on the fly what to keep, what to leave, while with Prepare.sh you only need to move a file from enabled to disabled directory.
If you like Super Preload approach, you can do something like:
* Prepare.sh kdm (gdm, xdm...)
* open all program you like in that session
* log-out from session manager.
* Let prepare.sh fills the file list.
...Or, of course, use Super preload : -P

It was the method i used in the beginning (prepare.sh kdm) , but i feel more comfortable now.
Anyway, it's funny how similar are the approaches!

Did you manage to try them both (opensuse one and mopreload)?
What d'you think about?
Back to top
View user's profile Send private message
Zacho
n00b
n00b


Joined: 28 Nov 2007
Posts: 5

PostPosted: Fri Nov 30, 2007 9:20 pm    Post subject: Reply with quote

I have tried both methods(super_preload and mopreload). Super preloading doesn't keep the files in cache. It just preloads a list of files during boot up and thats it. PePr is a way to get more programs/files to preload during boot, but it doesn't do anything else. Pepr is just there so that suse users don't have to do the list manually.

Your script is a little bit better because it tries to keep the files cached all the time. Of course, that would be very easy to add to the super_preload method.

I like your script better because of the enabled/disabled folder idea. So nothing wrong with the start_preload script. And yes, i definitely like the idea of having a different file list for every program.

The problems i had was while doing a list. When i run prepare.sh sometimes it works and sometimes it doesn't do the job. Or it just takes a long long time to do it. The actual reason why i liked pepr better was because it asked you to press ENTER and then killed the strace when you did press enter. So if the strace hangs or something it doesn't matter cos it kills the process and continues running the script. Seems to be working better that way.

Also there seems to be a lot of useless stuff on the list when i use your script (prepare.sh). I have to manually remove some of the lines. Pepr seems to be doing the parse process better.

Right now i am using modified StartPreload.sh(added mem check) to run on the background. Still trying to find the best method of creating a filelist.
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Fri Nov 30, 2007 11:54 pm    Post subject: Reply with quote

It's strange that Preload.sh does not the job in some circumstances, can you give me an example so i can try myself?
Please note that as soon as the program you want to cache pop up, you have to close it (maybe you can open some menu and other little things), but don't use it as normal, or else Preload.sh will have to process tons of files you probably don't even need.
Or maybe that task did not ends, try with killall -HUP programname.
Quote:
The actual reason why i liked pepr better was because it asked you to press ENTER and then killed the strace when you did press enter.

...Or i can add [PRESS ENTER] and kill strace too, quite easy, but it would be better to find out what makes strace hangs, so some examples from you could be very useful.

Anyway, yes, there is some garbage left on the list which is created, but that's normal; the more the trash, the slower the apps will load.
Also, the smallest files will incredibly slow down accesses because of the HD seek time, that's why i decided to keep them and let the user specify only a maximum filesize for the files to be preloaded; but i forgot to remove browser caches. :-)

Actually, StartPreload.sh will copy files to /dev/null with "cp -H" which does the job well (copy the entire file, not just the link reference), while StartPreload.sh version which uses mmap (which i removed in the evening) copies just the link, so nothing ; that's why i spoke about bad performances using mmap.
I'll backtrace them using readlink in the next version.
Would you like to post your Version of StartPreload.sh with memory limits?

Thanks, koko.


Last edited by kokoko3k on Sun Dec 30, 2007 10:22 pm; edited 1 time in total
Back to top
View user's profile Send private message
Zacho
n00b
n00b


Joined: 28 Nov 2007
Posts: 5

PostPosted: Sat Dec 01, 2007 9:28 am    Post subject: Reply with quote

kokoko3k wrote:
Or maybe that task did not ends, try with killall -HUP programname.


I have a very strong feeling that the problem was related to that. Tasks didn't end or they end very slowly. I tested the script more and had no problems today. Weird.

Anyway i added this:
Code:
strace -f -F -e trace=open,access $1 2> /tmp/out &

echo "When you are finished, press ENTER."
read

#stop the tracing
killall strace
sleep 1


Now i can continue if something hangs. But i think i need to change the script even more -> How do i start programs with wine?

Quote:
Would you like to post your Version of StartPreload.sh with memory limits?


I just did it like this:
Code:
#MemLimit
#When memorylimit has been reached script will stop copying
   MemLimit="1500"

while [ 1 ];
do
AllUsedMem=`free -om | grep Mem | awk '{print $3}'`
CachedMem=`free -om | grep Mem | awk '{print $7}'`
let UsedMem=($AllUsedMem - $CachedMem)

if [ $UsedMem -lt $MemLimit ]; then

   LISTAFILE=`cat enabled/*.openfiles.txt|sort|uniq`

   if [ "$1" = "debug" ]; then

         TOTSIZE=0
         SIZE=0
         COUNT=0

---your code here---


It doesn't check buffers. Maybe it should tho. Honestly, i don't even know what the buffers are. 8)
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Tue Dec 04, 2007 9:01 am    Post subject: Reply with quote

* Added [Press enter] to Prepare.sh to force stalled applications to end and start computing opened files.
* Used readlink in Prepare.sh to manage symbolic links.
* File lists now contains MB estimations in filenames.
* Added MAXMB variable to StartPreload.sh too
* Removed DELAY variable in StartPreload.sh, useless now.
* StartPreload.sh now uses mapandload, a little tool i wrote using lazarus fpc (-edit- C version is on the way) which keeps the files resident in ram by using mmap and MAP_LOCKED, so copy to /dev/null is gone.


Last edited by kokoko3k on Thu Jan 03, 2008 10:56 pm; edited 3 times in total
Back to top
View user's profile Send private message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Thu Jan 03, 2008 10:50 pm    Post subject: GOPreload. Reply with quote

Changed the name and updated the first post.
Some more changes this time:

* Added documentation to archive.
* Mapandlock now is pure C, i included sourcecode as well as precompiled i686 and amd64 binaries.
* Don't load the same files every cycle if filelists is the same.
* Added some more debug messages.
(number of files not found per list, useful for you to decide if you must (re)Prepare your applications).
* MoPreload chenged name, is now GOPreload.

Download Link in the first post, thanks.
Back to top
View user's profile Send private message
Small_Penguin
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2005
Posts: 120

PostPosted: Thu Aug 20, 2009 11:22 am    Post subject: Reply with quote

Hi, I've tried to compile mapandlock.x86_64 (AMD64), and even if it succeeds and I get a binary, I get the following error in dmesg:

Code:

mapandlock.x86_[2598] general protection ip:3d7a271911 sp:7fff8735c000 error:0 in libc-2.10.1.so[3d7a200000+16d000]


There is no mapandlock.x86 in the process list, obviously it doesn't work.

Any ideas?
Back to top
View user's profile Send private message
Small_Penguin
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2005
Posts: 120

PostPosted: Wed Sep 02, 2009 7:54 am    Post subject: Reply with quote

I've solved this myself. The problem were some strange files which did not exist but caused a segmentation fault:

Code:

xfce4-settings-manager.openfiles-66MB.txt:/usr/share/X11/xkb/rules/evdev-LC_CTYPE=de_DE.UTF-8;LC_NUMERIC=de_DE.UTF-8;LC_TIME=de_DE.UTF-8;LC_COLLATE=de_DE.UTF-8;LC_MONETARY=de_DE.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=de_DE.UTF-8;LC_NAME=de_DE.UTF-8;LC_ADDRESS=de_DE.UTF-8;LC_TELEPHONE=de_DE.UTF-8;LC_MEASUREMENT=de_DE.UTF-8;LC_IDENTIFICATION=de_DE.UTF-8
xfce4-settings-manager.openfiles-66MB.txt:/usr/share/X11/xkb/rules/evdev-LC_CTYPE=de_DE.UTF-8;LC_NUMERIC=de_DE.UTF-8;LC_TIME=de_DE.UTF-8;LC_COLLATE=de_DE.UTF-8;LC_MONETARY=de_DE.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=de_DE.UTF-8;LC_NAME=de_DE.UTF-8;LC_ADDRESS=de_DE.UTF-8;LC_TELEPHONE=de_DE.UTF-8;LC_MEASUREMENT=de_DE.UTF-8;LC_IDENTIFICATION=de_DE.UTF-8.lst


Maybe the filenames are too long or there are bad characters in there which mapandlock cannot handle. After removing these files from the list, gopreload works again!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum