Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: emerge speed-up, compile in RAM not on disc.
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
funkyade
n00b
n00b


Joined: 12 Nov 2006
Posts: 33
Location: France (61)

PostPosted: Tue Feb 20, 2007 7:03 am    Post subject: TIP: emerge speed-up, compile in RAM not on disc. Reply with quote

Just had a 'why didn't I think of that before!' moments that I would like to share.

After testing this on a couple of emerges I am quite confident it works AS LONG AS YOU HAVE ENOUGH RAM - enough is 768M or more, although may be okay on 512M with a minimal desktop or X-less server. Tested on three machines with (768M, 1G, and 3G of RAM). The emerge of xorg-server went from 1.5 hours to about 20 minutes on an athlon-xp, for example.

Portage uses /var/tmp/portage (by default) as it's working directory, everything is built in there before it gets merged to /. So, why not stick /var/tmp/portage in RAM? It's a tmp directory after all, saves all that I/O bottleneck...

Interested? Okay, here's what you have to do...
(I assume you are su'd as root, if not prepend sudo to all the following commands)

Code:
# nano /etc/fstab

add the following line to the end -
Code:
none    /var/tmp/portage    tmpfs    nr_inodes=1M    0 0

before we mount it, you may want to clear out your on disc /var/tmp/portage -
Code:
# rm -fdr /var/tmp/portage/*

mount your new tmp directory -
Code:
# mount /var/tmp/portage

try an emerge
Code:
# emerge freeciv

quick wasn't it! ;-)

notes:

1. I haven't had more than 130M used on an emerge yet, that was on an update (emerge -u world), however, I am not sure of the upper limits of the size of this directory during big/multiple emerges. It depends on how often portage/emerge clears it out. Anybody know, and can this behaviour be changed?

2. If you want to set an exact size for this you can change the entry in your fstab to -
Code:
none    /var/tmp/portage    tmpfs    nr_inodes=1M,size=256M    0 0

I would not go below 256M for safety's sake!!! Saw this variant on the Jackass! forums.

--------

ADDENDUM:
1. creating a tmpfs seems to default to 50% of your total RAM size. Not to worry, as it doesn't use much if it isn't populated with files. I'm not sure precisely what will happen if it fills up...... 8O [EDIT: the emerge fails with "No space left on device"! Set size=100M and emerged mysql. mysql peaks at around 140M.]

[EDIT]1a. It would be useful to know what the biggest package in portage is.(?) When emerged how much space does it use at the maximum, if we added, say, 10% to the figure then that would give us our maximum acceptable size for the tmpfs.

2. Most of the space is taken up when emerge unpacks a dist tarball into /var/tmp/portage. It only seems to unpack one package at a time, and then removes it when it has finished. This behaviour may be different dependent on your setting for MAKEOPTS in make.conf, can't confirm this as I have only tested it with "-j2" (the default).

3. There's an entry on the WIKI that I missed that has a very cool script that turns this on and off when you need it, it's http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs


Last edited by funkyade on Tue Feb 20, 2007 1:26 pm; edited 3 times in total
Back to top
View user's profile Send private message
_droop_
l33t
l33t


Joined: 30 May 2004
Posts: 957

PostPosted: Tue Feb 20, 2007 8:27 am    Post subject: Re: TIP: emerge speed-up, compile in RAM not on disc. Reply with quote

Hi,

This is a good idea.

funkyade wrote:
I would not go below 256M for safety's sake!!! Saw this variant on the Jackass! forums.


In the documentation of tmpfs, there is a warning :

Quote:
size: The limit of allocated bytes for this tmpfs instance. The
default is half of your physical RAM without swap. If you
oversize your tmpfs instances the machine will deadlock
since the OOM handler will not be able to free that memory.


So you should not increase too much this parameter.

Of course, for some package (openoffice), tmpfs should be disabled :
Code:
umount /var/tmp/portage
Back to top
View user's profile Send private message
funkyade
n00b
n00b


Joined: 12 Nov 2006
Posts: 33
Location: France (61)

PostPosted: Tue Feb 20, 2007 8:42 am    Post subject: Reply with quote

I agree. Thanks.

Had added to the original post before I read yours...
Back to top
View user's profile Send private message
lost+found
Guru
Guru


Joined: 15 Nov 2004
Posts: 509
Location: North~Sea~Coa~s~~t~~~

PostPosted: Tue Feb 20, 2007 8:52 am    Post subject: Reply with quote

Create some (more) swap. When tmpfs is full, all other mem using apps might need it, and prevents crashing the machine. The last mounted swap partition/loop file get's a lower priority, and would only be used in case of emergency...
Back to top
View user's profile Send private message
funkyade
n00b
n00b


Joined: 12 Nov 2006
Posts: 33
Location: France (61)

PostPosted: Tue Feb 20, 2007 9:03 am    Post subject: Reply with quote

You are suggesting having more than one swap partition, or enlarging what you have?

On my desktop machine (this one) my swap is on another physical drive (3xIDE and 3xSCSI drives, swap is on an old 4G SCSI), on my server (redundant laptop with broken screen) it's a partition on the single hard disk (500M, with 1G RAM). I've never had swap be more than 25 % full... I have swappiness=25 in /etc/sysctl.conf.

Sounds like a good idea all the same!

Thanks.
Back to top
View user's profile Send private message
lost+found
Guru
Guru


Joined: 15 Nov 2004
Posts: 509
Location: North~Sea~Coa~s~~t~~~

PostPosted: Tue Feb 20, 2007 9:10 am    Post subject: Reply with quote

funkyade wrote:
You are suggesting having more than one swap partition, or enlarging what you have? ...

Both is OK. Gentoo installation default is 512 MB swap. If you don't want to repartition, and knowing it will hardly be used, it can be created on an old slow drive, or as a loopback file somewhere... Maybe another 512-1024 MB would be save (perhaps the same amount portage will eat out of your RAM).
Back to top
View user's profile Send private message
funkyade
n00b
n00b


Joined: 12 Nov 2006
Posts: 33
Location: France (61)

PostPosted: Tue Feb 20, 2007 10:28 am    Post subject: Reply with quote

ok.

Got 4G on my desktop machine on one drive, purely because, as you said, it's an old and slow drive and there was little point in partitioning it further. This is massive overkill as I've got 3G of RAM on the machine too.

Going to try compiling openoffice as a benchmark later.

Testing a server setup in a VM on another machine at the moment which I have only allocated 320M of RAM to (to replicate another old laptop), used 'size=256M' in fstab, and so far it has been stable and compiles are nice and quick too. Host is an athlon-xp 2600 running Gentoo.

results are (forgot to add 'time' before I ran them... doh! so have just been looking at the clock) -

ran this first -
Code:
# emerge --fetchonly apache php mysql

to remove any waiting for package retrievals later. Made a snapshot in VMWare.

with no tmpfs for /var/tmp/portage

Code:
# emerge apache php mysql

about 4 hours on a fresh install.

Then went back to snapshot I made earlier.

with tmpfs for /var/tmp/portage

Code:
# emerge apache php mysql

about 1.5 hours on a fresh install.


I'm as happy as Larry... (and he's as pleased as punch)... :-)
Back to top
View user's profile Send private message
JeliJami
Veteran
Veteran


Joined: 17 Jan 2006
Posts: 1086
Location: Belgium

PostPosted: Tue Feb 20, 2007 11:29 am    Post subject: Reply with quote

emerging openoffice requires 5G of disk space!

also see:
https://forums.gentoo.org/viewtopic-t-469758.html
https://forums.gentoo.org/viewtopic-t-491962.html
http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs
https://forums.gentoo.org/viewtopic-t-478658.html
_________________
Unanswered Post Initiative | Search | FAQ
Former username: davjel
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)

PostPosted: Tue Feb 20, 2007 11:55 am    Post subject: Reply with quote

This is nice to compile small things (yeah, xorg is not that big). But it will fail to compile most big things. To name a few: wine, kmail or openoffice. If you have enough swap, the compilation might success, but it will take much more than doing it the traditional way, cause the swap will kill the cpu and disk i/o.
Back to top
View user's profile Send private message
funkyade
n00b
n00b


Joined: 12 Nov 2006
Posts: 33
Location: France (61)

PostPosted: Tue Feb 20, 2007 1:22 pm    Post subject: Reply with quote

Quote:
emerging openoffice requires 5G of disk space!


ouch! Haven't done it as I use Abiword. Maybe not one to try...!

Quote:
http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs

weird, didn't find that when I searched the WIKI to see if this was common knowledge...

Quote:
This is nice to compile small things (yeah, xorg is not that big). But it will fail to compile most big things. To name a few: wine, kmail or openoffice. If you have enough swap, the compilation might success, but it will take much more than doing it the traditional way, cause the swap will kill the cpu and disk i/o.

I updated kmail last night without problem (my tmpfs for /var/tmp/portage is about 1.5G though) but not tried it with a fixed smaller size, nor did I check how much space it was using. I just think it means that for some very big apps like openoffice for example you will have to turn it off... :-(

I think now we have a more modular kde rather the old monolithic packages it makes it easier to split off the apps, so gives more success.
Back to top
View user's profile Send private message
lost+found
Guru
Guru


Joined: 15 Nov 2004
Posts: 509
Location: North~Sea~Coa~s~~t~~~

PostPosted: Tue Feb 20, 2007 1:42 pm    Post subject: Reply with quote

funkyade wrote:
... kde ...
Take the kdeenablefinal USE flag into account (takes lots of more RAM)... I emerged the latest kmail recently, with kdeenablefinal on, and saw a big increase of RAM+swap usage (total about 900 MB incl. cache files, running Gentoo non-graphically). Since I've gotonly 320 MB RAM, this was S-L-O-W. kdeenablefinal produces better optimized code, though... :wink: BTW I'm not using tmpfs for compiling.
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Sat Feb 24, 2007 1:32 am    Post subject: Reply with quote

sounds like it could become a portage feature in the future.
one could create some mechanism that checks how many RAM is in the machine and let portage do most things in RAM.
maybe there would have to be some flag or other information in the ebuilds for that.
Back to top
View user's profile Send private message
sirdilznik
l33t
l33t


Joined: 28 Apr 2005
Posts: 731

PostPosted: Sun Feb 25, 2007 10:59 am    Post subject: Reply with quote

alex.blackbit wrote:
sounds like it could become a portage feature in the future.
one could create some mechanism that checks how many RAM is in the machine and let portage do most things in RAM.
maybe there would have to be some flag or other information in the ebuilds for that.

I was thinking along those lines myself. Maybe a script that calculates the size that would be used by an emerge before it starts and compares it to the space allocated to tmpfs, or if that's not feasible, maybe make a list in a text file of packages that are too large. I'm thinking something along the lines of /etc/portage/package.blacklist or /etc/portage/package.notmpfs. Portage would check that file and if the package about to be emerged was on the list it would unmount tmpfs, emerge the package, then mount tmpfs afterward. Does this sound feasible?
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Sun Feb 25, 2007 12:00 pm    Post subject: Reply with quote

the idea itself is great.
but are you sure that /etc/portage is the right place for this(these) file(s)?
normally in this directory the user places some configuration files.
what we want to do in this case is provide portage some information in the tree.
i believe this should go somewhere in /usr/portage. :roll:
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Wed Feb 28, 2007 1:32 pm    Post subject: Reply with quote

did anybody try how much does it make the emerge faster? here is the catch: emerge time changes very little (my calculations: 5-10%) between tmpfs and ext2 filesystem with 2k block size, mostly because compile times are cpu bound not IO bound. linux kernel doesn't really reach out to disk when you create object files during compile, unless there is a long gap between compile and link. they will stay in cache if you have enough memory.

the real issue is the contradiction: smaller emerges (which are supposedly helped by this mechanism) do not generate enough IO, and larger emerges can't fit completely into the tmpfs memory. So, there is nothing great about this. test it yourself.
Back to top
View user's profile Send private message
vinboy
n00b
n00b


Joined: 18 Jun 2006
Posts: 69

PostPosted: Sat Mar 03, 2007 11:46 pm    Post subject: Reply with quote

this is me. I use 2GB tmpfs
Code:
none                    /var/tmp/portage   tmpfs  size=2000M,nr_inodes=1M         0 0


things work fine though.
I had a peek at memory & swap the system-monitor. swap only get used when memory is full.
when no emerging takes place, every memory related is run in the memory.

i don't see anything wrong with big tmpfs. any ideas?
Back to top
View user's profile Send private message
yamakawa
Guru
Guru


Joined: 28 Jul 2003
Posts: 340

PostPosted: Thu Mar 22, 2007 9:59 pm    Post subject: Reply with quote

devsk wrote:
compile times are cpu bound not IO bound. linux kernel doesn't really reach out to disk when you create object files during compile, unless there is a long gap between compile and link. they will stay in cache if you have enough memory.

the real issue is the contradiction: smaller emerges (which are supposedly helped by this mechanism) do not generate enough IO, and larger emerges can't fit completely into the tmpfs memory. So, there is nothing great about this. test it yourself.


Even though this method does not contribute speedup, isn't it better for longer drive life at least? Especially for those of laptops?
I had my HDD terminated while emerging series of KDE packages. Now that I am going to have a new laptop, I think this method could help the HDD somehow.
Any thought?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Mar 22, 2007 10:42 pm    Post subject: Reply with quote

yamakawa wrote:
devsk wrote:
compile times are cpu bound not IO bound. linux kernel doesn't really reach out to disk when you create object files during compile, unless there is a long gap between compile and link. they will stay in cache if you have enough memory.

the real issue is the contradiction: smaller emerges (which are supposedly helped by this mechanism) do not generate enough IO, and larger emerges can't fit completely into the tmpfs memory. So, there is nothing great about this. test it yourself.


Even though this method does not contribute speedup, isn't it better for longer drive life at least? Especially for those of laptops?
I had my HDD terminated while emerging series of KDE packages. Now that I am going to have a new laptop, I think this method could help the HDD somehow.
Any thought?
modern drives are made to withstand a lot of abuse. I have a WD ATA100 120GB drive which is 4.5 yrs old (mfgd in jun2002), and which I have hammered pretty bad over the years. It just refuses to die! so, your drive died doesn't mean that drives are not supposed to work for long times. We some times feel pity, "oh god, imagine those tiny little heads moving all over the disk to do file seeks and data fetches and writes, and doing it every few milliseconds, over time they will sure be tired and die. god forbid, we do 13 hour OO build on it". These "machines" are designed to do so, over extended periods of time!! so, don't worry, buy a new drive (I reco. WD for reliability, the only brand that hasn't failed on me, yet!) and let your swap and TMPDIR sit on it. And keep it as cool as you can! Most drives die prematurely because of heat, not because of excessive read/write!

anyway, the counter logic to your argument is that once the RAM fills up (with portage tmpfs, other programs and the linux buffer cache), and it does fill up fairly quickly on linux, linux will end up using swap to throw out in-memory programs and buffers to allow for compiler temp files to be written to tmpfs i.e. you end up doing disk IO. Actually, depending upon how you setup swappiness, linux may start to swap even much before all your RAM fills up. So, you are no better than just using disk for the portage TMPDIR.
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Thu Mar 22, 2007 11:34 pm    Post subject: Reply with quote

the harddisk in my internet gateway is a damn old fujitsu 10G drive.
according to S.M.A.R.T. the power-on-time is 20811h+38m+54s. well, it still works. :D and since i use gentoo it has quite a lot to too when doing a "emerge -uDN world" every couple of days... but i do backups, because it can fail every day.
since you gave "thumbs up" for WD drives... the only brand that has never failed for me is seagate. i made the experience that they are reliable, silent, fast. as a bonus the model descriptions follow a understandable scheme, the size of the disk is always printed on the sticker (which is not the case with all manufacturers!) and the jumbers are always good described. so, they are my favorites.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Fri Mar 23, 2007 12:01 am    Post subject: Reply with quote

alex.blackbit wrote:
the harddisk in my internet gateway is a damn old fujitsu 10G drive.
according to S.M.A.R.T. the power-on-time is 20811h+38m+54s. well, it still works. :D and since i use gentoo it has quite a lot to too when doing a "emerge -uDN world" every couple of days... but i do backups, because it can fail every day.
since you gave "thumbs up" for WD drives... the only brand that has never failed for me is seagate. i made the experience that they are reliable, silent, fast. as a bonus the model descriptions follow a understandable scheme, the size of the disk is always printed on the sticker (which is not the case with all manufacturers!) and the jumbers are always good described. so, they are my favorites.
actually, seagate itself never failed on me, but now they own Maxtor as well. And maxtor failed on me twice.
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Fri Mar 23, 2007 12:17 am    Post subject: Reply with quote

yeah, for me too. maxtor is crap.
Back to top
View user's profile Send private message
Akkara
Bodhisattva
Bodhisattva


Joined: 28 Mar 2006
Posts: 6702
Location: &akkara

PostPosted: Fri Mar 23, 2007 1:55 am    Post subject: Reply with quote

I had used a 1500MB ramsf on /var/tmp/portage a long time without problems, including gcc, glibc, wine, kde (but no office). This is with 2GB of physical ram installed.

A few months ago there was some package that needed a lot more space to compile (I don't recall which offhand, it wasn't office it might have been wxGTK).

So I upped it to my current 3GB, along with 3GB of swap.

The big compile went through fine. At its peak using all of memory and 1.5GB into swap. I didn't check whether it was actually faster than simply unmounting the ramfs and using disk directly but it did work.

It is still working well.

Just a datapoint to add to the mix.
Back to top
View user's profile Send private message
yamakawa
Guru
Guru


Joined: 28 Jul 2003
Posts: 340

PostPosted: Fri Mar 23, 2007 1:53 pm    Post subject: Reply with quote

Thanks guys for comments!!

Actually the dead poor HDD was in a small space in a small laptop.
The temperature sometimes went up to 60 C for MB and over 50 C for HDD, the sensor said.
I did emerges almost everyday and for every update!! 8)
When emerging OOo, it took more than two straght days if it ended up with success. :lol:
Maybe I had tortured the poor HDD too much and he simply wanted to kill himself to escape from the weight.
The manufacturer of the HDDs I have used are FUJITSU and HITACHI for laptops. Both died anyway.
The one I reported was of HITACHI, which I had been relied on since the FUJITSU HDD died some years ago.
I chose Maxtor and Seagate for a desktop. They are just fine.

So, I still think Akkara's report is really interesting. My new laptop will be 2GB of physical, so I will try the same and report my benchmark here...in just a few weeks. 8)
Back to top
View user's profile Send private message
Torangan
Apprentice
Apprentice


Joined: 21 Mar 2003
Posts: 178

PostPosted: Fri Mar 23, 2007 3:14 pm    Post subject: Reply with quote

I've used this approach and it did speed up compiles. The data for object files is written to disk rather fast usually as it's dangerous to keep hundreds of MB in RAM. It could stay in read cache but it's not read, just written. Therefore the RAM disk approch gains you less disk access and especially helps for the unpack / install stage. Normall that's copying data disk -> RAM -> disk and now it's disk -> RAM for unpack and RAM -> disk for install. Quite a bit faster and less fragmenting of the file system.
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 475
Location: Flakey Shake & Bake Caliornia, USA

PostPosted: Fri Oct 12, 2007 6:45 pm    Post subject: Reply with quote

Funkyade:

Interesting Tip that I'll have to try (got 4GB) by doing a world rebuild. Will let you know how it works with kdeenablefinal in make.conf using monolithic builds instead of split packages.
_________________
AsRock B550 Phantom Gaming 4
128GB 3200 Mhz memory
4x 4TB Sata - 2x 2TB Sata SSD - 4x 450GB SaS - 3x 900GB SaS - 72GB SaS for Boot
LSI 9211-8i in HBA mode for all of the SaS drives
Radeon 6800 (Non XT) for GPU
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum