Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
RAM disks and Portage = more speed ?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
dennis
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jun 2002
Posts: 84

PostPosted: Sat Oct 12, 2002 2:22 am    Post subject: RAM disks and Portage = more speed ? Reply with quote

Hi,

I guess you all know that slow Harddisks, can affect the execution-speed of some programms really bad ;) To 'save acces times' the UT-Live CD can 'cache' itself in the computers memory. Now i thought that I could as well speed up portage by loading its temp directories in a RAM disk ... what directories does portage compile in? would this be worth it?

dennis schaaf
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Sat Oct 12, 2002 3:19 am    Post subject: Re: RAM disks and Portage = more speed ? Reply with quote

dennis wrote:
what directories does portage compile in?

/var/tmp/portage by default. Configurable by setting PORTAGE_TMPDIR.
Quote:
would this be worth it?

IMO, only if it were able to have several scratch directories, and fail over seamlessly when one filled up. Otherwise, when compiling big packages like X you had better have a boatload of RAM.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Sat Oct 12, 2002 3:28 am    Post subject: Re: RAM disks and Portage = more speed ? Reply with quote

rac wrote:
when compiling big packages like X you had better have a boatload of RAM.

Hehe... That's right: http://forums.gentoo.org/viewtopic.php?t=6607
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
dennis
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jun 2002
Posts: 84

PostPosted: Sat Oct 12, 2002 3:58 pm    Post subject: makes sence Reply with quote

hi

yea i can immagine that it requires a lot of space ...

but would it be possible to create such a fail over system?

[random idea]maybe you could software raid the RAM disk and a harddisk but that seems to be a more complicated than neccessary method for me ...[/random idea]

dennis
Back to top
View user's profile Send private message
phypor
n00b
n00b


Joined: 25 Jun 2002
Posts: 68
Location: Texas

PostPosted: Sun Oct 13, 2002 2:08 am    Post subject: RAMdisk vs kernel caching Reply with quote

dennis wrote:
hi

yea i can immagine that it requires a lot of space ...

but would it be possible to create such a fail over system?

[random idea]maybe you could software raid the RAM disk and a harddisk but that seems to be a more complicated than neccessary method for me ...[/random idea]

dennis


The OS cacheing will (very smartly) handle file caching for you...
The more free ram you have, the more it will grab for file cache, which will be released as need for application memory.

run free or top in another console when you emerge, youll see the 'cached' bouncing around as files are cached and purged.

ram disks for file access intensive stuff are not the best way...
let the kernel handle your excess ram with dynamic caches.

(until you get over a certain amt of ram (1 gig? 4?) which will be used automatically as ram disk for swapping only)

~phypor
_________________
ALWAYS stay away from tanks enguled in fire.
- 2000 Emergency Response Guidebook, pg307
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Fri Nov 15, 2002 11:39 am    Post subject: Re: makes sence Reply with quote

dennis wrote:
yea i can immagine that it requires a lot of space ...

but would it be possible to create such a fail over system?

[random idea]maybe you could software raid the RAM disk and a harddisk but that seems to be a more complicated than neccessary method for me ...[/random idea]

What you want is already done for you with the tmpfs filesystem. It is a file system that resides in ram but can be paged to disk as the OS deams necessary... Just make sure you have a LARGE enough swap space for compiles (over 2 gig for Open Office) and
Code:

mount tmpfs /var/tmp/portage -t tmpfs -o size=1250m

You will need tmpfs compiled into your kernel for this to work... By default it only uses half the available memory though so you'd want to use the size paramter to set it to RAM + SWAP - some sane amount so it wouldn't be possible to use 100% of storage for tmpfs...

When I get around to repartitioning my system (again) I think I might set it up this way... I have 768mb of RAM. I wouldn't recommend this for < 512mb...

EDIT: OK, I just created a 1.25gig tmpfs mounted at /var/tmp/portage (I have 1.5 gig including swap). Now to find something to compile ;-)
Back to top
View user's profile Send private message
dennis
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jun 2002
Posts: 84

PostPosted: Fri Nov 15, 2002 11:31 pm    Post subject: Reply with quote

can you compile whatever you plan to compile twice? once with tmpfs and once without?

dennis
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Sat Nov 16, 2002 12:44 am    Post subject: Reply with quote

Hehehe, for a real speed comparison, eh? I imagine it's more subjective than actual performance... The only real speed increase would come from less disk I/O, and I have a decently fast disk... for a slow HD I think there would be a bigger difference seen.

A benchmark also wouldn't work because I'm always using my PC for other tasks in the foreground and of course that would effect things. Sorry.

I compiled a bunch of small games and stuff (DOOM, yeah!). They went fast (cause they were small) but the thing I loved was no disk access... whisper quiet... me like. :o I'm going to leave it like this for a while...

Just try it, not like it'll hurt anything... you can always undo it with the press of the enter key.
Code:
mount tmpfs /var/tmp/portage -t tmpfs -o size=1024m

to undo it
Code:
umount /var/tmp/portage

It should go without saying don't change this in the MIDDLE of emerging something... You will need to change the size parameter to suit your system... I'm using 1250m, but only because I can't repartition now to get a bigger swap file... If I could I'd probably set it for 2.5 gig and a 3 or 4 gig swap...

When doing it permentantly I'd probably mount it on /tmp, then create a /tmp/var -> /var/tmp symlink, so all my temp files are on the tmpfs.

Edit: You could also use the ramit script (above) to manage this for you, though you (I will when I get around to it) would have to be using the tmpfs mode and add the size parameter to the mount command.
Code:
ramit /tmp
ramit /var/tmp

If you currently have a lot of junk in your /tmp or /var/tmp files it would do to flush them out before using ramit since it copies the files over before mounting tmpfs.
Back to top
View user's profile Send private message
TheEternalVortex
Apprentice
Apprentice


Joined: 15 Oct 2002
Posts: 207
Location: San Jose, CA

PostPosted: Sat Nov 16, 2002 2:28 am    Post subject: Reply with quote

I was interested in this, so I made a quick (unscientific) test:
armagetron-0.1.4.9
emerge on regular hard drive: 3m 27s
emerge on 128 MB ramdisk: 2m 18s

I would like to point out that my hard drive is very slow (15 MB/s disk reads by hdparm). Still, it seems to provide a nice benefit.
_________________
-- Andy
Back to top
View user's profile Send private message
tkdack
n00b
n00b


Joined: 12 Apr 2002
Posts: 44
Location: Sydney, Australia

PostPosted: Sat Nov 16, 2002 5:38 am    Post subject: Reply with quote

My box has ~780MB of RAM and the same of swap.

By default I have a tmpfs mount of 768M set up for /var/tmp

Using this and a 2GB ccache my compiles are considerably faster (KDE version and release bumps in around 2 hours on a 1GHz Athlon)

For the larger packages that need heaps of swap (OOo and j2sdk) I have a spare partition that I mount for the purpose and I take the performance hit.

So far those are the only two apps that I need to use a physical disk for, X will just fit into the 768M tmps file system.

During most normal builds my system is running KDE in X with a heap of open applications, including a VMWare session and the box rarely uses more than 10% of swap, if that much.
_________________
Troy
http://linux.tkdack.com

Former Gentoo contributor TaD
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Sat Nov 16, 2002 5:59 am    Post subject: Reply with quote

tkdack wrote:
Using this and a 2GB ccache my compiles are considerably faster (KDE version and release bumps in around 2 hours on a 1GHz Athlon)

What's this ccache you speak of?
Back to top
View user's profile Send private message
TheEternalVortex
Apprentice
Apprentice


Joined: 15 Oct 2002
Posts: 207
Location: San Jose, CA

PostPosted: Sat Nov 16, 2002 6:12 am    Post subject: Reply with quote

emerge ccache

It works quite well, it caches compiles making building the same app more than once a LOT faster.
_________________
-- Andy
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Sat Nov 16, 2002 6:32 am    Post subject: Reply with quote

TheEternalVortex wrote:
emerge ccache

It works quite well, it caches compiles making building the same app more than once a LOT faster.

Does it work automatically after I emerge it?
Back to top
View user's profile Send private message
puddpunk
l33t
l33t


Joined: 20 Jul 2002
Posts: 681
Location: New Zealand

PostPosted: Sat Nov 16, 2002 7:47 am    Post subject: Reply with quote

IIRC you just have to add a small line of text at the bottom of /etc/make.conf.

It tells you how after you finished building it :)

EDIT: Scratch that:
Code:
 * To use ccache, add /usr/bin/ccache to your path before /usr/bin.
 * Portage 2.0.6+ will automatically take advantage of ccache with no additional steps.
 * If this is your first install of ccache, type something like this to set a maximum
 * cache size of 2GB (or whatever you desire):
 * # /usr/bin/ccache/ccache -M 2G
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Sat Nov 16, 2002 7:52 am    Post subject: Reply with quote

puddpunk wrote:
IIRC you just have to add a small line of text at the bottom of /etc/make.conf.

It tells you how after you finished building it :)

This ONLY helps if you're compiling something for the SECOND time, right?
Back to top
View user's profile Send private message
puddpunk
l33t
l33t


Joined: 20 Jul 2002
Posts: 681
Location: New Zealand

PostPosted: Sat Nov 16, 2002 7:57 am    Post subject: Reply with quote

yes, unless someone wants to lend you their ~/.ccache dir ;) (not recommended!!!)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum