Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage and tmpfs...a myth or a reality???
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Judge584
Tux's lil' helper
Tux's lil' helper


Joined: 18 Apr 2005
Posts: 109

PostPosted: Sat Mar 06, 2010 6:40 pm    Post subject: Portage and tmpfs...a myth or a reality??? Reply with quote

Hi,

I'm doing some tests with portage and tmps, to see if i can speedup compilation time.

So I've added theses lines to my make.conf:

PORTAGE_TMPFS="/dev/shm"
PORTAGE_TMPDIR="/dev/shm"
BUILD_PREFIX="/dev/shm"

Then I have compiled gcc, here are the results:

about 13m30s in both cases!

So what? It seems that compilation time is not affected by tmps utilisation.

What's your results?

PS: my tmps is set to 50% of my ram, and I do have 8Go of Ram. CPU is an Intel Q9550. (MAKEOPTS="-j5")
Back to top
View user's profile Send private message
Shining Arcanine
Veteran
Veteran


Joined: 24 Sep 2009
Posts: 1110

PostPosted: Sat Mar 06, 2010 7:12 pm    Post subject: Reply with quote

It could be that your hard drive's write cache is able to buffer the write requests and they occur infrequently enough that they do not overload your disk's write cache, which is where a slowdown would occur.

Anyway, try doing a parallel compilation of multiple packages with a tmpfs using --jobs. You should see a difference.
Back to top
View user's profile Send private message
Judge584
Tux's lil' helper
Tux's lil' helper


Joined: 18 Apr 2005
Posts: 109

PostPosted: Sat Mar 06, 2010 7:36 pm    Post subject: Reply with quote

My bad...Slowdown was coming from the stupid superkaremba theme (for hardware monitoring) that I have added just before setting up "portage tmps"...in was eating 10% cpu on each core!!!

Now it is removed:

Sat Mar 6 20:27:23 2010 >>> sys-devel/gcc-4.4.3
merge time: 12 minutes and 18 seconds.

that's better!

Can you tell me more about using --jobs?
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1931

PostPosted: Sat Mar 06, 2010 7:38 pm    Post subject: Reply with quote

Judge584 wrote:
Can you tell me more about using --jobs?


This thread has a pretty good discussion about it:
https://forums.gentoo.org/viewtopic-t-817160-highlight-.html
Back to top
View user's profile Send private message
JustJoe
Tux's lil' helper
Tux's lil' helper


Joined: 30 Jul 2005
Posts: 80

PostPosted: Tue Apr 06, 2010 4:09 pm    Post subject: Reply with quote

I've done some testing with tmpfs.

Some info first:
- Emerging dev-db/mysql-5.0.90-r2.
- System: 2.6.31-gentoo-r10 SMP x86_64 Intel(R) Pentium(R) Dual CPU E2140 @ 1.60GHz (CFLAGS="-march=core2 -O2 -pipe") 2GB DDR2
- cpufreq governor performance sets cpufreq to 1600MHz
- cpufreq governor ondemand (default setting) switches from 1200Mhz to 1600MHz.
- Modified tmerge scripts (http://en.gentoo-wiki.com/wiki/Speeding_up_portage_with_tmpfs)

tmerge.tmpfs: mounts 800MB tmpfs on /var/tmp/portage before merge.
tmerge.performance: sets cpufreq to performance before merge and back to ondemand afterward.
tmerge.tmpfs.performance: mounts 800MB tmpfs on /var/tmp/portage and sets cpufreq to performance before merge and back to ondemand afterward.

Results without ccache:
emerge -v dev-db/mysql
merge time: 11 minutes and 57 seconds.
merge time: 11 minutes and 49 seconds.
merge time: 11 minutes and 49 seconds.
->average: 11 minutes and 52 seconds.

./tmerge.tmpfs -v dev-db/mysql
merge time: 11 minutes and 43 seconds.
merge time: 11 minutes and 45 seconds.
merge time: 11 minutes and 41 seconds.
->average: 11 minutes and 43 seconds.

./tmerge.performance -v dev-db/mysql
merge time: 11 minutes and 6 seconds.
merge time: 11 minutes and 7 seconds.
merge time: 11 minutes and 10 seconds.
->average: 11 minutes and 8 seconds.

./tmerge.tmpfs.performance -v dev-db/mysql
merge time: 10 minutes and 58 seconds.
merge time: 11 minutes and 1 second.
merge time: 11 minutes.
->average: 11 minutes.

Conclusion (for merging dev-db/mysql without ccache on this system) :
- emerging with only tmpfs mounted on /var/tmp/portage is 9 sec faster than emerging without tmpfs.
- emerging with only cpufreq-governor set to performance is 44 sec faster than emerging with cpufreq-governor set to ondemand.
- emerging with both cpufreq-governor set to performance and tmpfs mounted on /var/tmp/portage is 52 sec faster then plain (no tmpfs, ondemand) emerge.

Results with ccache:
emerge -v dev-db/mysql
merge time: 6 minutes and 3 seconds.
merge time: 6 minutes and 1 second.
merge time: 6 minutes and 2 seconds.
->average: 6 minutes and 2 seconds.

./tmerge.tmpfs -v dev-db/mysql
merge time: 5 minutes and 58 seconds.
merge time: 5 minutes and 56 seconds.
merge time: 5 minutes and 54 seconds.
->average: 5 minutes and 56 seconds.

./tmerge.performance -v dev-db/mysql
merge time: 5 minutes and 24 seconds.
merge time: 5 minutes and 22 seconds.
merge time: 5 minutes and 23 seconds.
->average: 5 minutes and 22 seconds.

./tmerge.tmpfs.performance -v dev-db/mysql
merge time: 5 minutes and 17 seconds.
merge time: 5 minutes and 17 seconds.
merge time: 5 minutes and 17 seconds.
->average: 5 minutes and 17 seconds.

Conclusion (for merging dev-db/mysql with ccache on this system) :
- emerging with only tmpfs mounted on /var/tmp/portage is 6 sec faster than emerging without tmpfs.
- emerging with only cpufreq-governor set to performance is 40 sec faster than emerging with cpufreq-governor set to ondemand.
- emerging with both cpufreq-governor set to performance and tmpfs mounted on /var/tmp/portage is 45 sec faster then plain (no tmpfs, ondemand) emerge.


So a single merge using tmpfs is only a little bit faster than when not using tmpfs.
However, setting the cpufreq-governor to performance does seem to make more of a difference.
Back to top
View user's profile Send private message
Judge584
Tux's lil' helper
Tux's lil' helper


Joined: 18 Apr 2005
Posts: 109

PostPosted: Wed Apr 07, 2010 3:46 pm    Post subject: Reply with quote

That's very interesting! Thanks a lot JustJoe.

I think I have to do some tests again...will post again in a few days
Back to top
View user's profile Send private message
mahatma
n00b
n00b


Joined: 01 Sep 2005
Posts: 7

PostPosted: Thu Aug 11, 2011 4:21 pm    Post subject: Reply with quote

Sometimes in low (or normal PC) RAM environment good results are to mount ramfs on every build on $TMPDIR (/var/tmp/portage/sys-devel/gcc-4.6.0/tmp - for example). This place will be used for logs, gcc's tmp files and many others hardworks. Last night I make it. Extremely easy! For common users outside my overlay it is just to add near 4 lines into /etc/portage/bashrc:

[ "$RAMTMPDIR" = yes -a -n "$TMPDIR" -a -n "$PORTAGE_BUILDDIR" -a -z "${TMPDIR##$PORTAGE_BUILDDIR/*}" ] && case "$EBUILD_PHASE" in
clean)umount -l "$TMPDIR";;
setup)mount -t ramfs -o noatime none "$TMPDIR";;
esac

And to activate - add "RAMTMPDIR=yes" into /etc/make.conf or before emerge. Or suggest developers to add it into portage FEATURES ;)

PS This is tricky and dirty, but works.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum