Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Using 'tmpfs' as tempdir for portage builds
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
paradigm-X
Apprentice
Apprentice


Joined: 19 Sep 2013
Posts: 168

PostPosted: Thu Dec 19, 2013 3:57 am    Post subject: Using 'tmpfs' as tempdir for portage builds Reply with quote

On this page, http://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs , I found an explanation of how to use the 'tmpfs' in a way that may be helpful to me, like so:

" During emerge, you can build packages in tmpfs. Instead of having large build files in hdd. This speeds up emerge times and reduces hdd/ssd wearing."

As I am using an ssd on one system for running a KVM/Qemu host and several guests, including Gentoo guests, it seems to me that this would be a good application for minimizing the ssd writes, and saving time at the very least, which come along with every ebuild, not only on the host but also on the guests. Is the information as to the relevance of that method currently applicable, or has some aspect of the Gentoo way of building packages changed in such a way that makes the information no longer relevant? It certainly sounds like a swell idea, but I am not yet able to know with absolute certainty.


Last edited by paradigm-X on Thu Dec 19, 2013 7:32 pm; edited 1 time in total
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Thu Dec 19, 2013 4:18 am    Post subject: Reply with quote

It's still relevant; I use the setup myself.

Note that some big packages like Firefox and OpenOffice/LibreOffice take a lot of space to build (> 6GB). I just use the pre-built *-bin versions of those packages.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
paradigm-X
Apprentice
Apprentice


Joined: 19 Sep 2013
Posts: 168

PostPosted: Thu Dec 19, 2013 8:03 pm    Post subject: Reply with quote

Hi, Hypnos, and thanks for the tips. The instructions on that page, whose link I just fixed after noticing it was broken (by my putting the comma directly afterwards instead of adding a space first!), mentioned that a few packages, having very large requirements for building, should be identified ahead of time with configuration files, like so:

"For example: app-office/openoffice: 10G or so. www-client/chromium: More than 2GB.
Per package choise for TMPFS / HDD

You can configure emerge to build large packages outside of the tmpfs drive."


So, how can I go about knowing ahead of time which packages should be handled specially? Where does the "required build space" show up for me to see, in other words? Also, on that page it showed this means for getting the portage build folder put onto a tmpfs:

none /var/tmp/portage tmpfs size=2048M,noatime 0 0


I was wondering why the "size=2048" was chosen rather than a larger one? So what would happen if a particular build happen to need more than the arbitrarily(?) stated value? Thanks.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54097
Location: 56N 3W

PostPosted: Thu Dec 19, 2013 9:18 pm    Post subject: Reply with quote

paradigm-X,

If you have the RAM to build in tmpfs, then you don't really need to.
The reason is that the kernel will keep everything around in RAM in the disk cache anyway.
tmpfs will stop the SSD writes though.

2G is enough tmpfs to build most packages. By default, tmpfs is limited to 50% or RAM.

A few but not all of the bigger packages do a resource check and fail is resources are insufficent. Not all do this.
Its in the individual ebuilds.

There is a trap for the unwary tmpfs user here too.
You do an update, it wants to build several big packages. They are good to you and the resource checks pass.
Now, libreoffice and firefox build at the same time. tmpfs is filled and they both fail to build.
There are other combinations like this.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
schorsch_76
Guru
Guru


Joined: 19 Jun 2012
Posts: 450

PostPosted: Thu Dec 19, 2013 9:19 pm    Post subject: Reply with quote

You can use the portage/env [1] mechanism to get the big packages build on hdd.

[1] https://wiki.gentoo.org/wiki//etc/portage/env
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Thu Dec 19, 2013 9:19 pm    Post subject: Reply with quote

Major consumers of space are fairly obvious. They usually take a long time to compile, are huge to run, and are hard to maintain. Some ebuilds for major consumers will perform a check at start because so many people have been burned by the space requirements.

2G was chosen because it works for most packages that are not absurdly space intensive. If the build needs more than that, it will fail, just like if you tried to build it on a real filesystem that had only 2G of space free.
Back to top
View user's profile Send private message
saellaven
l33t
l33t


Joined: 23 Jul 2006
Posts: 646

PostPosted: Fri Dec 20, 2013 12:57 am    Post subject: Reply with quote

NeddySeagoon wrote:
paradigm-X,

If you have the RAM to build in tmpfs, then you don't really need to.
The reason is that the kernel will keep everything around in RAM in the disk cache anyway.
tmpfs will stop the SSD writes though.


I tried going with tmpfs for a month or two... and the result was that tmpfs was actually a little SLOWER than my spinning rust precisely because of the caching. Why tmpfs turned out slightly slower, I'm not sure. I'm guessing that, at 6GB, I'm right on that threshold where there's more overhead involved in tmpfs swapping to rust (if you exceed your memory usage, it uses your swap space) where in the standard setup, pages can simply be dropped from cache and then read from the drive as needed (so in one case, you're writing and reading on demand while in the other, just reading on demand).

Of course, that was purely for speed. Saving write cycles on the SSD is a different matter, though if memory contention is an issue, you may still find yourself writing to swap anyway.
_________________
Ryzen 3700X, Asus Prime X570-Pro, 64 GB DDR4 3200, GeForce GTX 1660 Super
openrc-0.17, ~vanilla-sources, ~nvidia-drivers, ~gcc
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Fri Dec 20, 2013 1:49 am    Post subject: Reply with quote

saellaven wrote:
Of course, that was purely for speed. Saving write cycles on the SSD is a different matter, though if memory contention is an issue, you may still find yourself writing to swap anyway.

You can always turn off swap.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
saellaven
l33t
l33t


Joined: 23 Jul 2006
Posts: 646

PostPosted: Fri Dec 20, 2013 2:28 am    Post subject: Reply with quote

Hypnos wrote:
saellaven wrote:
Of course, that was purely for speed. Saving write cycles on the SSD is a different matter, though if memory contention is an issue, you may still find yourself writing to swap anyway.

You can always turn off swap.


then you have the problem of managing the OOM killer if you have RAM constraints.

In the end, for me, tmpfs for building packages didn't provide the speed advantage I was looking for, so I went back to spinning rust. Outside of build time, there isn't enough of an advantage to the hardware cost of having to replace my cpu+mobo for more memory (chipset limitation) since I tend to not stress this system elsewhere.

As I said, using it to prevent SSD writes may be advantageous with the caveat that if you have swap on the SSD, you might end up doing writes you weren't intending on.
_________________
Ryzen 3700X, Asus Prime X570-Pro, 64 GB DDR4 3200, GeForce GTX 1660 Super
openrc-0.17, ~vanilla-sources, ~nvidia-drivers, ~gcc
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Fri Dec 20, 2013 2:47 am    Post subject: Reply with quote

saellaven wrote:
then you have the problem of managing the OOM killer if you have RAM constraints.

I do have swap enabled on my machine, but swappiness is set to zero. This way the swap will only be used if all of the physical RAM is exhausted.

My machine has 4GB of RAM. I know that my desktop almost never uses more than 1.5GB of RAM. So, I set my tmpfs max size to 2.5GB.

This way, I save writes on my SSD in almost every instance, but still avoid OOM killer when my desktop usage does exceed the physical RAM. Luckily the newer versions of Firefox no longer have the memory leak problem :)
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54097
Location: 56N 3W

PostPosted: Fri Dec 20, 2013 3:28 pm    Post subject: Reply with quote

I have a diskless desktop box with 4G RAM that builds everything except libreoffice and firefox in tmpfs.
Swap over NFS does work but its so slow. The box runs swapless
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
JackieMe
n00b
n00b


Joined: 18 Jan 2013
Posts: 16

PostPosted: Mon Jan 13, 2014 1:53 am    Post subject: Reply with quote

How do I know which packages need more than 2GB build space? I mean, I know firefox & libreoffice are big. But what about others?
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3104

PostPosted: Mon Jan 13, 2014 7:51 pm    Post subject: Reply with quote

Packages with insanely high compile time requirements perform check as soon as you run emerge with argument pulling them (sometimes it's long before portage actually attempts to build them). If those tests fail, they tend to kill emerge, so just read the messages received over emerge standard output ->the terminal usually.
Back to top
View user's profile Send private message
creaker
l33t
l33t


Joined: 14 Jul 2012
Posts: 651

PostPosted: Mon Jan 13, 2014 9:22 pm    Post subject: Reply with quote

JackieMe wrote:
How do I know which packages need more than 2GB build space? I mean, I know firefox & libreoffice are big. But what about others?


From my experience: I have a 2Gb tmpfs and it was enough to build all the packages during system and KDE install (without swap). I have MAKEOPTS set to "-j5". I regularly checked "df -h" output and it never exceeded 650 Mb (it was at some of qt packages installation).
I even built a firefox with 2Gb, that was enough though emerge complained about a tmpfs size. Nevetheless firefox was built and works just fine till now.
I do not know where emerge got this kind of information, but seems some configuration files should contain required amount of memory
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 231
Location: Brazil

PostPosted: Tue Jan 14, 2014 2:31 am    Post subject: Reply with quote

creaker wrote:

I do not know where emerge got this kind of information, but seems some configuration files should contain required amount of memory

http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/www-client/firefox/firefox-26.0.ebuild?view=markup
Line 129:
Code:
        # Ensure we have enough disk space to compile
130           if use pgo || use debug || use test ; then
131                   CHECKREQS_DISK_BUILD="8G"
132           else
133                   CHECKREQS_DISK_BUILD="4G"
134           fi

8GB for firefox with test/debug/pgo flags or 4GB without them.
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
JackieMe
n00b
n00b


Joined: 18 Jan 2013
Posts: 16

PostPosted: Tue Jan 14, 2014 12:17 pm    Post subject: Reply with quote

creaker wrote:
JackieMe wrote:
How do I know which packages need more than 2GB build space? I mean, I know firefox & libreoffice are big. But what about others?


From my experience: I have a 2Gb tmpfs and it was enough to build all the packages during system and KDE install (without swap). I have MAKEOPTS set to "-j5". I regularly checked "df -h" output and it never exceeded 650 Mb (it was at some of qt packages installation).
I even built a firefox with 2Gb, that was enough though emerge complained about a tmpfs size. Nevetheless firefox was built and works just fine till now.
I do not know where emerge got this kind of information, but seems some configuration files should contain required amount of memory


I am just being curious. I have both firedox and libreoffice(only two giants I can recall now). Will give it a try at 4GB. And want to know that I am using ccahe too. Won't be any trouble I suppose?

rudregues wrote:

8GB for firefox with test/debug/pgo flags or 4GB without them.

Well I guess 4GB for me will just do all fine?
Back to top
View user's profile Send private message
creaker
l33t
l33t


Joined: 14 Jul 2012
Posts: 651

PostPosted: Tue Jan 14, 2014 2:58 pm    Post subject: Reply with quote

JackieMe wrote:

I am just being curious. I have both firedox and libreoffice(only two giants I can recall now). Will give it a try at 4GB. And want to know that I am using ccahe too. Won't be any trouble I suppose?


Somewhere I read that libreoffice requires up to 18 Gb of space. I doubt it can be built with 4 Gb, but for firefox it should be enough.
I think I was able to build firefox with 2 Gb because I disabled all the USE flags except "minimal"
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 231
Location: Brazil

PostPosted: Tue Jan 14, 2014 3:20 pm    Post subject: Reply with quote

JackieMe wrote:
Well I guess 4GB for me will just do all fine?
For LibreOffice I would put at least 6.5GB based on the ebuild here (line 251) http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-office/libreoffice/libreoffice-4.1.3.2-r2.ebuild?view=markup
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
N8Fear
Tux's lil' helper
Tux's lil' helper


Joined: 15 Apr 2013
Posts: 140
Location: Berlin (Germany)

PostPosted: Tue Jan 14, 2014 4:31 pm    Post subject: Reply with quote

I didn't test it, but about three years ago, linking Firefox required more than 3 GB of RAM (at least the Windows version, see [1]). If you don't have swap (if you want to save writes that's most likely a save assumption) it's likely that you need at least 8 GB of RAM to build Firefox.


[1] http://developers.slashdot.org/story/11/12/14/1725205/firefox-too-big-to-link-on-32-bit-windows
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 231
Location: Brazil

PostPosted: Tue Jan 14, 2014 9:17 pm    Post subject: Reply with quote

N8Fear wrote:
I didn't test it, but about three years ago, linking Firefox required more than 3 GB of RAM (at least the Windows version, see [1]). If you don't have swap (if you want to save writes that's most likely a save assumption) it's likely that you need at least 8 GB of RAM to build Firefox.


[1] http://developers.slashdot.org/story/11/12/14/1725205/firefox-too-big-to-link-on-32-bit-windows

I think your assumption is not correct.

Windows 32bit normal version can use until 3GB. The system itself would consume 500MB~1GB, remaining near 2GB to compile firefox in RAM.

Today and in Gentoo, if you want to compile firefox in RAM you need at least 4GB free beyond the memory used by the system itself.
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Wed Jan 15, 2014 3:22 am    Post subject: Reply with quote

rudregues wrote:
I think your assumption is not correct.

Windows 32bit normal version can use until 3GB. The system itself would consume 500MB~1GB, remaining near 2GB to compile firefox in RAM.

Today and in Gentoo, if you want to compile firefox in RAM you need at least 4GB free beyond the memory used by the system itself.
No, normal Windows 32bit cannot use 3GB for an application. Windows uses a 2/2 split by default. Even if you boot with the kernel parameter to use a 3/1 split, Windows will only allow 3GB to user programs marked as handling that correctly.
Back to top
View user's profile Send private message
JackieMe
n00b
n00b


Joined: 18 Jan 2013
Posts: 16

PostPosted: Wed Jan 15, 2014 5:13 am    Post subject: Reply with quote

creaker wrote:

Somewhere I read that libreoffice requires up to 18 Gb of space. I doubt it can be built with 4 Gb, but for firefox it should be enough.
I think I was able to build firefox with 2 Gb because I disabled all the USE flags except "minimal"

I've got this for firefox:
Quote:
www-client/firefox-26.0 was built with the following:
USE="alsa bindist dbus jit libnotify minimal (policykit) startup-notification -custom-cflags -custom-optimization -debug -gstreamer (-pgo) -pulseaudio (-selinux) -system-cairo -system-icu -system-jpeg -system-sqlite -wifi" LINGUAS="en_GB zh_CN -af -ak -ar -as -ast -be -bg -bn_BD -bn_IN -br -bs -ca -cs -csb -cy -da -de -el -en_ZA -eo -es_AR -es_CL -es_ES -es_MX -et -eu -fa -fi -fr -fy_NL -ga_IE -gd -gl -gu_IN -he -hi_IN -hr -hu -hy_AM -id -is -it -ja -kk -km -kn -ko -ku -lg -lt -lv -mai -mk -ml -mr -nb_NO -nl -nn_NO -nso -or -pa_IN -pl -pt_BR -pt_PT -rm -ro -ru -si -sk -sl -son -sq -sr -sv_SE -ta -ta_LK -te -th -tr -uk -vi -zh_TW -zu"

And this:
rudregues wrote:

For LibreOffice I would put at least 6.5GB based on the ebuild here (line 251) http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-office/libreoffice/libreoffice-4.1.3.2-r2.ebuild?view=markup

Seems 4GB is not GIANT enough faced with firefox &libreoffice. But fact is I only have 4GB of RAM and 2GB of swap. Any solution? THX a lot for all advice because I don't really understand how to set it up now:(
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 231
Location: Brazil

PostPosted: Wed Jan 15, 2014 5:40 am    Post subject: Reply with quote

JackieMe wrote:

Seems 4GB is not GIANT enough faced with firefox &libreoffice. But fact is I only have 4GB of RAM and 2GB of swap. Any solution? THX a lot for all advice because I don't really understand how to set it up now:(

I use a nice workaround for build any package in RAM but Libreoffice, Chromium and Firefox.
See here http://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
JackieMe
n00b
n00b


Joined: 18 Jan 2013
Posts: 16

PostPosted: Wed Jan 15, 2014 5:49 am    Post subject: Reply with quote

rudregues wrote:

I use a nice workaround for build any package in RAM but Libreoffice, Chromium and Firefox.
See here http://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs

Thx, and yes I've refered to it earlier. In my case(4GB RAM & 2GB swap ), maybe I'll set it to only 2GB and build firefox & libreoffice outside.
But I've checked I don't have /etc/portage/env/ and /etc/portage/package.env. Is it alright to just touch it as wiki told? And I am using ccache, will it be any trouble since TMPFS will be cleared after reboot? Thx!
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 231
Location: Brazil

PostPosted: Wed Jan 15, 2014 7:46 pm    Post subject: Reply with quote

JackieMe wrote:
Thx, and yes I've refered to it earlier. In my case(4GB RAM & 2GB swap ), maybe I'll set it to only 2GB and build firefox & libreoffice outside.
But I've checked I don't have /etc/portage/env/ and /etc/portage/package.env. Is it alright to just touch it as wiki told? And I am using ccache, will it be any trouble since TMPFS will be cleared after reboot? Thx!

You can create /etc/portage/env/ and /etc/portage/package.env with no problems. Don't know about ccache....
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum