Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to extend Portage TMPDIR in ramdisk through to disk(s)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
josephg
l33t
l33t


Joined: 10 Jan 2016
Posts: 783
Location: usually offline

PostPosted: Sun Aug 06, 2017 8:31 am    Post subject: How to extend Portage TMPDIR in ramdisk through to disk(s) Reply with quote

my particular pain point this how-to targets are the extremely (long and) resource intensive emerges for larger packages which can't fully compile in tmpfs. i have to create notmpfs workarounds, which means no benefit at all from tmpfs for these bloatwares. so my choices were either tmpfs or notmpfs. and these large bloatwares are the particular pain points which could most benefit from tmpfs. hence, this piece.

instead of tmpfs, i use zram to create a btrfs filesystem in memory, which i extend to disk(s) only when required with as much space as needed by emerge. now emerge can use ramdisk and spill over to disk, rather than an either or only. not touching physical disks gives an immense performance boost. so i try not to use physical swap devices. for step by step instructions, read on.. http://nixventure.blogspot.com/2017/08/gentoo-portage-emerge-tmpdir-extending.html

this might be useful for those who want to emerge in tmpfs but don't have enough space there. this could also be useful for low-memory systems which can additionally benefit from swap on compcache. you might find this useful to extend other use-cases too.

comments and brickbats welcome..
_________________
"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey


Last edited by josephg on Wed Aug 09, 2017 8:38 am; edited 1 time in total
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3310
Location: Rasi, Finland

PostPosted: Sun Aug 06, 2017 8:49 am    Post subject: Reply with quote

I just have swap. Mainly the swap is for hibernation, but if I happen to run out of ram during emerging, then the kernel moves the least used stuff in tmpfs to swap. It has been working before on my earlier setups where I had 8GiB or less RAM. Although it might need some fine tuning of some vm.* settings to get the best out of it.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
josephg
l33t
l33t


Joined: 10 Jan 2016
Posts: 783
Location: usually offline

PostPosted: Sun Aug 06, 2017 11:51 am    Post subject: Reply with quote

Zucca wrote:
I just have swap. Mainly the swap is for hibernation, but if I happen to run out of ram during emerging, then the kernel moves the least used stuff in tmpfs to swap. It has been working before on my earlier setups where I had 8GiB or less RAM. Although it might need some fine tuning of some vm.* settings to get the best out of it.

i have 4G memsize on this one, and emerge continued to crash even with much bigger swap sizes. i particularly remember boost etc was one of my nightmares.. it would take so long, and then eventually crash. meanwhile my keyboard/mouse/etc would just die. sometimes i've had to pull the power.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3104

PostPosted: Sun Aug 06, 2017 11:54 am    Post subject: Reply with quote

Quote:
Although it might need some fine tuning of some vm.* settings to get the best out of it.

Code:
vm.swappiness = 0
Some people claim that kernel will never use swap at 0, which means 1 would be the preferred value, down from 60 out of 100. Haven't tested it myself, so you might want to verify it before using either option.
I found 8 GB enough for building stuff in RAM during typical desktop workload. I just had to bump the size of tmpfs from default 50% to 85% (fstab option: size=85%), which is also a neat trick if you're close to the borderline.

Quote:
meanwhile my keyboard/mouse/etc would just die. sometimes i've had to pull the power.
This might be related to general load rather than size of SWAP.
Bumping portage niceness from 0 to 19 helps often a lot with such issues
Back to top
View user's profile Send private message
mir3x
Guru
Guru


Joined: 02 Jun 2012
Posts: 455

PostPosted: Mon Aug 07, 2017 9:56 pm    Post subject: Reply with quote

Quote:
meanwhile my keyboard/mouse/etc would just die. sometimes i've had to pull the power.

try oom killer - magic sys rq + f
Probably u will need to use it few times, it oftens kills X, but its better that hard reset.
_________________
Sent from Windows
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3310
Location: Rasi, Finland

PostPosted: Mon Aug 07, 2017 10:53 pm    Post subject: Few ideas... Reply with quote

One thing came into my mind... You could work out something by using bcache feature of Linux. That way you cannot use tmpfs... at least directly. Afaik bcache needs a actual device. zram can create virtual (compressed) storage devices. You could use a zram device as the fast cache drive (also saves memory by compressing) then you'd need the another device (or maybe just a partition) where to put "overflowing" stuff.
The problem with zram is that it consumes more memory as it fills, but does not free it unless you unmount the device and reset it's size (and then create a filesystem back on it).
So you'd propably need to automate the process.

Second idea I have is to use swap as the "overflow" storage as I mentioned before. But now add zswap in to the game. zswap compresses memory pages in ram if memory usage is too high (user selectable limit). If later the memory fills up (even when the memory is compressed), then the least used memory pages are moved to actual swap.
This should be much easier to set up than the zram+bcache hack.

I've not tested either method, but the first one really needs quite a lot of tinkering to get it working propely. For example resetting the bcache-zram device after emerge exits...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
jonathan183
Guru
Guru


Joined: 13 Dec 2011
Posts: 318

PostPosted: Tue Aug 08, 2017 12:08 am    Post subject: Reply with quote

If you had 16G of RAM would you still use zram, would you split between tmpfs and swap? would that split change for emerging packages vs use of the system for other things?
I ended up setting up portage on disc for my previous desktop to build things like Libreoffice ... time for me to revisit this since my system now has more RAM ...
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3310
Location: Rasi, Finland

PostPosted: Tue Aug 08, 2017 10:15 am    Post subject: Reply with quote

jonathan183 wrote:
If you had 16G of RAM would you still use zram, would you split between tmpfs and swap?
With zram you can create zram device nodes under /dev and they appear like a block (mass storage) device.
With tmpfs you don't get any device nodes, you just specify a mount point where the "ram portion" starts. tmpfs works more like a network share; you don't format it with a filesystem - tmpfs IS a filesystem of its own. So you cannot use it as a cache -part of bcache (afaik bcache needs a device node). Creating a filesystem image inside tmpfs gives you something close to what zram gives. But it automatically reserves the amount of memory the image takes disk space. So, don't do that. Use zram instead if you really want to go complex.
Now back to your question: I wouldn't use zram. I'd use zswap and tmpfs. Lastly I'd tune vm.swappiness and zswap settings. With zswap I'd propably set 50% of memory for zswap to compress (with that 16GB of RAM). Then I'd create a swap of around 4-8GB. YMMV... I have no need for any memory compression, so I haven't tested this.

jonathan183 wrote:
would that split change for emerging packages vs use of the system for other things?
I'm not sure if I understood your question here... But you can use other applications at the same time, but the performance will be slower if zswap needs to compress (and uncompress) memory pages in addition to compiling overhead. I think kernel (zswap) keeps track which memory areas are in more use and tries to keep those in the uncompressed area while the temporary files that are generated while compiling software are most propably in compressed form in memory (by zswap, if needed). But if the system still runs out of memory after zswap has tried its best the swapping starts and will slow the machine down. vm.swappiness might play HUGE role if/when this starts. I'd guess if you set vm.swappiness to around 50 or 60 kernel has already moved some memory to swap area before you really start running out of memory, thus making swapping at this point less demending for the system.

In conclusion: Enable zswap and tune its settings. Create a (enough large) tmpfs for your portage temp files.
If the performance is poor, tune vm.* and zswap settings.
If you want to go wild, try out zram device as a cache part of bcache (I doubt it'll be any better than zswap+tmpfs).
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum