Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Using /tmp to store GCC temp. files while kernel compiles?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Sat Nov 04, 2017 11:29 am    Post subject: Using /tmp to store GCC temp. files while kernel compiles? Reply with quote

Hi all.

I'm configuring a Gentoo machine with a SSD and a mechanical disk, both are SATA drives. I've used the mechanical drive to store /var, portage files and, well, mainly as much as possible of files that [are supposed to] change a lot of times. However as /usr belongs to the SSD, I wonder if I can optimize even further by sending all temporary compile results to somewhere like /tmp or even a tmpfs when I'm compiling the kernel.

For kicks I've tried setting TMPDIR to /tmp in a file under /etc/env.d/ but it doesn't seem to work (I see no temporary file pop in /tmp). In case you wonder, yes I've even rebooted the machine to have a proper starting point and TMPDIR is indeed set to /tmp. Can it be done at all? and how?

Thanks in advance for any hint/suggestion.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10589
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Nov 04, 2017 12:47 pm    Post subject: Reply with quote

How about mounting /usr/src to some partition or directory on your rotating rust? Do you consider .o files to be temporary? If so, I believe it'd take modifications to the source to achieve what you're literally asking about.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy

PostPosted: Sat Nov 04, 2017 12:50 pm    Post subject: Reply with quote

bind mount
mount --bind /usr/src /tmp/kernel_buiild
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Sat Nov 04, 2017 12:55 pm    Post subject: Reply with quote

yeah, trying to force TMPDIR=/tmp is ... just what /tmp is made for :D

those are temp files for you, not really temporary files for kernel, that re-use them on next compile if they haven't change.
and it might (or not) use /tmp itself for some files that are really just garbage, but it's something most users won't see because they have cflags set with -pipe which is to avoid that.
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Sat Nov 04, 2017 1:24 pm    Post subject: Reply with quote

John R. Graham wrote:
Do you consider .o files to be temporary?

krinn wrote:
yeah, trying to force TMPDIR=/tmp is ... just what /tmp is made for :D

those are temp files for you, not really temporary files for kernel, that re-use them on next compile if they haven't change.
and it might (or not) use /tmp itself for some files that are really just garbage, but it's something most users won't see because they have cflags set with -pipe which is to avoid that.

No, they're not temporary file for *me*, you probably missed my point. Fact is GCC uses temporary files when compiling — I'm not talking about *.o files and the likes, I'm talking about... well, temporary files used by GCC before it spits out the compiled object files and binaries aso. I haven't checked ps when I compile a kernel, which I probably should have; I just wanted to make sure there's not much more I can do otherwise. If I understand -pipe (see make.conf after all, right?) is there to precisely avoid using temporary files and use pipes instead. So if compiling a kernel always uses -pipe then I'm fine :) .

I don't even know why I asked in the first time. But well, that's me. Thanks for the hint.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10589
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Nov 04, 2017 2:07 pm    Post subject: Reply with quote

FYI, the kernel makefile doesn't honor (nor know about) make.conf, which is purely a Gentooism. It does honor CFLAGS in the environment, though, so
Code:
CFLAGS=-pipe make
works. And, no, -pipe is not a default option for the kernel makefile.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54232
Location: 56N 3W

PostPosted: Sat Nov 04, 2017 2:40 pm    Post subject: Reply with quote

<nit>
CFLAGS=-pipe is an advisory not an instruction to gcc.
It means to use pipes between compile phases if memory permits.
</nit>

gcc will silently write intermediate files to hdd if there is not enough RAM to honour the -pipe advisory.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Sat Nov 04, 2017 5:12 pm    Post subject: Reply with quote

I wouldn't bother with all this, just tweak laptop_mode and related sysctls and the kernel will avoid writing to disk on its own.
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Mon Nov 13, 2017 9:56 am    Post subject: Reply with quote

Well, thanks for all the hints, guys.

Ant P. wrote:
I wouldn't bother with all this, just tweak laptop_mode and related sysctls and the kernel will avoid writing to disk on its own.

Sounds interesting. Could you detail a little?
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
grumblebear
Apprentice
Apprentice


Joined: 26 Feb 2008
Posts: 202

PostPosted: Mon Nov 13, 2017 11:54 am    Post subject: Reply with quote

What sort of optimizing is that? A SSD is not as fragile as you might think. Even with the portage tree and daily kernel compiles it will normally last for many, many years.

Besides speaking from personal experience, you can also explore the great wisdom Google has to offer on that topic.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Nov 15, 2017 6:21 am    Post subject: Reply with quote

John R. Graham wrote:
It does honor CFLAGS in the environment, though

Are you sure about this? Last time I tried, it did not. Upstream is rather picky about modifying compiler flags and still has not accepted patches which allow to add e.g. -march=native.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Nov 15, 2017 6:28 am    Post subject: Reply with quote

VinzC wrote:
Fact is GCC uses temporary files when compiling

AFAIK, gcc always places them in the directory where it creates the output files. To my knowledge there is no option to override that behaviour (gcc would need a mktemp-type algorithm/library to create safe filenames otherwise, but apparently it was decided not to include this complication into a compiler). I concluded this information from the documentation of the -save-temps option on the gcc manpage.
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Wed Nov 15, 2017 9:13 am    Post subject: Reply with quote

grumblebear wrote:
What sort of optimizing is that? A SSD is not as fragile as you might think. Even with the portage tree and daily kernel compiles it will normally last for many, many years.

The kind of optimization that brings me even more many years indeed.

For the record, this is only one step as I also plan to use my SSD in read-only mode, just as I've already done with a Raspberry Pi to protect against power outages. It has worked a treat for years even in an industrial environment without damaging the SD card. But that very part is off-topic and has nothing to do with my initial question, I'm just trying to feed your curiosity ;-).
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
grumblebear
Apprentice
Apprentice


Joined: 26 Feb 2008
Posts: 202

PostPosted: Wed Nov 15, 2017 1:18 pm    Post subject: Reply with quote

It just sounded to me as if you were concerned about the lifetime of your SSD. So excuse me for being a little off-topic, but the myths about writing to SSDs are going around a little too long for my taste.

I just want to point out to anyone reading this, that it is just not worth too much effort to avoid writing to SSDs. Even in your case, whatever it might be, I highly doubt you will even get a single year additional lifetime. The propability is much higher that some part of the electronics gets damaged or even the SATA standard gets replaced before the flash cells wear out.

PS
To get back on-topic: You can unpack and compile the kernel sources anywhere you like. Just don't forget to make modules_install and put the kernel image where your bootloader expects to find it. And of course you should put /lib/modules on your mechanical drive or compile a monolithic kernel.
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Wed Dec 20, 2017 12:33 pm    Post subject: Reply with quote

grumblebear wrote:
It just sounded to me as if you were concerned about the lifetime of your SSD.

Thank you for your insights but actually no, I'm not concerned about the lifetime of SSD's. At least not more than usual. Instead, in the very case of an SSD, I've been considering optimizing the Hell out of it so a) write to it only when needed and 2) sustain power outages just like the Raspberry Pi's I once configured. I want to most robust SSD I could ever configure. Just for the sake of it and see how far I can go, if you ask ;-) .

For the record I did see SSD's fail BTW. And not just once so I don't especially trust them (or, say, their lifetime expectancy) more than others. I may be wrong but, well, that's my experience.

grumblebear wrote:
To get back on-topic: You can unpack and compile the kernel sources anywhere you like. Just don't forget to make modules_install and put the kernel image where your bootloader expects to find it. And of course you should put /lib/modules on your mechanical drive or compile a monolithic kernel.

Interesting. However from what I've understood from this thread is that if I can get the kernel make process to use -pipe and as long as there's enough memory, there'll be no temp. (intermediate) files, which is all I'm interested in — I can cope with the rest.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum