[SOLVED] Distcc doesn't clean up temporary files (20GB!)

Daemon14 · Last edited by Daemon14 on Mon Oct 25, 2021 6:08 pm; edited 1 time in total

Hi everyone!

I have 2 machines, Dell Optiplex 990 with Core i7 2600 and Dell Latitude E6440 with Core i5 4600. Both are pretty decent and emerging even huge packages is not a big deal but since I usually update everything when I'm home, I figure why wouldn't they work together so I decided to install distcc. I used to use it years ago and it worked decently. I was following the Gentoo handbook but I was struggling a lot, that manual is not any handy. After a couple of hours I managed to have it work but I noticed something odd. While emerging, laptop ran out of RAM. I use Zram for /var/tmp and that's where distcc stores its tmp files as well. I took a look to it and it turned out there were over 4000 files (~20GB). This doesn't happen on the other PC, Distcc created tmp files but then it deletes them when they are not used anymore. Any idea why it doesn't happen on the laptop?

thanks in advance!
_________________
Dell Optiplex 7010 / CPU Core i7-3770K / RAM 16GB Patriot Viper III / GTX 1050 2GB / SSD 500GB + 2x HDD 2TB
Dell Latitude E6440 / CPU Core i5-4300M / RAM 8GB / SSD 120GB + HDD 500GB

eccerr0r · Posted: Thu Sep 09, 2021 4:28 pm Post subject:

/var/tmp/portage ? Do you mean portage tempdir files or distcc explicitly?

Distcc normally does delete temp files, unless the machine crashed while running a distcc job. And I think distcc normally creates temp files in /tmp ?

Portage on the other hand, if it fails a build it will leave it there. It will clean it up if you try it again but if it fails a second time it will once again leave it.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Daemon14 · Posted: Thu Sep 09, 2021 4:46 pm Post subject:

I mean the distcc tmp files. On my PC it cleans them up as they are not necessary but on the laptop it doesn't.

I changed the default /tmp to /var/tmp so it uses ramdisk (I have Zram for /var/tmp).

While emerging on laptop it gives me:

eccerr0r · Posted: Thu Sep 09, 2021 5:09 pm Post subject:

So did Kodi compile properly on the laptop? Did it go OOM and you --keep-going?

Having PORTAGE_TMPDIR pointing to a RAM backed filesystem is tricky, because if gcc starts using ram and the TMPDIR won't relieve the RAM, you'll go OOM.

I'm not familiar with Kodi's disk/build requirements but with 8GB I think it'd be hard to keep some packages in without dying especially if you bump up -j and it fails to distribute.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Daemon14 · Posted: Thu Sep 09, 2021 5:18 pm Post subject:

I have 8GB but it's possible to emerge kodi with even /var/tmp on Ramdisk. The problem appears when I use distcc, its tmp files take up a lot of space. Tmp files are not deleted as they are not necessary, it stores several thousand files till emerging is completed which causes OOM
I emerged kodi on the PC and I noticed that tmp files are deleted on-the-go. It keeps just 10-15 tmp files at a time.
_________________
Dell Optiplex 7010 / CPU Core i7-3770K / RAM 16GB Patriot Viper III / GTX 1050 2GB / SSD 500GB + 2x HDD 2TB
Dell Latitude E6440 / CPU Core i5-4300M / RAM 8GB / SSD 120GB + HDD 500GB

eccerr0r · Posted: Thu Sep 09, 2021 5:33 pm Post subject:

So how many jobs are you running? When using distcc you need to bump up -j so that they go to other machines to build. If they happen to fail due to OOM or other reason, regardless of which machine, it causes problems on the machine doing emerge and tends to leave files around.

You have more RAM on your desktop, which may make the difference. Does distcc work fine if you don't use RAM as PORTAGE_TMPDIR ?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Daemon14 · Posted: Thu Sep 09, 2021 5:41 pm Post subject:

make.conf:
MAKEOPTS="-j25 -l8" (PC)
MAKEOPTS="-j25 -l4" (laptop)

distcc distributes job correctly. The only problem is that distcc on laptop doesn't clean up tmp files. In the beginning I used /tmp which is on the SSD but I ran out of space on /. The tmp files took up to 20GB.

I haven't tried without Ramdisk but it doesn't seem to be the problem, I use ramdisk on the PC as well and it works perfectly.
_________________
Dell Optiplex 7010 / CPU Core i7-3770K / RAM 16GB Patriot Viper III / GTX 1050 2GB / SSD 500GB + 2x HDD 2TB
Dell Latitude E6440 / CPU Core i5-4300M / RAM 8GB / SSD 120GB + HDD 500GB

Hu · Moderator Joined: 06 Mar 2007 Posts: 21624

Your log says distcc chose to skip cleanup of those files. Why did it make that choice? Perhaps if you understand that, you can change its behavior.

Daemon14 · Posted: Thu Sep 09, 2021 6:20 pm Post subject:

HU yes, I've noticed it but I don't know where and how I made that choice. How can I change it?
is it DISTCC_SAVE_TEMPS="1"? I changed it to 0 but no effect.
_________________
Dell Optiplex 7010 / CPU Core i7-3770K / RAM 16GB Patriot Viper III / GTX 1050 2GB / SSD 500GB + 2x HDD 2TB
Dell Latitude E6440 / CPU Core i5-4300M / RAM 8GB / SSD 120GB + HDD 500GB

eccerr0r · Posted: Thu Sep 09, 2021 6:22 pm Post subject:

I still think the distcc job ran out of memory and died due to OOM killer. It left files around that the OP think was left around for no reason - but it did, it died.

OP: Please test without ramdisk. If you're extremely against doing this experiment, either reduce your -j to say no more than 12 or so since you don't have that many real cores or add 4GB of real swap on your ssd -- you don't have a whole lot of physical memory. I've found that -l frequently can still get you into a lot of trouble with running too many jobs and causing OOM.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Daemon14 · Posted: Thu Sep 09, 2021 6:30 pm Post subject:

eccerr0r I actually did, in the beginning but I ran out of space on <b>/</b>. RAM, ramdisk or disk space are not the issue. The problem is that distcc doesn't remove unneeded tmp files on-the-go. On the PC it does, on the laptop it doesn't.
_________________
Dell Optiplex 7010 / CPU Core i7-3770K / RAM 16GB Patriot Viper III / GTX 1050 2GB / SSD 500GB + 2x HDD 2TB
Dell Latitude E6440 / CPU Core i5-4300M / RAM 8GB / SSD 120GB + HDD 500GB

Hu · Moderator Joined: 06 Mar 2007 Posts: 21624

Daemon14 · Posted: Thu Sep 09, 2021 7:06 pm Post subject:

export | grep TEMP on the PC:

Daemon14 · Posted: Mon Oct 25, 2021 6:08 pm Post subject:

the solution was in /etc/env.d/02distcc. Setting DISTCC_SAVE_TEMPS="0" made it permanent.

Thanks everyone!
_________________
Dell Optiplex 7010 / CPU Core i7-3770K / RAM 16GB Patriot Viper III / GTX 1050 2GB / SSD 500GB + 2x HDD 2TB
Dell Latitude E6440 / CPU Core i5-4300M / RAM 8GB / SSD 120GB + HDD 500GB

Hu · Moderator Joined: 06 Mar 2007 Posts: 21624

For me, that file already set DISTCC_SAVE_TEMPS="0" by default. I never set it that way.