Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Problems with TMPFS space not getting freed
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Eversor
n00b
n00b


Joined: 28 Jun 2006
Posts: 17

PostPosted: Thu Jan 10, 2013 11:24 pm    Post subject: Problems with TMPFS space not getting freed Reply with quote

Hi all,

I've been using TMPFS as a way to have most caches sitting on a RAM based filesystem to avoid wearing out ssds (mostly /tmp, /var/tmp/portage and xdg-caches) but I've seen to bumped into a bug where space on TMPFS isn't freed after the files have been deleted (even after accounting for a supposed 15min delay I read about somewhere).

So, let's say I have my /tmp at 4GiB and then copy a 1GiB file into there, then delete it after it is done copying. I wait a while and do a "df -h" and I get the same used space, the 1GiB, as before deleting the file. I wait for days and the space is never freed. This obviously makes it a rather big issue to using TMPFS for any kind of filesystem temp dir.

lsof for open files in /tmp shows nothing that is taking up the space and while rebooting the system, during shutdown, the init scripts also claim that /tmp is being used but fuser claims no files in /tmp - this is another bug, /tmp is always in use, even in console mode where it isn't using any files there, which I can see with lsof.

Am I the only one witnessing this? Any help is welcomed.

Kernel is gentoo-sources-3.6.11, don't know what else is relevant.
Back to top
View user's profile Send private message
blain3
n00b
n00b


Joined: 02 Dec 2012
Posts: 30
Location: Southern Pensylvania

PostPosted: Fri Jan 11, 2013 3:57 am    Post subject: Reply with quote

whats your fstab look like? i was just looking into TRIM and found a few things, i see a lot of this:
tmpfs defaults,noatime,mode=1777, as im using a ssd as well im going to edit in discard and nodiratime to reduce writes(for root and var partitions of mine on SSD), the discard is trim. i have my /var/tmp set at 12G with 16G of ram, as one of the mods posted and have yet to have a problem.

Edit: also are you using only a SSD and no other drive? and are you using a swap partition? might help those smarter then me to help u to know this
_________________
#uname -s -m -i -p -v -o -n
Linux
x86_64
GenuineIntel
Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
#1 SMP PREEMPT Mon Nov 19 11:17:39 UTC 2012
GNU/Linux
xobtihs
Back to top
View user's profile Send private message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Fri Jan 11, 2013 9:30 am    Post subject: Reply with quote

for portage, the proper way of using RAM is to add these two in make.conf:

Code:
PORTAGE_TMPFS="/dev/shm"
PORTAGE_TMPDIR="/dev/shm"


then you'll see /dev/shm/portage/ being used. This way is automatic no need to manually mount a directory and no need to fiddle with mount options.
Back to top
View user's profile Send private message
Eversor
n00b
n00b


Joined: 28 Jun 2006
Posts: 17

PostPosted: Fri Jan 11, 2013 12:15 pm    Post subject: Reply with quote

_______0 wrote:
for portage, the proper way of using RAM is to add these two in make.conf:

Code:
PORTAGE_TMPFS="/dev/shm"
PORTAGE_TMPDIR="/dev/shm"


then you'll see /dev/shm/portage/ being used. This way is automatic no need to manually mount a directory and no need to fiddle with mount options.


I was using

PORTAGE_TMPDIR="/tmp"
DISTDIR="/tmp/distfiles"

PORTAGE_TMPFS doesn't seem to be documented anywhere. What does it do?

I also have interest in using TMPFS /tmp for xdg-cache, as chrome is one of the biggest culprits of disk IO activity. xsession-errors is another one, and I think I will be adding some init scripts to move /var/log to tmpfs at boot and shutdown to restore them between shutdowns.
Back to top
View user's profile Send private message
Eversor
n00b
n00b


Joined: 28 Jun 2006
Posts: 17

PostPosted: Fri Jan 11, 2013 12:20 pm    Post subject: Reply with quote

blain3 wrote:
whats your fstab look like? i was just looking into TRIM and found a few things, i see a lot of this:
tmpfs defaults,noatime,mode=1777, as im using a ssd as well im going to edit in discard and nodiratime to reduce writes(for root and var partitions of mine on SSD), the discard is trim. i have my /var/tmp set at 12G with 16G of ram, as one of the mods posted and have yet to have a problem.

Edit: also are you using only a SSD and no other drive? and are you using a swap partition? might help those smarter then me to help u to know this


These are my relevant sections of fstab:

none /tmp tmpfs size=7G 0 0
# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
# use almost no memory if not populated with files)
shm /dev/shm tmpfs nodev,nosuid,noexec 0 0

I tried _______0's suggestion and files created by portage are correctly freed. Guess it's either a mount option or I should not be using "none" but "shm". Thoughts?

EDIT:

About TRIM, I've added that to the FSTAB entry, completely forgot about that.
Back to top
View user's profile Send private message
Eversor
n00b
n00b


Joined: 28 Jun 2006
Posts: 17

PostPosted: Fri Jan 11, 2013 3:08 pm    Post subject: Reply with quote

Forget that it works... Somehow it is working on the console but once I get X server running and it starts writing files to xdg-cache dir, writing a big file and deleting it will not free up the space :/

Tried everything, can't get this to work ok.
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3951
Location: USA

PostPosted: Fri Jan 11, 2013 3:45 pm    Post subject: Reply with quote

Did you try kernel 3.5.7? Might be a kernel regression too.

I have three SSD machines that I run portage in tmpfs and have not run into this issue... "can't reproduce" on 3.5.7 yet...

Also can you unmount the tmpfs and remount it? Though lsof should tell you if you can or not...

(just to cover all bases, it's not a swapfile on tmpfs right?)
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
Eversor
n00b
n00b


Joined: 28 Jun 2006
Posts: 17

PostPosted: Fri Jan 11, 2013 3:52 pm    Post subject: Reply with quote

eccerr0r wrote:
Did you try kernel 3.5.7? Might be a kernel regression too.

I have three SSD machines that I run portage in tmpfs and have not run into this issue... "can't reproduce" on 3.5.7 yet...

Also can you unmount the tmpfs and remount it? Though lsof should tell you if you can or not...

(just to cover all bases, it's not a swapfile on tmpfs right?)


No swap of any kind ATM. And if this was working ok, I would rarely need it. Mostly only use it when compiling stuff like open office or turning some big VMs on. Then I turn off swap afterwards.

Can I ask you to dd from /dev/zero 2GB into a file in your tmpfs, while you're compiling there? The trigger for me seems to be extra open files/I-O activity.

I was on 3.2 before, upgraded to 3.6 and just see the same behavior.
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3951
Location: USA

PostPosted: Fri Jan 11, 2013 4:27 pm    Post subject: Reply with quote

My tmpfs is only 2GB large on this machine I'm using now, so if I made a 2G file the emerge will fail... A 1G file seems to have no effect: I tried making a 1GB file and deleting it repeatedly and it hasn't failed to free the space yet...

This is a 4G RAM machine...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 8990

PostPosted: Sat Jan 12, 2013 1:12 am    Post subject: Reply with quote

_______0 wrote:
for portage, the proper way of using RAM is to add these two in make.conf:

Code:
PORTAGE_TMPFS="/dev/shm"
PORTAGE_TMPDIR="/dev/shm"


then you'll see /dev/shm/portage/ being used. This way is automatic no need to manually mount a directory and no need to fiddle with mount options.
That is a way of using RAM, but I would not call it the proper way. Many users will mount /dev/shm as noexec. Portage temporary files must be written to a filesystem that permits execution.
Back to top
View user's profile Send private message
Eversor
n00b
n00b


Joined: 28 Jun 2006
Posts: 17

PostPosted: Mon Jan 14, 2013 11:03 pm    Post subject: Reply with quote

eccerr0r wrote:
My tmpfs is only 2GB large on this machine I'm using now, so if I made a 2G file the emerge will fail... A 1G file seems to have no effect: I tried making a 1GB file and deleting it repeatedly and it hasn't failed to free the space yet...

This is a 4G RAM machine...


That was what I was saying, I can't reproduce until I let xorg load and two things happen:

- /tmp/xsession-errors is open and written to
- /tmp/xdg-cache is written with various files from xdg compatible programs, like chromium

After this, any files being written, including caches from chromium in xdg-cache, are not getting freed :/
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3951
Location: USA

PostPosted: Tue Jan 15, 2013 12:54 am    Post subject: Reply with quote

Eversor wrote:
That was what I was saying, I can't reproduce until I let xorg load and two things happen:

- /tmp/xsession-errors is open and written to
- /tmp/xdg-cache is written with various files from xdg compatible programs, like chromium

After this, any files being written, including caches from chromium in xdg-cache, are not getting freed :/


Wait, what are you saying now? This seems contradictory, or are these not the files causing issues? If you have X11 open and it's writing log files and keeping the handles open, deleting them will not free the space by design, until the writing program exits. But generally the files are small - are they growing to GB-sized files due to some other issue? If this is the case, this is the underlying problem. Perhaps Chromium is writing GB sized files there but these should clear up once you quit Chromium.

So what exactly are you doing? Does the space free up once you quit X11? As said, I cannot reproduce the problem by creating a file and deleting it over and over again - if this had been able to cause the problem, then this is a real kernel bug. But it's not... please give a more detailed method of how to reproduce the issue.

BTW, I don't do xsession-errors or xdg-cache move on any of my SSD-only machines (32G, 128G, and an 180G). Only portage tmp is moved to tmpfs when possible, I do nothing else special. So far I have negligible wear on my SSDs, the major endurance killers currently are just multi-GB software installs.

Though I can't see why xsession-errors or xdg-cache moves would cause problems, could you restore them to the default locations and is the problem reproduceable in the default setup?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
blain3
n00b
n00b


Joined: 02 Dec 2012
Posts: 30
Location: Southern Pensylvania

PostPosted: Tue Jan 15, 2013 3:25 am    Post subject: Reply with quote

Quote:
These are my relevant sections of fstab:

none /tmp tmpfs size=7G 0 0
# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
# use almost no memory if not populated with files)
shm /dev/shm tmpfs nodev,nosuid,noexec 0 0

I tried _______0's suggestion and files created by portage are correctly freed. Guess it's either a mount option or I should not be using "none" but "shm". Thoughts?

EDIT:

About TRIM, I've added that to the FSTAB entry, completely forgot about that.


Correct me if im wrong but should anything running off your ram be this:
tmpfs /tmp tmpfs noatime,size=7G 0 0
notice that its not labeled as "none", duno if thats going to make any difference, also i believe it defaults at 8G without the "size" variable? i could be wrong. Also how much ram do you have exactly? And where is your /var/tmp mounted? because /var/tmp is a much longer lasting tmp system then the others from what i understand. Personally even with 16G of 1600 DDR3 i have all of my tmp systems using ram, I also keep a 32G swap partition on my second HD, and i can emerge on several terminals at once, and my system rarely uses the swap from what i can see with system monitor. I have also read, i/o scheduler's can bottleneck resources if not managed correctly. One last thing to look at possibly? is this: http://www.redips.net/linux/tmpwatch-delete-files/
Hope this helps, alas im not very good with all this yet so please, if im wrong by all means, correct me. cheers
_________________
#uname -s -m -i -p -v -o -n
Linux
x86_64
GenuineIntel
Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
#1 SMP PREEMPT Mon Nov 19 11:17:39 UTC 2012
GNU/Linux
xobtihs
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3951
Location: USA

PostPosted: Tue Jan 15, 2013 8:59 pm    Post subject: Reply with quote

Tmpfs defaults to half physical memory size if none is specified. I actually specified 1.5GB on my 2GB netbook to fit more of portage temp in RAM without overflowing the tmpfs (1GB isn't quite enough sometimes).

The labelling shouldn't matter, it's just for decorative purposes/making it clear what the temp filesystem is...

As for the OP - no NFSD running? I had some some strange behavior due to nfsd.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
blain3
n00b
n00b


Joined: 02 Dec 2012
Posts: 30
Location: Southern Pensylvania

PostPosted: Tue Jan 15, 2013 11:32 pm    Post subject: Reply with quote

eccerr0r wrote:
Tmpfs defaults to half physical memory size if none is specified. I actually specified 1.5GB on my 2GB netbook to fit more of portage temp in RAM without overflowing the tmpfs (1GB isn't quite enough sometimes).

The labelling shouldn't matter, it's just for decorative purposes/making it clear what the temp filesystem is...

As for the OP - no NFSD running? I had some some strange behavior due to nfsd.


Yes i realized that soon after i posted and forgot, 50% of mine is 8G so yeah my bad on that. Only reason i brought up the labeling is because if i edit fstab in vim it shows up a different color if i use none, and for some reason even shm is colored over in red, so i truly wasn't sure on that, just thinking out loud.. because if i use tmpfs instead of none its the default color and not white. As for clearing ram / tmp files yscou could easily script that into a single command i would assume, name it, chmod a+x, mv to /bin and run as needed or even run on a timer if all else fails. Im no expert but i'v made a few bash scripts with Geany and used them, iirc something like cron daemon would run it on a timer or some such? worth looking into. ( and no im not sure if its cron or something else ). good luck, hope you figure it out.
_________________
#uname -s -m -i -p -v -o -n
Linux
x86_64
GenuineIntel
Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
#1 SMP PREEMPT Mon Nov 19 11:17:39 UTC 2012
GNU/Linux
xobtihs
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum