View previous topic :: View next topic |
Author |
Message |
alexbuell Guru
Joined: 18 Jul 2002 Posts: 490 Location: "Hemp"shire, UK
|
Posted: Sat Jan 03, 2004 10:14 pm Post subject: |
|
|
M104 wrote: |
I tried it with the maximum bs=???m that it would let me (1600m, for the first run) and then increased it as more memory was reclaimed. I got to about 1720m before it wouldn't reclaim any more. At that point though, I was still out about 30 MB in "lost" memory and it took a while to get to that point. So I guess it is possible to reclaim most of the lost memory, but it's still pretty gross. |
That fits in what I've found. That's probably the mimimum amount the kernel needs to keep open. I've posted to the linux-kernel mailing list about this problem, and hopefully there might be a patch soon to resolve this issue. _________________ Cheers,
Alex.
Linux - the best text adventure game ever.
Last edited by alexbuell on Sun Jan 04, 2004 1:18 pm; edited 1 time in total |
|
Back to top |
|
|
alexbuell Guru
Joined: 18 Jul 2002 Posts: 490 Location: "Hemp"shire, UK
|
Posted: Sun Jan 04, 2004 6:37 am Post subject: |
|
|
According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem. _________________ Cheers,
Alex.
Linux - the best text adventure game ever. |
|
Back to top |
|
|
NeighborhoodGullwings Apprentice
Joined: 05 Dec 2003 Posts: 159
|
Posted: Sun Jan 04, 2004 7:40 am Post subject: |
|
|
alexbuell wrote: | According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem. | There is? Is it a separate patch for 2.4.23 or is it included? Because I've been running the 2.4.24 pres and I've been having this problem too. Maybe I'll try 2.4.23 again to see if that corrects it. |
|
Back to top |
|
|
alexbuell Guru
Joined: 18 Jul 2002 Posts: 490 Location: "Hemp"shire, UK
|
Posted: Sun Jan 04, 2004 1:17 pm Post subject: |
|
|
NeighborhoodGullwings wrote: | alexbuell wrote: | According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem. | There is? Is it a separate patch for 2.4.23 or is it included? Because I've been running the 2.4.24 pres and I've been having this problem too. Maybe I'll try 2.4.23 again to see if that corrects it. |
Try the aa-sources - that'll have the fixes in them. _________________ Cheers,
Alex.
Linux - the best text adventure game ever. |
|
Back to top |
|
|
Epyon l33t
Joined: 11 Sep 2003 Posts: 754 Location: NJ, USA
|
Posted: Sun Jan 04, 2004 10:21 pm Post subject: |
|
|
alexbuell wrote: | According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem. |
Well it isn't in 2.6.1-rc1-mm1 because I'm still having the problem... |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Mon Jan 05, 2004 4:54 am Post subject: |
|
|
Alex,
can you post the link to the thread you created in kernel mailing list archives?
Also, anybody understands why in the old thread in the kernel mailing list http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0072.html the fix is to disable ntfs support? I have compiled ntfs in as a module, but it is not loaded so I do not think that recompiling without ntfs support would help. |
|
Back to top |
|
|
aengus9 n00b
Joined: 21 May 2002 Posts: 25
|
Posted: Tue Jan 06, 2004 4:41 am Post subject: |
|
|
I'm running 2.4.22-gentoo-r2 and did *not* compile ntfs support in the kernel and am having the same problems. |
|
Back to top |
|
|
NeighborhoodGullwings Apprentice
Joined: 05 Dec 2003 Posts: 159
|
Posted: Tue Jan 06, 2004 8:27 am Post subject: |
|
|
mhodak wrote: |
Also, anybody understands why in the old thread in the kernel mailing list http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0072.html the fix is to disable ntfs support? I have compiled ntfs in as a module, but it is not loaded so I do not think that recompiling without ntfs support would help. | That thread is over 2 years old and is highly likely a different problem than what we're experiencing now. |
|
Back to top |
|
|
alexbuell Guru
Joined: 18 Jul 2002 Posts: 490 Location: "Hemp"shire, UK
|
Posted: Wed Jan 07, 2004 8:59 am Post subject: |
|
|
Red herring. The real problem appears to be a leak in ext3. Here's a patch for you to try.
Code: | diff -Nru a/fs/ext3/dir.c b/fs/ext3/dir.c
--- a/fs/ext3/dir.c Wed Apr 23 21:37:34 2003
+++ b/fs/ext3/dir.c Wed Apr 23 21:37:34 2003
@@ -501,7 +501,7 @@
static int ext3_release_dir (struct inode * inode, struct file * filp)
{
- if (is_dx(inode) && filp->private_data)
+ if (filp->private_data)
ext3_htree_free_dir_info(filp->private_data);
return 0;
|
_________________ Cheers,
Alex.
Linux - the best text adventure game ever. |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Wed Jan 07, 2004 10:12 am Post subject: |
|
|
That is nice, but I have reiserfs and this bug affects me as well. It eats quite a lot of my memory overnight. Other people also see this on reiser, so it is not ext3 specific. It may be worse on ext3, but it is already bad on reiser.
BTW, has this patch been accepted to a recent kernel? |
|
Back to top |
|
|
wll n00b
Joined: 25 Aug 2003 Posts: 38
|
Posted: Wed Jan 07, 2004 4:43 pm Post subject: |
|
|
Hey, Alex,
I'm about to compile a kernel for my new box after leaving behind a UML kernel. Any recommendations for which sources? It'd be nice if I knew I could leave this bug behind.
I'm a total newbie at the Linux kernel although not to compiling or software development in general. Something about this puzzles me -- why aren't more people affected and this bug more "web-visible"? Surely slocate is widespread enough that thousands of people should be experiencing this.
Thanks,
Wes |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Wed Jan 07, 2004 5:41 pm Post subject: |
|
|
I have been wondering the same thing, this must affect many linux users. The fact that this bug did not get enough publicity is probably because (as somebody pointed out previously), the memory "loss" seems to stabilize over time and kernel seems to reclaim part of the memory over time, so that my system, even if it is on over two weeks still runs. Each locatedb run increases my memory usage very close to 100 %, and after its done, starting any program, even xterm hits swap (should not happen with my 512 MB memory) and the system feels slow for a while. But in a matter of hours, kernel reclaims some of the "lost" memory (about 100MB in 24 hours) which is enough for many tasks (once GUI is up and running).
So I think that people that do not monitor their memory usage (I use gkrellm) do not notice this. |
|
Back to top |
|
|
aengus9 n00b
Joined: 21 May 2002 Posts: 25
|
Posted: Wed Jan 07, 2004 6:41 pm Post subject: |
|
|
That seems to make sense. I'm running on a Pentium 100 with 48 MB of memory, so my system is pretty much useless after this problem occurs. I might not notice the slight slowdown if I was running bigger hardware
I've removed the slocate from the cron.daily folder in /etc, which means that my system stays up and running now, but I have a question about how will I go about actually fixing this problem? Running 'emerge sync' causes the machine to go into thrash mode (Does emerge run updatedb?) and even after letting it run for two days never finished. So I'm not sure how I'll go about actually getting the update once it comes out.
Also, am I okay running a system without the daily updatedb job running?
Thanks |
|
Back to top |
|
|
MadScientist n00b
Joined: 11 Apr 2003 Posts: 62
|
Posted: Wed Jan 07, 2004 10:07 pm Post subject: |
|
|
aengus9 wrote: | so, am I okay running a system without the daily updatedb job running? | The only side effect is that the "locate" command will have out of date information. |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Thu Jan 08, 2004 3:13 am Post subject: |
|
|
Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.
Mine: 2.4.22-gentoo-r2, ReiserFS |
|
Back to top |
|
|
ahr n00b
Joined: 14 Jul 2002 Posts: 31 Location: Reykjavik, Iceland
|
Posted: Thu Jan 08, 2004 3:22 am Post subject: |
|
|
mhodak wrote: | Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.
Mine: 2.4.22-gentoo-r2, ReiserFS |
2.6.1-rc1-mm2 I believe. EXT3.
512 MB ram. Fills up every midnight or so |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Thu Jan 08, 2004 3:44 am Post subject: |
|
|
Quote: |
2.6.1-rc1-mm2 I believe. EXT3.
512 MB ram. Fills up every midnight or so
|
Wow, so this problem shows even with latest 2.6 kernel. According to Alex post,
it was supposed to be fixed in 2.6.1, but apparently it is not. |
|
Back to top |
|
|
r1chardgreen n00b
Joined: 10 Feb 2003 Posts: 43
|
Posted: Thu Jan 08, 2004 4:09 am Post subject: |
|
|
Mhodak:
mhodak wrote: | Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.
|
Mine is 2.4.22-gentoo-r2, using ext3 - definitely - can reproduce with slocate, emerge rsync, etc.
I think I get this on one of my servers too which has very different hardware... testing there isn't really possible but the kernel it uses a lot more RAM than I would expect - though it's stable now using just under 1GB. |
|
Back to top |
|
|
aengus9 n00b
Joined: 21 May 2002 Posts: 25
|
Posted: Thu Jan 08, 2004 5:30 am Post subject: |
|
|
2.4.22-gentoo-r2, with ext3 on all partitions. Come midnight my Pentium 100 with 48 Meg doesn't stand a chance |
|
Back to top |
|
|
M104 Tux's lil' helper
Joined: 13 Jan 2003 Posts: 132 Location: Riverside, CA
|
Posted: Thu Jan 08, 2004 5:50 am Post subject: |
|
|
alexbuell wrote: | Red herring. The real problem appears to be a leak in ext3. Here's a patch for you to try.
Code: | diff -Nru a/fs/ext3/dir.c b/fs/ext3/dir.c
--- a/fs/ext3/dir.c Wed Apr 23 21:37:34 2003
+++ b/fs/ext3/dir.c Wed Apr 23 21:37:34 2003
@@ -501,7 +501,7 @@
static int ext3_release_dir (struct inode * inode, struct file * filp)
{
- if (is_dx(inode) && filp->private_data)
+ if (filp->private_data)
ext3_htree_free_dir_info(filp->private_data);
return 0;
|
|
No, this patch is already in 2.6.0, my affected system, and not in 2.4.19-gentoo-r10, my unaffected system. _________________ "Pulling together is the aim of despotism and tyranny. Free men pull in all kinds of directions."
Terry Pratchett, The Truth |
|
Back to top |
|
|
alexbuell Guru
Joined: 18 Jul 2002 Posts: 490 Location: "Hemp"shire, UK
|
Posted: Thu Jan 08, 2004 11:52 am Post subject: |
|
|
How odd. That patch really does solve the problem for me and I'll know for sure by tomorrow if the problem is really gone. It takes two days before the problem manifests itself for me, and I applied the patch the day before on both boxes. _________________ Cheers,
Alex.
Linux - the best text adventure game ever. |
|
Back to top |
|
|
Epyon l33t
Joined: 11 Sep 2003 Posts: 754 Location: NJ, USA
|
Posted: Thu Jan 08, 2004 12:59 pm Post subject: |
|
|
M104 wrote: |
No, this patch is already in 2.6.0, my affected system, and not in 2.4.19-gentoo-r10, my unaffected system. |
I think that's because its not a problem with ext3 in 2.6.0. I and some other people here are using reiserfs exclusively and we're still having the problem. |
|
Back to top |
|
|
dudds n00b
Joined: 03 Dec 2003 Posts: 28
|
Posted: Thu Jan 08, 2004 2:09 pm Post subject: |
|
|
Guys,
Not sure this is relevant or not, but I'm adding because I noticed memory problems with my PC. I am using kernel 2.4.20-gentoo-r6. I first noticed the problem after copying very large files (>700MB) between disk partitions. I have 1GB of RAM in my system. When I use nautilus to kick off a file copy and have the Gnome system monitor open, the used memory will steadily increase during the file copy until all bar about 20MB of available memory is used up.
I have also noticed this behaviour when using MPlayer to play movies. This is totally repeatable behaviour. Closing the above apps does not free all the memory again. I was unsure if the bug might be with Gnome or the System monitor.
Is there other commands I can use to verify the amount of memory used. I'm still a pretty new Linux user.
Cheers,
Dave |
|
Back to top |
|
|
imageek Tux's lil' helper
Joined: 29 Aug 2002 Posts: 77 Location: Richardson, TX
|
Posted: Thu Jan 08, 2004 2:59 pm Post subject: |
|
|
i have been memory problems alot lately. Both my work box and home box. Mostly my work box i have noticed because i run an ftp server for some of our mandrake servers and i rsync with mandrakes site. I would come in on mondays and the box would be unresponsive. X crashed on it. And the log files full of "memory allocation" errors. I had never noticed or looked at how much memory was being used BEFORE these crashes.
Here are my specs
Work: Dell Optiplex GX400
kernel - 2.4.22-gentoo-r2
CPU - Intel(R) Pentium(R) 4 CPU 1400MHz
memory - 1gig
file system - ext3
Home: Custom built
kernel - 2.4.22-gentoo-r3
CPU - AMD Athlon XP 1700
memory - 1.5gig
file system - ext3
Let me know if i can provide anymore information. |
|
Back to top |
|
|
aengus9 n00b
Joined: 21 May 2002 Posts: 25
|
Posted: Fri Jan 09, 2004 2:54 am Post subject: |
|
|
Ok, I'm screwed. If anyone has any suggestions I'd really appreciate it. My extremely low end hardware (Pentium 100 with 48 Meg of RAM) simply can't finish any large task with the memory bug. 'Emerge sync' won't finish. "emerge gentor-sources-2.4.20-r7.ebuild" won't finish. So I can't even download new kernel sources, and even if I could I'm not sure the kernel would be able to compile.
I have a bzImage from my previous kernel (2.4.20-r7) but I deleted the source from /usr/src and the modules from /lib/modules.
I tried having grub point to the 2.4.20-r2 bzImage and copied everything from /lib/modules/2.4.22-gentoo-r2 to /lib/modules/2.4.20-gentoo-r7
This boots, but the module for my network card won't load. Are modules from 2.4.22 not compatible with modules for 2.4.20? Do I actually need the source directory to be in /usr/src?
Can anyone help me? I've got two bzImage files, one from 2.4.22-r2 and one from 2.4.20-r7. I have the /usr/src directory for 2.4.22-r2 and the /lib/modules directory for 2.4.22-r2. I can't emerge sync, I can't emerge *kernel-source* I'm kind of at a loss.
Thanks for any help |
|
Back to top |
|
|
|