Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
2.4.22-gentoo-r2 eats memory?!
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
alexbuell
Guru
Guru


Joined: 18 Jul 2002
Posts: 490
Location: "Hemp"shire, UK

PostPosted: Sat Jan 03, 2004 10:14 pm    Post subject: Reply with quote

M104 wrote:

I tried it with the maximum bs=???m that it would let me (1600m, for the first run) and then increased it as more memory was reclaimed. I got to about 1720m before it wouldn't reclaim any more. At that point though, I was still out about 30 MB in "lost" memory and it took a while to get to that point. So I guess it is possible to reclaim most of the lost memory, but it's still pretty gross.


That fits in what I've found. That's probably the mimimum amount the kernel needs to keep open. I've posted to the linux-kernel mailing list about this problem, and hopefully there might be a patch soon to resolve this issue.
_________________
Cheers,
Alex.

Linux - the best text adventure game ever.


Last edited by alexbuell on Sun Jan 04, 2004 1:18 pm; edited 1 time in total
Back to top
View user's profile Send private message
alexbuell
Guru
Guru


Joined: 18 Jul 2002
Posts: 490
Location: "Hemp"shire, UK

PostPosted: Sun Jan 04, 2004 6:37 am    Post subject: Reply with quote

According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem.
_________________
Cheers,
Alex.

Linux - the best text adventure game ever.
Back to top
View user's profile Send private message
NeighborhoodGullwings
Apprentice
Apprentice


Joined: 05 Dec 2003
Posts: 159

PostPosted: Sun Jan 04, 2004 7:40 am    Post subject: Reply with quote

alexbuell wrote:
According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem.
There is? Is it a separate patch for 2.4.23 or is it included? Because I've been running the 2.4.24 pres and I've been having this problem too. Maybe I'll try 2.4.23 again to see if that corrects it.
Back to top
View user's profile Send private message
alexbuell
Guru
Guru


Joined: 18 Jul 2002
Posts: 490
Location: "Hemp"shire, UK

PostPosted: Sun Jan 04, 2004 1:17 pm    Post subject: Reply with quote

NeighborhoodGullwings wrote:
alexbuell wrote:
According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem.
There is? Is it a separate patch for 2.4.23 or is it included? Because I've been running the 2.4.24 pres and I've been having this problem too. Maybe I'll try 2.4.23 again to see if that corrects it.


Try the aa-sources - that'll have the fixes in them.
_________________
Cheers,
Alex.

Linux - the best text adventure game ever.
Back to top
View user's profile Send private message
Epyon
l33t
l33t


Joined: 11 Sep 2003
Posts: 754
Location: NJ, USA

PostPosted: Sun Jan 04, 2004 10:21 pm    Post subject: Reply with quote

alexbuell wrote:
According to replies I've had on the linux-kernel list, there is a fix in 2.6.1, and 2.4.23 has a fix for this problem.


Well it isn't in 2.6.1-rc1-mm1 because I'm still having the problem...
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Mon Jan 05, 2004 4:54 am    Post subject: Reply with quote

Alex,
can you post the link to the thread you created in kernel mailing list archives?

Also, anybody understands why in the old thread in the kernel mailing list http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0072.html the fix is to disable ntfs support? I have compiled ntfs in as a module, but it is not loaded so I do not think that recompiling without ntfs support would help.
Back to top
View user's profile Send private message
aengus9
n00b
n00b


Joined: 21 May 2002
Posts: 25

PostPosted: Tue Jan 06, 2004 4:41 am    Post subject: Reply with quote

I'm running 2.4.22-gentoo-r2 and did *not* compile ntfs support in the kernel and am having the same problems.
Back to top
View user's profile Send private message
NeighborhoodGullwings
Apprentice
Apprentice


Joined: 05 Dec 2003
Posts: 159

PostPosted: Tue Jan 06, 2004 8:27 am    Post subject: Reply with quote

mhodak wrote:

Also, anybody understands why in the old thread in the kernel mailing list http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0072.html the fix is to disable ntfs support? I have compiled ntfs in as a module, but it is not loaded so I do not think that recompiling without ntfs support would help.
That thread is over 2 years old and is highly likely a different problem than what we're experiencing now.
Back to top
View user's profile Send private message
alexbuell
Guru
Guru


Joined: 18 Jul 2002
Posts: 490
Location: "Hemp"shire, UK

PostPosted: Wed Jan 07, 2004 8:59 am    Post subject: Reply with quote

Red herring. The real problem appears to be a leak in ext3. Here's a patch for you to try.

Code:
diff -Nru a/fs/ext3/dir.c b/fs/ext3/dir.c
--- a/fs/ext3/dir.c     Wed Apr 23 21:37:34 2003
+++ b/fs/ext3/dir.c     Wed Apr 23 21:37:34 2003
@@ -501,7 +501,7 @@
 
 static int ext3_release_dir (struct inode * inode, struct file * filp)
 {
-       if (is_dx(inode) && filp->private_data)
+       if (filp->private_data)
                ext3_htree_free_dir_info(filp->private_data);
 
        return 0;



_________________
Cheers,
Alex.

Linux - the best text adventure game ever.
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Wed Jan 07, 2004 10:12 am    Post subject: Reply with quote

That is nice, but I have reiserfs and this bug affects me as well. It eats quite a lot of my memory overnight. Other people also see this on reiser, so it is not ext3 specific. It may be worse on ext3, but it is already bad on reiser.

BTW, has this patch been accepted to a recent kernel?
Back to top
View user's profile Send private message
wll
n00b
n00b


Joined: 25 Aug 2003
Posts: 38

PostPosted: Wed Jan 07, 2004 4:43 pm    Post subject: Reply with quote

Hey, Alex,

I'm about to compile a kernel for my new box after leaving behind a UML kernel. Any recommendations for which sources? It'd be nice if I knew I could leave this bug behind.

I'm a total newbie at the Linux kernel although not to compiling or software development in general. Something about this puzzles me -- why aren't more people affected and this bug more "web-visible"? Surely slocate is widespread enough that thousands of people should be experiencing this.

Thanks,
Wes
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Wed Jan 07, 2004 5:41 pm    Post subject: Reply with quote

I have been wondering the same thing, this must affect many linux users. The fact that this bug did not get enough publicity is probably because (as somebody pointed out previously), the memory "loss" seems to stabilize over time and kernel seems to reclaim part of the memory over time, so that my system, even if it is on over two weeks still runs. Each locatedb run increases my memory usage very close to 100 %, and after its done, starting any program, even xterm hits swap (should not happen with my 512 MB memory) and the system feels slow for a while. But in a matter of hours, kernel reclaims some of the "lost" memory (about 100MB in 24 hours) which is enough for many tasks (once GUI is up and running).
So I think that people that do not monitor their memory usage (I use gkrellm) do not notice this.
Back to top
View user's profile Send private message
aengus9
n00b
n00b


Joined: 21 May 2002
Posts: 25

PostPosted: Wed Jan 07, 2004 6:41 pm    Post subject: Reply with quote

That seems to make sense. I'm running on a Pentium 100 with 48 MB of memory, so my system is pretty much useless after this problem occurs. I might not notice the slight slowdown if I was running bigger hardware

I've removed the slocate from the cron.daily folder in /etc, which means that my system stays up and running now, but I have a question about how will I go about actually fixing this problem? Running 'emerge sync' causes the machine to go into thrash mode (Does emerge run updatedb?) and even after letting it run for two days never finished. So I'm not sure how I'll go about actually getting the update once it comes out.

Also, am I okay running a system without the daily updatedb job running?

Thanks
Back to top
View user's profile Send private message
MadScientist
n00b
n00b


Joined: 11 Apr 2003
Posts: 62

PostPosted: Wed Jan 07, 2004 10:07 pm    Post subject: Reply with quote

aengus9 wrote:
so, am I okay running a system without the daily updatedb job running?
The only side effect is that the "locate" command will have out of date information.
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Thu Jan 08, 2004 3:13 am    Post subject: Reply with quote

Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.

Mine: 2.4.22-gentoo-r2, ReiserFS
Back to top
View user's profile Send private message
ahr
n00b
n00b


Joined: 14 Jul 2002
Posts: 31
Location: Reykjavik, Iceland

PostPosted: Thu Jan 08, 2004 3:22 am    Post subject: Reply with quote

mhodak wrote:
Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.

Mine: 2.4.22-gentoo-r2, ReiserFS


2.6.1-rc1-mm2 I believe. EXT3.
512 MB ram. Fills up every midnight or so :)
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Thu Jan 08, 2004 3:44 am    Post subject: Reply with quote

Quote:

2.6.1-rc1-mm2 I believe. EXT3.
512 MB ram. Fills up every midnight or so


Wow, so this problem shows even with latest 2.6 kernel. According to Alex post,
it was supposed to be fixed in 2.6.1, but apparently it is not.
Back to top
View user's profile Send private message
r1chardgreen
n00b
n00b


Joined: 10 Feb 2003
Posts: 43

PostPosted: Thu Jan 08, 2004 4:09 am    Post subject: Reply with quote

Mhodak:

mhodak wrote:
Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.


Mine is 2.4.22-gentoo-r2, using ext3 - definitely - can reproduce with slocate, emerge rsync, etc.

I think I get this on one of my servers too which has very different hardware... testing there isn't really possible but the kernel it uses a lot more RAM than I would expect - though it's stable now using just under 1GB.
Back to top
View user's profile Send private message
aengus9
n00b
n00b


Joined: 21 May 2002
Posts: 25

PostPosted: Thu Jan 08, 2004 5:30 am    Post subject: Reply with quote

2.4.22-gentoo-r2, with ext3 on all partitions. Come midnight my Pentium 100 with 48 Meg doesn't stand a chance
Back to top
View user's profile Send private message
M104
Tux's lil' helper
Tux's lil' helper


Joined: 13 Jan 2003
Posts: 132
Location: Riverside, CA

PostPosted: Thu Jan 08, 2004 5:50 am    Post subject: Reply with quote

alexbuell wrote:
Red herring. The real problem appears to be a leak in ext3. Here's a patch for you to try.

Code:
diff -Nru a/fs/ext3/dir.c b/fs/ext3/dir.c
--- a/fs/ext3/dir.c     Wed Apr 23 21:37:34 2003
+++ b/fs/ext3/dir.c     Wed Apr 23 21:37:34 2003
@@ -501,7 +501,7 @@
 
 static int ext3_release_dir (struct inode * inode, struct file * filp)
 {
-       if (is_dx(inode) && filp->private_data)
+       if (filp->private_data)
                ext3_htree_free_dir_info(filp->private_data);
 
        return 0;



No, this patch is already in 2.6.0, my affected system, and not in 2.4.19-gentoo-r10, my unaffected system. :(
_________________
"Pulling together is the aim of despotism and tyranny. Free men pull in all kinds of directions."
Terry Pratchett, The Truth
Back to top
View user's profile Send private message
alexbuell
Guru
Guru


Joined: 18 Jul 2002
Posts: 490
Location: "Hemp"shire, UK

PostPosted: Thu Jan 08, 2004 11:52 am    Post subject: Reply with quote

How odd. That patch really does solve the problem for me and I'll know for sure by tomorrow if the problem is really gone. It takes two days before the problem manifests itself for me, and I applied the patch the day before on both boxes.
_________________
Cheers,
Alex.

Linux - the best text adventure game ever.
Back to top
View user's profile Send private message
Epyon
l33t
l33t


Joined: 11 Sep 2003
Posts: 754
Location: NJ, USA

PostPosted: Thu Jan 08, 2004 12:59 pm    Post subject: Reply with quote

M104 wrote:

No, this patch is already in 2.6.0, my affected system, and not in 2.4.19-gentoo-r10, my unaffected system. :(


I think that's because its not a problem with ext3 in 2.6.0. I and some other people here are using reiserfs exclusively and we're still having the problem.
Back to top
View user's profile Send private message
dudds
n00b
n00b


Joined: 03 Dec 2003
Posts: 28

PostPosted: Thu Jan 08, 2004 2:09 pm    Post subject: Reply with quote

Guys,

Not sure this is relevant or not, but I'm adding because I noticed memory problems with my PC. I am using kernel 2.4.20-gentoo-r6. I first noticed the problem after copying very large files (>700MB) between disk partitions. I have 1GB of RAM in my system. When I use nautilus to kick off a file copy and have the Gnome system monitor open, the used memory will steadily increase during the file copy until all bar about 20MB of available memory is used up.

I have also noticed this behaviour when using MPlayer to play movies. This is totally repeatable behaviour. Closing the above apps does not free all the memory again. I was unsure if the bug might be with Gnome or the System monitor.

Is there other commands I can use to verify the amount of memory used. I'm still a pretty new Linux user.

Cheers,
Dave
Back to top
View user's profile Send private message
imageek
Tux's lil' helper
Tux's lil' helper


Joined: 29 Aug 2002
Posts: 77
Location: Richardson, TX

PostPosted: Thu Jan 08, 2004 2:59 pm    Post subject: Reply with quote

i have been memory problems alot lately. Both my work box and home box. Mostly my work box i have noticed because i run an ftp server for some of our mandrake servers and i rsync with mandrakes site. I would come in on mondays and the box would be unresponsive. X crashed on it. And the log files full of "memory allocation" errors. I had never noticed or looked at how much memory was being used BEFORE these crashes.
Here are my specs

Work: Dell Optiplex GX400
kernel - 2.4.22-gentoo-r2
CPU - Intel(R) Pentium(R) 4 CPU 1400MHz
memory - 1gig
file system - ext3

Home: Custom built
kernel - 2.4.22-gentoo-r3
CPU - AMD Athlon XP 1700
memory - 1.5gig
file system - ext3

Let me know if i can provide anymore information.
Back to top
View user's profile Send private message
aengus9
n00b
n00b


Joined: 21 May 2002
Posts: 25

PostPosted: Fri Jan 09, 2004 2:54 am    Post subject: Reply with quote

Ok, I'm screwed. If anyone has any suggestions I'd really appreciate it. My extremely low end hardware (Pentium 100 with 48 Meg of RAM) simply can't finish any large task with the memory bug. 'Emerge sync' won't finish. "emerge gentor-sources-2.4.20-r7.ebuild" won't finish. So I can't even download new kernel sources, and even if I could I'm not sure the kernel would be able to compile.


I have a bzImage from my previous kernel (2.4.20-r7) but I deleted the source from /usr/src and the modules from /lib/modules.

I tried having grub point to the 2.4.20-r2 bzImage and copied everything from /lib/modules/2.4.22-gentoo-r2 to /lib/modules/2.4.20-gentoo-r7

This boots, but the module for my network card won't load. Are modules from 2.4.22 not compatible with modules for 2.4.20? Do I actually need the source directory to be in /usr/src?

Can anyone help me? I've got two bzImage files, one from 2.4.22-r2 and one from 2.4.20-r7. I have the /usr/src directory for 2.4.22-r2 and the /lib/modules directory for 2.4.22-r2. I can't emerge sync, I can't emerge *kernel-source* I'm kind of at a loss.

Thanks for any help
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page Previous  1, 2, 3, 4  Next
Page 2 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum