View previous topic :: View next topic |
Author |
Message |
Martux Veteran
Joined: 04 Feb 2005 Posts: 1917
|
Posted: Thu Dec 24, 2009 9:22 am Post subject: Why does my system freeze when 1/2 RAM is used? |
|
|
Hi!
I´ve got a Intel Quad Q9450 with 4 gigs of Ram. I also added 2 gigs of swap.
/var/tmp/portage is mounted in tmpfs (3500MB).
Every now and then (when compiling boost, for example) my system totally freezes for 10-20 m.
I can see RAM usage increase till ~2000MB aqnf then this happens. But why?
I´ve never seen that gkrellm showed more than 2000MB RAM used. _________________ "Coincidence is God's way of remaining anonymous."
Albert Einstein
"The road to success is always under construction" |
|
Back to top |
|
|
Mike Hunt Watchman
Joined: 19 Jul 2009 Posts: 5287
|
Posted: Thu Dec 24, 2009 2:20 pm Post subject: |
|
|
Hmmm, weird because I have seen boost build just fine on 1Gig RAM boxes no problem.
Maybe something in your emerge --info might provide a clue. |
|
Back to top |
|
|
pigeon768 l33t
Joined: 02 Jan 2006 Posts: 683
|
Posted: Thu Dec 24, 2009 10:45 pm Post subject: |
|
|
Does memtest86+ turn up any errors? Does the compile eventually complete? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu Dec 24, 2009 11:58 pm Post subject: |
|
|
I have an older Q6600 with 4GB RAM, 32-bit install. I'm sure I've compiled boost many times, but it's not my biggest consumer of RAM. I can run two copies of WoW on this machine and each copy uses at least 1-2GB RAM after a while, and sometimes runs out of RAM (wine memory leak?)...affecting just that instance of WoW.
When you say "system totally freezes" it's confusing when you then continue with "for 10-20 m" ... when a system freezes, usually it implies it's dead forever, do you mean it freezes after 10-20 minutes? In this case I'd look to see if your chipset is overheating or not? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
beli.sk n00b
Joined: 25 Dec 2009 Posts: 11
|
Posted: Fri Dec 25, 2009 6:54 pm Post subject: |
|
|
The tmpfs can be eating quite a lot of memory, and the rest (500 MB?) may not be enough for the processes. Try decreasing the tmpfs max size to for example 2GB, or less, just for testing. The meters may, or may not, count the cache memory (used for tmpfs) to the memory usage shown. Or the problem can be somewhere else, not memory related. _________________ Beli | independent IT consultant | public(a)beli.sk | www.beli.sk |
|
Back to top |
|
|
Martux Veteran
Joined: 04 Feb 2005 Posts: 1917
|
Posted: Fri Dec 25, 2009 7:32 pm Post subject: |
|
|
Hi all!
Thanks for all the input.
Tonight I will emerge -e my system to hopefully fix another problem I have with korganizer.
After that, I am going to fiddle with the tmpfs size a bit.
If that does not fix it, I´ll gonna test with memtest.
To make clearer what I meant with "freezing":
The system hangs when 2gig of memory are used (as shown by gkrellm). I then can see how swap space is occupied. This brings the system to halt for some minutes, mouse stops moving and music stops.
If I do nothing, it will work again after a while.
So, I need some time for all of your suggestions.
Thanks,
Marcus _________________ "Coincidence is God's way of remaining anonymous."
Albert Einstein
"The road to success is always under construction" |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Fri Dec 25, 2009 9:21 pm Post subject: |
|
|
Okay, that clarification is better. If it's really slow (meaning, it eventually comes back) then it's more likely a software issue (or poorly behaved software due to hardware fault).
When I run out of swap and RAM it somewhat behaves much like how it's described - freezes for a while, as the computer freaks out trying to figure out what to do when there's no more free ram and it's requested.
Easier to trigger this OOM condition where it slows really badly as it tries to evict text pages for data/new text. Having gobs of swap helps this, I found. Having no swap *really* behaves poorly. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Martux Veteran
Joined: 04 Feb 2005 Posts: 1917
|
Posted: Sat Dec 26, 2009 2:10 pm Post subject: |
|
|
Well, fiddling with tmpfs size did not help. I came to a point, when it was set to 1500MB, when boost simply stated it ran out of space. Neither 200MB or 2500MB solved the hanging.
Memtest86+ did not find any errors after 3 passes and several hours running.
Maybe the hanging is not exactly related to RAM usage, but I saw insanely high "load" values: 31 (!!!)
Shouldn´t the "load average" parameter take care of this? I set it to 5.0 (btw, setting "jobs" to 2 doesn´t help either).
So, running out of options right now, here´s the output of emerge --info:
I really hope you guys have more ideas!
Code: |
Portage 2.2_rc61 (default/linux/amd64/10.0/desktop, gcc-4.4.2, glibc-2.11-r1, 2.6.32-gentoo x86_64)
=================================================================
System uname: Linux-2.6.32-gentoo-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q9450_@_2.66GHz-with-gentoo-2.0.1
Timestamp of tree: Sat, 26 Dec 2009 14:00:01 +0000
ccache version 2.4 [enabled]
app-shells/bash: 4.0_p35
dev-java/java-config: 2.1.10
dev-lang/python: 2.6.4, 3.1.1-r1
dev-util/ccache: 2.4-r8
dev-util/cmake: 2.8.0
sys-apps/baselayout: 2.0.1
sys-apps/openrc: 0.6.0
sys-apps/sandbox: 2.2
sys-devel/autoconf: 2.64
sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils: 2.20
sys-devel/gcc-config: 1.4.1
sys-devel/libtool: 2.2.6b
virtual/os-headers: 2.6.30-r1
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=5 --load-average=5.0 --with-bdeps=y --keep-going"
FEATURES="assume-digests ccache distlocks fixpackages news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="ftp://ftp.gentoo.mesh-solutions.com/gentoo/distfiles/ http://mirrors.sec.informatik.tu-darmstadt.de/gentoo/ http://distfiles.gentoo.org http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LANG="de_DE@euro.UTF-8"
LC_ALL="de_DE.UTF-8"
LDFLAGS="-Wl,-O1"
LINGUAS="de"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac aiglx alsa amd64 asf audiofile bash-completion berkdb bitmap-fonts bzip2 cairo cdda cdparanoia cdr cli cracklib css cups custom-cflags custom-cxxflags cxx dbus de_DE digest divx dri dts dvd dvdr dvdread eds emboss encode evo fam ffmpeg firefox flac foomaticdb fortran fuse gdbm gif gimp glib glibc-omitfp gpm hal hddtemp iconv id3tag java jpeg jpeg2k kde kdehiddenvisibility kdexdeltas lame libnotify mad mikmod mmx mng modules mp3 mp4 mpeg mplayer mudflap multilib ncurses nls noamazon nptl nptlonly nsplugin nvidia ogg opengl openmp pch pcre pdf perl pic png ppds pppd python qt3support qt4 quicktime readline reflection sdl session smp spell spl srt sse sse2 sse3 ssl ssse3 startup-notification stream svg symlink sysfs tcpd theora threads thunar tiff transparency truetype truetype-fonts twolame type1-fonts unicode usb userlocales vcd vdpau vmware vorbis wavpack wmp x264 xcomposite xine xinerama xml xorg xulrunner xv xvid xvmc zlib" ALSA_CARDS="snd-ca0106" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="de" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nv nvidia"
Unset: CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
|
_________________ "Coincidence is God's way of remaining anonymous."
Albert Einstein
"The road to success is always under construction" |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Mon Dec 28, 2009 2:05 am Post subject: |
|
|
Question still remains - how deeply are you into swap when this happens? Was your swap partition full?
If all your ram was used to store compile temp files, did it run out of space to work with compiling? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Martux Veteran
Joined: 04 Feb 2005 Posts: 1917
|
Posted: Mon Dec 28, 2009 11:18 am Post subject: |
|
|
Yes, indeed swap space is used. I´ve got 2GB.
Strange with that is, that I have set vm.swappiness to 0 and it begins swapping like hell when RAM is used ~1800MB. How could I take control of the load better? _________________ "Coincidence is God's way of remaining anonymous."
Albert Einstein
"The road to success is always under construction" |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Mon Dec 28, 2009 6:44 pm Post subject: |
|
|
You *want* it to swap versus text paging. Swap is much faster than text paging.
How does the machine behave if you don't use tmpfs for /var/tmp/portage?
For me in all honesty I don't believe in having portage use only RAM for building new packages. This is because disk RAM cache usually does a good job as it is, dealing with minimizing disk writes -- it will still write stuff to disk that's unnecessary but so what? It's clearly not perfect, after all it's a heuristic and balancing act, but it's much better than if you were building with absolutely no disk cache at all (or equivalently, with insufficient RAM).
If your load average is hitting 31 while emerge with -j5, likely means that the CPU can't get forward progress done because of RAM constraints. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Tue Dec 29, 2009 6:48 pm Post subject: Re: Why does my system freeze when 1/2 RAM is used? |
|
|
Martux wrote: | I´ve never seen that gkrellm showed more than 2000MB RAM used. |
I'm a big fan of gkrellm but it seems to consistently under-report ram usage when compared to conky or free or htop. |
|
Back to top |
|
|
Martux Veteran
Joined: 04 Feb 2005 Posts: 1917
|
Posted: Tue Dec 29, 2009 9:16 pm Post subject: |
|
|
Yeah it really seems to under-report RAM usage.
However, to test if this also is happening without tmpfs I threw the unholy triad of evil (tm) upon it:
emerge --oneshot boost wxGTK PyQt4
And well it really works well with -j5, even without ccache.
Top btw. showed a completely different RAM usage before system freezed than gkrellm, so it really seems I ran out of RAM. Thanks guys for helping me out.
It would still be lovely, if one could set per package tempdir for those special candidates
Thx, Marcus _________________ "Coincidence is God's way of remaining anonymous."
Albert Einstein
"The road to success is always under construction" |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Wed Dec 30, 2009 5:53 am Post subject: |
|
|
Boost is heavy duty that way. It freezes my 6GB system temporarily because of swap activity. It doesn't matter what your swappiness is. I think you are pumping too many cc1plus processes at a time. Reduce your MAKEOPTS a bit or get rid of the tmpfs on /var/tmp/portage. You are killing your ram on /var/tmp/portage as well as on a large number of 250MB+ cc1plus processes. System doesn't know whether to give ram to the files on /var/tmp/portage or to the cc1plus processes. So, kswapd is busy finding free pages (and wasting CPU cycles, remember pages are 4KB and there are a large number of them in modern systems to sift through) and there aren't too many free, so X gets paged out, kwin gets paged out, all your other programs get paged out, and hence you get the feeling of a freeze. |
|
Back to top |
|
|
Martux Veteran
Joined: 04 Feb 2005 Posts: 1917
|
Posted: Wed Dec 30, 2009 10:15 am Post subject: |
|
|
I found out, there is a possibility to set MAKEOPTS on a per package basis.
I just had to create /etc/portage/env/category/package and put "export MAKEOPTS=-j1" in there.
That makes perfect sense for me, as even setting down MAKEOPTS to -j2 stalled the computer for me.
For all smaller packages except boost, wxGTK and PyQt4 -j5 works really great. This way I don´t have to give up tmpfs for compiling either, because what devsk stated is true: The stalling is not a matter of RAM.
Thanks y´all, I suppose this thread is fixed _________________ "Coincidence is God's way of remaining anonymous."
Albert Einstein
"The road to success is always under construction" |
|
Back to top |
|
|
|