Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Gento workstation going down
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Thu Nov 08, 2012 9:31 am    Post subject: Gento workstation going down Reply with quote

Hello,
I have some issues with one workstation here in my lab. It is a fujitsu workstation with two xeon X5690 cpus and 192GiB of ram. It is used to run our physics simulator (a really heavy task). Sometimes this computer is completely unresponsive. I can't access it via ssh nor with direct physical access. I suspect that the ram is completely filled and the heavy swap (4GiB swap area) usage and the system becomes unresponsive because of the I/O activity. Now the questions:

1)Is there a written log that survives a reboot that can help me understand what is happening?
2)How can I prevent this issue from happening? I would like something like "if main memory gets 99% filled then kill the process who is using more memory than any other. no swap usage is allowed"

EDIT:
Digging in /var/log/kern.log I've found these messages

Code:

Nov  8 08:20:22 localhost kernel: [515202.085852] INFO: rcu_bh detected stall on CPU 8 (t=0 jiffies)
Nov  8 08:20:22 localhost kernel: [515202.085857] INFO: rcu_bh detected stall on CPU 9 (t=0 jiffies)
Nov  8 08:20:22 localhost kernel: [515202.085860] INFO: rcu_bh detected stall on CPU 11 (t=0 jiffies)
Nov  8 08:20:22 localhost kernel: [515202.085864] INFO: rcu_bh detected stall on CPU 7 (t=0 jiffies)
Nov  8 08:20:22 localhost kernel: [515202.085867] INFO: rcu_bh detected stall on CPU 10 (t=0 jiffies)
Nov  8 08:20:22 localhost kernel: [515202.085869] sending NMI to all CPUs:
Nov  8 08:20:22 localhost kernel: [515202.085875] NMI backtrace for cpu 0
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2280
Location: Adendorf, Germany

PostPosted: Thu Nov 08, 2012 7:35 pm    Post subject: Reply with quote

These messages belong to the RCU's CPU stall detector. As those messages can happen on very high loads if preemption is disabled, enabling preemption might work.
On the other hand, the backtraces show which functions cause the stall, the offending function should be near the top.

This is the description of the stall detector: http://www.kernel.org/doc/Documentation/RCU/stallwarn.txt - I would have to copy its text, which is highly specialized, thus a link seems to be better. ;)
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Thu Nov 08, 2012 10:38 pm    Post subject: Reply with quote

Problem is that the last line before the one I posted is dated November 6th and I have no lines after. My preemption model is Voluntary Kernel Preemption
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Fri Nov 09, 2012 12:33 am    Post subject: Reply with quote

Some ideas:

1) You can deactivate swap before running your process using "swapoff", or better yet get rid of it all together. If you have 192GiB of RAM then a 4GiB swap area isn't useful for much.

2) You can use "nice" to reduce the priority of your simulation. If you aren't using the computer, it will use up all the CPU, but then when you want to ssh in it will be demoted and your ssh process will get priority.

3) Here is a page on limiting process time and memory consumption. Unfortunately, it's not so easy ... resource management has always been a weak point of *nix-like operating systems.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2280
Location: Adendorf, Germany

PostPosted: Fri Nov 09, 2012 8:42 am    Post subject: Reply with quote

And further, if preemption is not a problem, the stack traces are what should be analyzed.

How high is the RAM footprint of your simulator? It might be the other way round, you have not enough swap.

On our database servers we have 131 GiB of memory, and the 64GiB swap is used now and so often on very high loads with many clients doing a lot of work.

Further there might be a problem with memory allocation in the simulator as well. The problem is, that memory can get fragmented. And a malloc() will try to get the specified amount of continuous memory. If there isn't enough continuous memory for a large allocation, the allocator will try to reorganize pages of used memory to get a large enough area.
In the worst case a program can terminate with "out of memory" although several GiB are officially free. We had this once, but it is very rare case these days. (But you should have seen the face of our CTO that day. *hehe*)

However, this is all shooting in the dark, actually.
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Fri Nov 09, 2012 12:32 pm    Post subject: Reply with quote

Yamakuzure wrote:
And further, if preemption is not a problem, the stack traces are what should be analyzed.
I can't find any stack trace

Quote:
How high is the RAM footprint of your simulator? It might be the other way round, you have not enough swap.
That is unpredictable. Could range from 4 to 90 GiB. It depends on what components are activated and on how many people are using this workstation

Quote:
Further there might be a problem with memory allocation in the simulator as well. The problem is, that memory can get fragmented. And a malloc() will try to get the specified amount of continuous memory. If there isn't enough continuous memory for a large allocation, the allocator will try to reorganize pages of used memory to get a large enough area.
Shouldn't the 64 bits of address space mitigate this effect?

Quote:
However, this is all shooting in the dark, actually.
I know. Are there any logging options I could activate?
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Fri Nov 09, 2012 1:04 pm    Post subject: Reply with quote

You might have better results if you lower sysctl vm.min_free_kbytes and/or enable memory compaction.
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Sun Nov 11, 2012 9:42 pm    Post subject: Reply with quote

Hypnos wrote:
1) You can deactivate swap before running your process using "swapoff", or better yet get rid of it all together. If you have 192GiB of RAM then a 4GiB swap area isn't useful for much.
Could try

Quote:
2) You can use "nice" to reduce the priority of your simulation. If you aren't using the computer, it will use up all the CPU, but then when you want to ssh in it will be demoted and your ssh process will get priority.
Is there a way to do this automatically? (Or, increase priority of sshd process)

Ant P. wrote:
You might have better results if you lower sysctl vm.min_free_kbytes and/or enable memory compaction.

1)Lowering min_free_kbytes won't leave even less memory for other process? Default value is around 9 megs, which is already pretty low
2)How can I enable memory compaction?
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Sun Nov 11, 2012 10:16 pm    Post subject: Reply with quote

devilheart wrote:
Is there a way to do this automatically? (Or, increase priority of sshd process)

I'm not sure what "automatically" means here. You usually just do "nice <command>", then the process and all daughter processes have the same nice level.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Mon Nov 12, 2012 8:27 am    Post subject: Reply with quote

Hypnos wrote:
devilheart wrote:
Is there a way to do this automatically? (Or, increase priority of sshd process)

I'm not sure what "automatically" means here. You usually just do "nice <command>", then the process and all daughter processes have the same nice level.
I mean without using nice, because nobody will remember to use it. Maybe some sysctl trick? All non-root processes should have low proprity wrt ssh processes
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Mon Nov 12, 2012 8:39 am    Post subject: Reply with quote

You can modify /etc/security/limits.conf to reduce the default priority of the processes of certain users, as explored in this thread (presumably sshd and your simulation are run under different UIDs). You can also try to limit the memory used and other quantities, subject to the provisos in my link above.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Mon Nov 12, 2012 3:00 pm    Post subject: Reply with quote

Hypnos wrote:
You can modify /etc/security/limits.conf to reduce the default priority of the processes of certain users, as explored in this thread (presumably sshd and your simulation are run under different UIDs). You can also try to limit the memory used and other quantities, subject to the provisos in my link above.
This is a bit tricky. This workstation is accessed only via ssh. To change the priority would mean that login shells and simulations will have the same priority, which is pointless
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Mon Nov 12, 2012 3:13 pm    Post subject: Reply with quote

Yeah, I don't know how to have two different process have two different priorities for the same user by default.

If you as root want to ssh in to kill runaway processes then it's easy by the method I describe. This is consistent with the Unix tradition, where users are free to harm themselves, but he can be sandboxed and root can deign to help the user out.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Mon Nov 12, 2012 4:32 pm    Post subject: Reply with quote

Hypnos wrote:
Yeah, I don't know how to have two different process have two different priorities for the same user by default.

If you as root want to ssh in to kill runaway processes then it's easy by the method I describe. This is consistent with the Unix tradition, where users are free to harm themselves, but he can be sandboxed and root can deign to help the user out.
It's worth a shot. Let's hope this will work if the issue appears again
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2280
Location: Adendorf, Germany

PostPosted: Tue Nov 13, 2012 8:24 am    Post subject: Reply with quote

Well, maybe you should not concentrate too hard on cpu stall warnings.
Paul E. McKenney wrote:
(...)This is a debugging feature, not something that non-kernel-hackers would be expected to care about.
This feature can detect looping CPUs in !PREEMPT builds and looping CPUs with preemption disabled in PREEMPT builds. (...)
And if the stalling makes the system hangs, a kubuntu user had this idea:
albandy wrote:
Re: Disabling the CPU stall test
as root:

echo 1 > /sys/module/rcutree/parameters/rcu_cpu_stall_suppress

Then try the system (don't reboot it, simply work with it). If all works fine, set it in sysctl.conf
Maybe it is worth to take a look into /sys/module/rcutree/parameters/rcu_cpu_stall_timeout what the timeout is anyway?

However, if the stall_suppress does work, a high priority for root on sshd might help getting back into the system like Hypnos wrote.
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Tue Nov 13, 2012 8:48 am    Post subject: Reply with quote

Today it happened again. This time there are no suspicious log messages (no mention to stalling or NMI). The workstation is unresponsive to both ssh and direct physical login. Are there any kernel logging options I could enable? Some sysrq magic?
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Tue Nov 13, 2012 9:02 am    Post subject: Reply with quote

If the keyboard is unresponsive there is little you can do other than magic SysRq or a hard reset. According to the Wikipedia sysrq command listing 'f' and 'k' may be useful to you. Otherwise, will have to use 's', 'u' and 'b'.

As for logging, look at the conf file for your system logger, and make sure all kernel messages are recorded somewhere. I use metalog, and /etc/metalog.conf specifies all critical (level 1) kernel messages are in /var/log/kernel. The level can be reduced (e.g., to 6) to get more info.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Tue Nov 13, 2012 2:12 pm    Post subject: Reply with quote

I use rsyslog. I've setted up an all-catching rule. Another thing, I've noticed that when the workstation becomes unresponsive, the display is in stand-by mode and I cannot wake it up, so I can't read the messages written by a sysrq combination. Using
Code:
setterm -powersave off

I get
Code:
setterm: cannot (un)set powersave mode: Invalid argument

X is not running

EDIT:
nevermind, doing this via ssh won't work
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2280
Location: Adendorf, Germany

PostPosted: Wed Nov 14, 2012 10:31 am    Post subject: Reply with quote

Does "unresponsive" mean that it takes a long time, or can't you get in using ssh at all? If you can get in, what does top/htop and ps say about your top processes like the simulator? Are they in interruptible sleep ("D") or even disfunctional ("[Z]ombie") ?

If it is a "D", and if it is a normal case where IO is responsible, you might be able to SysRq "thaw" the D-state with 'j' - Forcibly "Just thaw it" - filesystems frozen by the FIFREEZE ioctl.

Edith wants to add: Of course the thawing of the filesystem freezes do not resolve "D" states of processes, it just might help if the sleep is caused by a frozen filesystem.
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
devilheart
l33t
l33t


Joined: 17 Mar 2005
Posts: 848
Location: Villach, Austria

PostPosted: Wed Nov 14, 2012 9:37 pm    Post subject: Reply with quote

By unresponsive I mean:
1) unreachable via network. Every connection attempt times out
2) hitting any key on the keyboard physically connected to the workstation doesn't wake the display from sleep
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum