Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
MPI Program suddenly slows to a crawl
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
odeSolver
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jul 2010
Posts: 84
Location: NJ, USA

PostPosted: Sun Oct 13, 2013 2:51 am    Post subject: MPI Program suddenly slows to a crawl Reply with quote

Hi. I managed to write a Gentoo Linux MPI program in C which was chugging along nicely. I have a BASH file which is submitting executions. Each execution was running along nicely, spitting out 100,000 line data files in about 7 minutes each. Suddenly, during one of the jobs, it seemed to stop. Closer investigation reveals that it is now generating about 100 lines of data every minute - that's quite the slowdown. What can cause this? Where do I begin looking to get this running again?

Note that we do not have a job scheduler (or whatever they're called) - and there is no option to install one.

Edit: Oh, and I checked that there is still enough disk space (see https://forums.gentoo.org/viewtopic.php?p=7338412) - there's plenty.
_________________
Depserately needs help learning Gentoo Linux in order to use a 32-node cluster for my master's thesis in mathematics.
Back to top
View user's profile Send private message
Akkara
Bodhisattva
Bodhisattva


Joined: 28 Mar 2006
Posts: 6702
Location: &akkara

PostPosted: Sun Oct 13, 2013 3:16 am    Post subject: Reply with quote

Is it processing very small numbers (decaying exponentials, convergence to zero type of stuff)?

On some processors, the hardware floating-point doesn't handle "denormal numbers", so when one is encountered, it traps out to the OS for emulation. If many numbers are denormal, this would slow things down a lot.

(Denormals are different from 0 but smaller than the minimum exponent can represent. "Small" means less than (in magnitude) 2^-1022 for double; 2^-125 for float.)

A remedy for the slow-denormals problem - if less precise small-magnitude calculations are acceptable - is to turn on the flush-to-zero option in the floating point control word. I don't recall offhand how. Post back if you think this is the problem you're having; or search for it.
_________________
Many think that Dilbert is a comic. Unfortunately it is a documentary.
Back to top
View user's profile Send private message
odeSolver
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jul 2010
Posts: 84
Location: NJ, USA

PostPosted: Sun Oct 13, 2013 3:28 am    Post subject: Reply with quote

I don't think that's the problem. The program is solving a second order ODE with values that fluctuate between -1 and 1.

I think it could be a load problem. I suspect, some other student - besides me, that is - is not spending his (or her) Saturday night out chasing girls (or guys - or whatever they prefer). And this despite the nice weather here. Who the heck spends their Saturday night programming (when I have a thesis to work on)?

I found the top command and see that someone has 16 jobs sitting above mine on one of the processors! :(
_________________
Depserately needs help learning Gentoo Linux in order to use a 32-node cluster for my master's thesis in mathematics.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum