AMD64 system slow/unresponsive during disk access...

Bornio · Tux's lil' helper Joined: 16 Dec 2002 Posts: 129

Okay, here I bring more interesting news.
I ran dstat, and iostat. I noticed that during high iowait in iostat, dstat showed that the network card is going insane.
Here is the log:

Bornio · Tux's lil' helper Joined: 16 Dec 2002 Posts: 129

Okay, I think I might have a new, different lead.
I want to confirm:

Everyone who suffers from this issues by any chance is also using ReiserFS for his root ?
If yes: do you also have SMP enabled?

EDIT:
Please tell me what hardware you use too. Most important: CPU, Motherboard, Chipset and most important: IDE/SATA controller
You can find out about the Controller and others by invoking lcpci as such:

swimmer · Posted: Sun Jan 07, 2007 9:08 pm Post subject:

Nope - plain ext3 here ...

sachielle · n00b Joined: 20 Dec 2005 Posts: 23 Location: Paris, France

I don't use ReiserFS for root... And SMP is disabled for me.

Bornio · Tux's lil' helper Joined: 16 Dec 2002 Posts: 129

Jakub · Posted: Sun Jan 07, 2007 10:20 pm Post subject:

I have an ext3 partition and I do not have SMP enabled in the kernel. I do experience the problem.

My specs: AMD 3200+, NForce4, nv_sata, forcedeth.

Bornio · Tux's lil' helper Joined: 16 Dec 2002 Posts: 129

Update:

setting

swimmer · Posted: Mon Jan 08, 2007 12:37 am Post subject:

Jakub · Posted: Mon Jan 08, 2007 6:00 pm Post subject:

You have the same motherboard as me... personally I'd suspect the nvidia chipset (sata controller?)...

st0ne · n00b Joined: 22 Jan 2004 Posts: 18

hi,

no, same problem on my intel board with intel chipset... (Intel® P965 / ICH8R)
the same on another board with via-chipset...

greez

t0mcat · Posted: Mon Jan 08, 2007 8:50 pm Post subject:

anyone knows if the problem is reproducible rebuilding the system from scratch ?

i'm considering the option to blow up everything and restarting from 0, no config backup, no stage 4, no nothing, just plain reinstall.
_________________
il gattaccio
a.k.a etienne

Tonglebeak · Guru Joined: 23 Mar 2006 Posts: 356

I haven't really had this problem...

...except when updatedb runs. Then the entire system crawls. Top shows a low cpu usage, but updatedb is building an index of over 400000 files and sometimes locks up X for a minute of so. With that being said, I can usually go to another terminal and experience no issues in there, but then there's no GUI.

I'm using KDE+X...

psych · n00b Joined: 02 Sep 2006 Posts: 29

Hi!
I have the same problem with my new core2duo system (amd64).
Tested with 2.6.18-gentoo, 2.6.19-beyond2 and 2.6.20-rc3 vanilla .
The System seems to be a little bit "smoother" if i use nv instead of nvidia.

Psych

Bornio · Tux's lil' helper Joined: 16 Dec 2002 Posts: 129

Good news, everyone... I hope!

so I was thinking: everyone is using a wide verity of different kernels, FSs and hardware...
::sigh::

And then I had an :idea:

!

The only thing that is in common, is most probably the scheduler we are all using, which is by default, CFQ.
I remember of an old bug that used to cause CFQ to go insane under heavy load - causing very high IOWAIT.
This happens because CFQ tries to give the best performance for everyone. How? It lets everyone queue their writing. This is great, until the queue (buffer) runs full...

Long story short, I went ahead and set the Deadline elevator as the default one, in my grub.
I would love if you could do the same in your grub/lilo or kernel config, and let me know the results!

Can't wait to hear...!

st0ne · n00b Joined: 22 Jan 2004 Posts: 18

hi,

hm, i've tried the deadline-scheduler.... but on emerge sync the io-wait is high... i'll test with other programs...

greez

Bornio · Tux's lil' helper Joined: 16 Dec 2002 Posts: 129

st0ne · n00b Joined: 22 Jan 2004 Posts: 18

hi,

i think it's no difference than before...
maybe i will try an old kernel and look if it is the same...

krnlbg · Posted: Sat Jan 20, 2007 4:51 pm Post subject:

Hi all.
I have same problem. On x86 all works normaly.
My config:
2 drives in nvraid0 on nforce4 sata
jfs filesystem
Deadline I/O scheduler
Linux Kernel v2.6.19-gentoo-r4

segvault · n00b Joined: 20 Jan 2007 Posts: 2

Same problem here:

Linux localhost 2.6.19-ck2-r3 #1 SMP PREEMPT Fri Jan 12 01:20:53 EST 2007 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz GenuineIntel GNU/Linux

Core2 Duo overclocked to 3.2Ghz
Gigabyte DS3
2Gb RAM
500Gb SATA (dma enabled)
250Gb PATA (dma enabled)

I also get ridiculously low transfer rates.....
22mb./sec best case scenario btw different hds
15 Mb/sec (idle) between partitions in same hd...
5Mb/s ( bunch programs running)
0-2Mb/sec (while doing emerge --sync)

hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 2810 MB in 2.00 seconds = 1405.24 MB/sec
Timing buffered disk reads: 202 MB in 3.02 seconds = 66.96 MB/sec

in 2.6.17 the cached reads is double +- 2800mb/sec

I believe this is the same bug:

http://bugzilla.kernel.org/show_bug.cgi?id=7372

Unfortunately I can't downgrade to 2.6.17 because my network chipset wont work

Now all we have to do is hunt down this nasty bug introduced in the kernel 2.6.18

joaquin · n00b Joined: 05 Jan 2007 Posts: 16

I have the same symptoms: high iowait, 100% cpu load, freeze for a long time, etc.

I have and AMD Turion64X2 TL-50 with ATI chipset, so the problems is not nvidia, intel, etc. I have a PATA disk and windows run better than linux. I am frustrated, because i had an Athlon XP 2000+ with VIA chipset in a DFI mobo, and the performance was excellent, better than several windows with intel chipset.

But this PC is gone, and i upgraded to dual cpu & 64 bits and performance is very poor, and i don't know what i can do for correct the problem. I tested several kernels (vanilla, mm, ck, gentoo) without success, i have changed swap, filesystems, RAM size, drivers, and all is the same.

Please, if somebody find the solution, post it here for all us.

JanR · n00b Joined: 21 Jan 2007 Posts: 60

Hello,

on my MD64 system (Athlon 64 X2 4400+, kernel 2.6.16) I had similar problems with poor performance while I worked with very large files. I figured out that (at least for me) the problem is swap management. Starting with 2.6, linux preferes file cache in a way that pages from applications are swaped out to increase cache. If you work with large files (larger than available free memory) this leads to parallel swapping AND accessing the file. Linux 2.4 was much more responsive in such cases not doing that.

I solved the problem by adjusting SWAPPINESS. This parameter defines the behavior of filecache between the values 0 (same as 2.4 - file cache cannot force swapping out anything) to 100 (if more file cache is needed swap out). Default with 2.6 is 60 - I decreased this to 20 and got a much more responsive system especially while handling large files (video cutting... it is stupid to cache files that are larger than physical memory and are read linear).

I'm not sure if this improves the problem here or if this is really a related problem but try to decrease swappiness:

erikm · l33t Joined: 08 Feb 2005 Posts: 620

moesasji · Apprentice Joined: 10 May 2005 Posts: 263

I noticed that segvault provided a link to a bugreport in the kernel-bugzilla.
Its description and things that have been tried look awfully similar to what I experience and see for a lot of people in this topic.

That bugreport contains a patch by Andrew Morton.
It seems to be a small patch as it only changes the setting of one variable. Has anybody here tested that patch on his system??

ps) Note that the original reporter of that bug switched to another kernel and never tested the patch.

edit) I've tested the patch and rebooted. Problem unfortunately still remains.

Keruskerfuerst · Veteran Joined: 01 Feb 2006 Posts: 1722

I have a AMD Athlon64 3200, nForce 3 250GB and 1GB RAM. Geforce 5900XT graphics card. The NIC is onboard (forcedeth driver)
No SMP, CFQ scheduler and ext3 filesystem with journal writeback, dir_index and filetype.

During emerge -e world, the system has (sometimes) low I/O wait.

P.S.: max I/O wait during emerge --sync: 81,3%.

What about IOMMU on a AMD 64 system?

titusthefox · n00b Joined: 04 Mar 2005 Posts: 6 Location: PO Italy

I have tried vanilla kernel 2.6.19.2 but the problem is still. I have a notebook with nforce3 chipset and Athlon64 mobile CPU.
The kernel has is volountary preentive, and the system respond always but when I access to small files (loading java, emerge --sync) the I/O wait go up until 80%-90%. (I think that system respond because the kernel il preenptive)
I use raiserFs and Ext3.