AMD64 system slow/unresponsive during disk access...

Message

Bornio · Post by **Bornio** » Sat Dec 23, 2006 11:03 pm

Okay, here I bring more interesting news.
I ran dstat, and iostat. I noticed that during high iowait in iostat, dstat showed that the network card is going insane.
Here is the log:

Code: Select all

-------cpu0-usage--------------cpu1-usage------ --dsk/hde-----dsk/sda-----dsk/sdb-- --net/eth0->
usr sys idl wai hiq siq:usr sys idl wai hiq siq|_read _writ:_read _writ:_read _writ|_recv _send>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0   512k:   0     0 | 140k   20k>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |  58k   20k>
  2   1  97   0   0   0:  0   1   0  98   1   0|   0     0 :   0     0 :   0     0 |  37k   16k>
  1   0  99   0   0   0:  0   0   0  99   0   1|   0     0 :   0  4096B:   0     0 |  65k   17k>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |  63k   16k>
  0   0 100   0   0   0:  0   0   0  99   0   1|   0     0 :   0   448k:   0   256k|  63k   26k>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |  31k   21k>
  1   0  99   0   0   0:  2   0   0  98   0   0|   0     0 :   0     0 :   0     0 |  22k   13k>
  0   1  99   0   0   0:  0   0   0  98   1   1|   0     0 :   0     0 :   0     0 |  41k   12k>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |  34k   21k>
  0   0 100   0   0   0:  0   1   0  99   0   0|   0     0 :   0   448k:   0     0 |  39k   20k>
  1   0  99   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |  29k   14k>
  2   0  98   0   0   0:  0   0   0 100   0   0|   0     0 :   0  4096B:   0     0 |  20k   16k>
  0   1  99   0   0   0:  1   0   0  98   0   1|   0     0 :   0     0 :   0     0 |  21k   15k>
  0   0 100   0   0   0:  0   0   0  99   1   0|   0     0 :   0     0 :   0     0 |  11k 8144B>
  0   0 100   0   0   0:  0   0   0  99   0   1|   0     0 :   0   452k:   0   220k|  13k 4034B>
  1   0  99   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |8781B 6962B>
  0   0 100   0   0   0:  2   1   0  97   0   0|   0     0 :   0     0 :   0     0 |  20k 2864B>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |  13k 2326B>
  0   0 100   0   0   0:  0   0   0  99   0   1|   0     0 :   0     0 :   0     0 |  13k 8408B>
  1   0  99   0   0   0:  0   0   0 100   0   0|   0     0 :   0   448k:   0     0 |8986B 1922B>
  0   0 100   0   0   0:  0   0   0  99   1   0|   0     0 :   0     0 :   0     0 |7605B 1961B>
  0   1  99   0   0   0:  1   1   0  97   0   1|   0     0 :   0  4096B:   0     0 |8336B 2032B>
  0   0 100   0   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :   0     0 |9638B 7851B>
  1   0  99   0   0   0:  1   0   0  99   0   0|   0     0 :   0     0 :   0     0 |9590B 2140B>
 13   0  87   0   0   0:  6   1  74  17   0   2|   0     0 :   0   460k:   0  2824k|7597B   17k>
  5   0  95   0   0   0:  8   0  92   0   0   0|   0     0 :   0     0 :   0     0 |  19k   18k>
 11   2  87   0   0   0: 10   0  89   0   1   0|   0     0 :   0     0 :   0     0 |  26k   13k>

-------cpu0-usage--------------cpu1-usage------ --dsk/hde-----dsk/sda-----dsk/sdb-- --net/eth0->
usr sys idl wai hiq siq:usr sys idl wai hiq siq|_read _writ:_read _writ:_read _writ|_recv _send>
  7   1  91   1   0   0:  7   1  85   6   0   0| 7.4B    0 :  55k  247k:  11k   82k|   0     0 >
  1   0   0  99   0   0:  2   1   0  97   0   0|   0     0 :   0     0 :   0     0 |  17k   12k>
  2   0   0  98   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :  64k    0 |  18k 9683B>
  4   1   0  95   0   0:  1   0   0  97   1   1|   0     0 :   0     0 :   0     0 |  13k 4755B>
  2   0   0  98   0   0:  1   0   0  99   0   0|   0     0 :   0     0 :  64k    0 |  15k 3785B>
  2   0   0  98   0   0:  0   0   0 100   0   0|   0     0 :   0   704k:   0     0 |  16k 4238B>
  3   1   0  96   0   0:  3   1   0  96   0   0|   0     0 :   0     0 :  64k    0 |  29k 2168B>
  2   0   0  98   0   0:  0   0   0  99   0   1|   0     0 :   0     0 :   0     0 |  13k 4761B>
  3   1   0  96   0   0:  1   0   0  98   1   0|   0     0 :   0     0 :  64k    0 |  18k 3402B>
  4   0  62  34   0   0:  1   0   0  99   0   0|   0     0 :   0  4096B:   0     0 |  14k 1549B>
  2   0  23  75   0   0:  1   0   0  98   0   1|   0     0 :   0   456k:   0     0 |7119B 3011B>
  2   1   0  97   0   0:  2   1   0  96   0   1|   0     0 :   0     0 :  64k    0 |  11k 2956B>
  3   0   0  97   0   0:  1   0   0  99   0   0|   0     0 :   0  4096B:   0     0 |  12k 1331B>
  4   1   0  95   0   0:  2   0   0  97   1   0|   0     0 :   0     0 :  64k    0 |  10k 9016B>
  3   0   0  97   0   0:  3   0   0  97   0   0|   0     0 :4096B    0 :   0     0 |  12k 8525B>
  3   1   0  96   0   0:  3   1  10  85   0   1|   0     0 :   0   500k:  64k    0 |9600B 1214B>
  6   1   0  93   0   0:  3   0   0  97   0   0|   0     0 :   0     0 :   0     0 |4701B 1220B>
  2   0   0  98   0   0:  0   0   0 100   0   0|   0     0 :   0     0 :  64k    0 |  18k 2194B>
  4   0   0  96   0   0:  4   0   0  95   1   0|   0     0 :   0     0 :   0     0 |8713B 1005B>
  3   0   0  97   0   0:  2   0   0  97   0   1|   0     0 :   0  4096B:  64k    0 |8415B 1215B>
 15   6  56  24   0   0: 17   2  56  24   0   1|   0     0 :   0   488k:   0  2268k|  20k   20k>
  6   0  94   0   0   0:  6   0  93   0   0   1|   0     0 :   0     0 :  64k    0 |  20k   12k>
  4   1  95   0   0   0:  3   0  96   0   1   0|   0     0 :   0     0 :   0     0 |  27k 9179B>
  4   0  96   0   0   0:  9   1  90   0   0   0|   0     0 :   0     0 :  64k    0 |  34k 8443B>
  5   1  94   0   0   0:  4   0  95   0   0   1|   0     0 :   0     0 :   0     0 |  35k 9247B>
  4   0  87   9   0   0:  5   0  93   2   0   0|   0     0 :   0   340k:  64k 3460k|  30k   11k>
  8   2  90   0   0   0:  9   2  88   0   1   0|   0     0 :   0     0 :   0     0 |  28k 4992B>
 12   0  88   0   0   0:  8   1  90   0   0   1|   0     0 :   0     0 :  64k    0 |  21k 3410B>
  4   1  95   0   0   0:  7   0  93   0   0   0|   0     0 :   0     0 :   0     0 |  23k   12k>
  5   0  95   0   0   0:  1   1  98   0   0   0|   0     0 :   0     0 :  64k    0 |  23k 2921B>

Ideas???
Sigh....

Bornio · Post by **Bornio** » Sun Jan 07, 2007 8:00 pm

Okay, I think I might have a new, different lead.
I want to confirm:

Everyone who suffers from this issues by any chance is also using ReiserFS for his root ?
If yes: do you also have SMP enabled?

EDIT:
Please tell me what hardware you use too. Most important: CPU, Motherboard, Chipset and most important: IDE/SATA controller
You can find out about the Controller and others by invoking lcpci as such:

Code: Select all

lspci | grep ATA

Please reply!

Thanks,

Alex.

swimmer · Post by **swimmer** » Sun Jan 07, 2007 9:08 pm

Nope - plain ext3 here ...

sachielle · Post by **sachielle** » Sun Jan 07, 2007 9:14 pm

I don't use ReiserFS for root... And SMP is disabled for me.

Bornio · Post by **Bornio** » Sun Jan 07, 2007 9:14 pm

swimmer wrote:Nope - plain ext3 here ...

I did not see your post here before.
Are you too having issues with iowait at "random" (like, when running emerge sync, or using vmware...) ?

Also, please tell me what hardware do you use? Is it SATA?
What about the NIC?

Jakub · Post by **Jakub** » Sun Jan 07, 2007 10:20 pm

I have an ext3 partition and I do not have SMP enabled in the kernel. I do experience the problem.

My specs: AMD 3200+, NForce4, nv_sata, forcedeth.

Bornio · Post by **Bornio** » Mon Jan 08, 2007 12:10 am

Update:

setting

Code: Select all

blockdev --setra 16384 /dev/sda

Helped the situation quiet a bit. Mostly by making the "freeze" much shorter.

Though I am still looking for a definite solution, but without knowing the source...

swimmer · Post by **swimmer** » Mon Jan 08, 2007 12:37 am

Bornio wrote: I did not see your post here before.
Are you too having issues with iowait at "random" (like, when running emerge sync, or using vmware...) ?

Also, please tell me what hardware do you use? Is it SATA?
What about the NIC?

Ups sorry - perhaps I was only watching this thread and confused it with another one about performance issues on AMD64 :-/

But I *do* have problems during an emerge with playing movies with mplayer. I even experience performance problems in Quake3 when updatedb starts or my intrusion-detection-app (ossec) scans my system for any intrusions ... I did *not* investigate the case as deep as you did though.

My system:
CPU: AMD Athlon(tm) 64 Processor 3500+
Motherboard: Asus A8N-SLI
Chipset: CK804
NICs: CK804 onboard (forcedeth) + RTL-8139/8139C/8139C+ (8139too)
IDE: CK804 IDE with 1 Maxtor 6Y080P0 on channel 0 + 1 NEC DVD_RW ND-3550A on channel 1
SATA: CK804 Serial ATA Controller with 1 Maxtor 6Y250M0

SMP is enabled in the kernel but not used according to /proc/cpuinfo ...

HTH
swimmer

Jakub · Post by **Jakub** » Mon Jan 08, 2007 6:00 pm

You have the same motherboard as me... personally I'd suspect the nvidia chipset (sata controller?)...

st0ne · Post by **st0ne** » Mon Jan 08, 2007 6:04 pm

hi,

no, same problem on my intel board with intel chipset... (Intel® P965 / ICH8R)
the same on another board with via-chipset...

greez

t0mcat · Post by **t0mcat** » Mon Jan 08, 2007 8:50 pm

anyone knows if the problem is reproducible rebuilding the system from scratch ?

i'm considering the option to blow up everything and restarting from 0, no config backup, no stage 4, no nothing, just plain reinstall.

Tonglebeak · Post by **Tonglebeak** » Tue Jan 09, 2007 6:11 pm

I haven't really had this problem...

...except when updatedb runs. Then the entire system crawls. Top shows a low cpu usage, but updatedb is building an index of over 400000 files and sometimes locks up X for a minute of so. With that being said, I can usually go to another terminal and experience no issues in there, but then there's no GUI.

I'm using KDE+X...

psych · Post by **psych** » Tue Jan 09, 2007 9:20 pm

Hi!
I have the same problem with my new core2duo system (amd64).
Tested with 2.6.18-gentoo, 2.6.19-beyond2 and 2.6.20-rc3 vanilla .
The System seems to be a little bit "smoother" if i use nv instead of nvidia.

Psych

Bornio · Post by **Bornio** » Thu Jan 11, 2007 7:51 am

Good news, everyone... I hope!

so I was thinking: everyone is using a wide verity of different kernels, FSs and hardware...
::sigh::

And then I had an

!

The only thing that is in common, is most probably the scheduler we are all using, which is by default, CFQ.
I remember of an old bug that used to cause CFQ to go insane under heavy load - causing very high IOWAIT.
This happens because CFQ tries to give the best performance for everyone. How? It lets everyone queue their writing. This is great, until the queue (buffer) runs full...

Long story short, I went ahead and set the Deadline elevator as the default one, in my grub.
I would love if you could do the same in your grub/lilo or kernel config, and let me know the results!

Can't wait to hear...!

st0ne · Post by **st0ne** » Thu Jan 11, 2007 8:33 am

hi,

hm, i've tried the deadline-scheduler.... but on emerge sync the io-wait is high... i'll test with other programs...

greez

Bornio · Post by **Bornio** » Thu Jan 11, 2007 8:40 am

st0ne wrote: hm, i've tried the deadline-scheduler.... but on emerge sync the io-wait is high... i'll test with other programs...

How high was it? Also, despite it being high, did you feel any choppiness still?
Was it the same, or less then with CFQ?

st0ne · Post by **st0ne** » Tue Jan 16, 2007 1:58 pm

hi,

i think it's no difference than before...
maybe i will try an old kernel and look if it is the same...

krnlbg · Post by **krnlbg** » Sat Jan 20, 2007 4:51 pm

Hi all.
I have same problem. On x86 all works normaly.
My config:
2 drives in nvraid0 on nforce4 sata
jfs filesystem
Deadline I/O scheduler
Linux Kernel v2.6.19-gentoo-r4

segvault · Post by **segvault** » Sat Jan 20, 2007 10:43 pm

Same problem here:

Linux localhost 2.6.19-ck2-r3 #1 SMP PREEMPT Fri Jan 12 01:20:53 EST 2007 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz GenuineIntel GNU/Linux

Core2 Duo overclocked to 3.2Ghz
Gigabyte DS3
2Gb RAM
500Gb SATA (dma enabled)
250Gb PATA (dma enabled)

I also get ridiculously low transfer rates.....
22mb./sec best case scenario btw different hds
15 Mb/sec (idle) between partitions in same hd...
5Mb/s ( bunch programs running)
0-2Mb/sec (while doing emerge --sync)

hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 2810 MB in 2.00 seconds = 1405.24 MB/sec
Timing buffered disk reads: 202 MB in 3.02 seconds = 66.96 MB/sec

in 2.6.17 the cached reads is double +- 2800mb/sec

I believe this is the same bug:

http://bugzilla.kernel.org/show_bug.cgi?id=7372

Unfortunately I can't downgrade to 2.6.17 because my network chipset wont work

Now all we have to do is hunt down this nasty bug introduced in the kernel 2.6.18

joaquin · Post by **joaquin** » Sun Jan 21, 2007 3:10 am

I have the same symptoms: high iowait, 100% cpu load, freeze for a long time, etc.

I have and AMD Turion64X2 TL-50 with ATI chipset, so the problems is not nvidia, intel, etc. I have a PATA disk and windows run better than linux. I am frustrated, because i had an Athlon XP 2000+ with VIA chipset in a DFI mobo, and the performance was excellent, better than several windows with intel chipset.

But this PC is gone, and i upgraded to dual cpu & 64 bits and performance is very poor, and i don't know what i can do for correct the problem. I tested several kernels (vanilla, mm, ck, gentoo) without success, i have changed swap, filesystems, RAM size, drivers, and all is the same.

Please, if somebody find the solution, post it here for all us.

JanR · Post by **JanR** » Sun Jan 21, 2007 4:02 pm

Hello,

on my MD64 system (Athlon 64 X2 4400+, kernel 2.6.16) I had similar problems with poor performance while I worked with very large files. I figured out that (at least for me) the problem is swap management. Starting with 2.6, linux preferes file cache in a way that pages from applications are swaped out to increase cache. If you work with large files (larger than available free memory) this leads to parallel swapping AND accessing the file. Linux 2.4 was much more responsive in such cases not doing that.

I solved the problem by adjusting SWAPPINESS. This parameter defines the behavior of filecache between the values 0 (same as 2.4 - file cache cannot force swapping out anything) to 100 (if more file cache is needed swap out). Default with 2.6 is 60 - I decreased this to 20 and got a much more responsive system especially while handling large files (video cutting... it is stupid to cache files that are larger than physical memory and are read linear).

I'm not sure if this improves the problem here or if this is really a related problem but try to decrease swappiness:

Code: Select all

#check
cat /proc/sys/vm/swappiness

#set it once
echo 20 > /proc/sys/vm/swappiness

# make it permanent
echo "vm.swappiness = 20" >> /etc/sysctl.conf

Hope this helps.

Edit: To see the impact... just create a file twice the size of your memory while watching swap using "top". With default setting you will notice an enormous (useless) swapout while it remains nearly unchanged with swappines = 20 or even less.

Greetings,

Jan

erikm · Post by **erikm** » Thu Jan 25, 2007 3:24 pm

Bornio wrote:
st0ne wrote: hm, i've tried the deadline-scheduler.... but on emerge sync the io-wait is high... i'll test with other programs...
How high was it? Also, despite it being high, did you feel any choppiness still?
Was it the same, or less then with CFQ?

I'm in more or less the same boat as everyone else in here - Core2Duo, SATA, Intel ICH7, amd64 install. I've always used the ck-sources on 32 bit systems before, and been nothing short of blown away by the beautiful multitasking with CFQ and voluntary preemption.

Ck-sources don't seem to work for me, so I switched to the regular gentoo sources, use the Anticipatory scheduler and voluntary preemption, and things have improved greatly, though I still think the can isn't up to its full potential.

My bet is a scheduler bug.

moesasji · Post by **moesasji** » Sat Jan 27, 2007 7:58 am

I noticed that segvault provided a link to a bugreport in the kernel-bugzilla.
Its description and things that have been tried look awfully similar to what I experience and see for a lot of people in this topic.

That bugreport contains a patch by Andrew Morton.
It seems to be a small patch as it only changes the setting of one variable. Has anybody here tested that patch on his system??

ps) Note that the original reporter of that bug switched to another kernel and never tested the patch.

edit) I've tested the patch and rebooted. Problem unfortunately still remains.

Keruskerfuerst · Post by **Keruskerfuerst** » Sat Jan 27, 2007 8:32 pm

I have a AMD Athlon64 3200, nForce 3 250GB and 1GB RAM. Geforce 5900XT graphics card. The NIC is onboard (forcedeth driver)
No SMP, CFQ scheduler and ext3 filesystem with journal writeback, dir_index and filetype.

During emerge -e world, the system has (sometimes) low I/O wait.

P.S.: max I/O wait during emerge --sync: 81,3%.

What about IOMMU on a AMD 64 system?

titusthefox · Post by **titusthefox** » Tue Jan 30, 2007 11:25 am

I have tried vanilla kernel 2.6.19.2 but the problem is still. I have a notebook with nforce3 chipset and Athlon64 mobile CPU.
The kernel has is volountary preentive, and the system respond always but when I access to small files (loading java, emerge --sync) the I/O wait go up until 80%-90%. (I think that system respond because the kernel il preenptive)
I use raiserFs and Ext3.

AMD64 system slow/unresponsive during disk access...

Core2 slow Disk IO with SATA