AMD64 system slow/unresponsive during disk access...

Message

ArneBab · Post by **ArneBab** » Fri May 01, 2009 11:09 am

neuron wrote:
nero37 wrote:Installed RC4 to see, and it's actually better. It's the first change I've done that's felt like more than a small tweak to a bad situation. I went from 2.6.29, and I use ext4 not ext3 so the writeback change doesn't affect my results. I dont have any numbers but it sure feels a lot better.
That sounds great!

As soon as 2.6.30 is in portage, I'll give it a try.

I currently have a test which shows the problem amost instantly: Firing off and exchanging commits in Mercurial automatically at a rate of 2 per second or so (stresstesting ).

graysky · Post by **graysky** » Fri May 01, 2009 6:32 pm

nero37 wrote:Installed RC4 to see, and it's actually better. It's the first change I've done that's felt like more than a small tweak to a bad situation. I went from 2.6.29, and I use ext4 not ext3 so the writeback change doesn't affect my results. I dont have any numbers but it sure feels a lot better.

That's good to hear... can you try the copying a 2 or 3 gig file from one partition to another experiment while using firefox or whatever? Move the window while the copy is in progress... does it get choppy?

Lepaca Kliffoth · Post by **Lepaca Kliffoth** » Sun May 03, 2009 9:18 am

The issue is FIXED in 2.6.30-rc4. The IO scheduler you choose doesn't matter.

If the Nvidia drivers won't compile with this kernel version, you can modify the ebuild so that it applies this patch I made:

http://pastebin.com/m63cf9998

I'm not an expert in this stuff but it works.

Happy testing and good luck.

DigitalCorpus · Post by **DigitalCorpus** » Mon May 04, 2009 8:57 pm

Can you provide more documentation as to what is the change in that RC of 2.6.30 that fixed this problem that was existent for so long?

neuron · Post by **neuron** » Tue May 05, 2009 5:43 am

I've seen reports that it's not fixed as well, but it's a huge improvement in my case atleast. And I dont know the exact patches sorry, I already read a information thing from linus complaining about it, and some problems in the io scheduler.

kernelOfTruth · Post by **kernelOfTruth** » Tue May 05, 2009 7:34 am

neuron wrote:I've seen reports that it's not fixed as well, but it's a huge improvement in my case atleast.

++

It has improved noticably for me, too but it's still not fixed

H0bb3z · Post by **H0bb3z** » Fri May 08, 2009 3:06 am

Just trying to look into this myself now as well and found this thread.

For 2.6.30-rc4, are you using gentoo-source or vanilla? I'm at 2.6.28-r5 gentoo sources. I'll flip over to vanilla if it drastically improves this issue.

I'm running Intel QX9770 (Quad Extreme), Core 2 optimizations (Funtoo build), DDR3 1600MHz mem w/Intel P45/ICH10R chipsets.

This thing simply flies even with all the heavy compiz whiz-bang extras running -- except when I'm transferring large chunks of data. I'm running striped 1Tb drives w/LVM for larger volumes (ext3). I also notice that the MS wireless mouse gets really sluggish during heavy disk access and the click function "misses" occasionally - requiring me to click a couple times for proper response. That's annoying...

I don't have tangible statistics, but a system with this level of hardware should show much better drive performance...

Lepaca Kliffoth · Post by **Lepaca Kliffoth** » Sat May 09, 2009 9:33 am

Coming back to report that the issue is not fixed completely after all; however, the awful mouse stuttering I used to get all the time appeared only once when I had to copy ~30GB of data from my linux main partition to a windows one (mounted with ntfs-3g). Since I switched to vanilla 2.6.30-rc4 I haven't noticed the problem in any other situation.

teapot · Post by **teapot** » Wed May 13, 2009 11:28 pm

the problem is not completely fixed for me either. whenever a big torrent download is finished , the hashing procedure sends latencies through the roof , even if i ionice the rtorrent process down. it used to be a lot worse , but the problem is very much still there.

is it just me , or do you also find it strange that a serious issue like this can exist for such a long time? ( 3 years or something like that )

is it because this problem just exist on consumer level hardware and isn't considered to be a problem on servers?

the problem has been documented for a long time , but has more or less been ignored by the kernel developers until recently.
good that something is finally happening , but i get the feeling that the latencies are caused by a whole bunch of errors.
bringing down a few of them will sure make things better , but i am not convinced that the real reason behind all this is found yet...

darklegion · Post by **darklegion** » Thu May 14, 2009 4:46 am

teapot wrote:the problem is not completely fixed for me either. whenever a big torrent download is finished , the hashing procedure sends latencies through the roof , even if i ionice the rtorrent process down. it used to be a lot worse , but the problem is very much still there.

is it just me , or do you also find it strange that a serious issue like this can exist for such a long time? ( 3 years or something like that )

is it because this problem just exist on consumer level hardware and isn't considered to be a problem on servers?

the problem has been documented for a long time , but has more or less been ignored by the kernel developers until recently.
good that something is finally happening , but i get the feeling that the latencies are caused by a whole bunch of errors.
bringing down a few of them will sure make things better , but i am not convinced that the real reason behind all this is found yet...

It's probably just one of those phantom bugs that's difficult to track down.And it doesn't occur on all configurations, I haven't had the problem on my system since about 2.6.24, and it only occured on one of my systems and not the other.Sure, I get a bit of latency when copying files around, but it's not any worse than a standard windows installation.

Elv13 · Post by **Elv13** » Thu May 21, 2009 5:35 am

Just updated to 2.6.30-rc5. I confirm that the problem is gone for me. I can unzip a kernel while watching a HD video, something that was impossible with earlier version. (with or without VDPAU or whatever the new nvidia thing is called)

albright · Post by **albright** » Thu May 21, 2009 12:00 pm

it's not any worse than a standard windows installation.

so that's the level of extreme excellence we're shooting for eh? Maybe
linux will someday attain such heights.

E001754 · Post by **E001754** » Thu May 28, 2009 1:38 pm

Yamakuzure wrote:Just a note on this issue with unresponsiveness:

I found, that setting up a preemptible kernel with 1000Hz clock is a very very bad idea!

Since I configured all my amd64 machines to build a kernel with "voluntary preemption" and 300Hz clock, performance went incredibly up everywhere.
Before I did this, whenever I emerged world updates, and locked the screen, I needed to move the mouse like mad and hammer on the keyboard for 20 - 30 seconds before the login window came up. Now I press "ctrl" and the login window is there in 1 - 2 seconds. Quite a difference, right?

This applies for a HP laptop with intel core2 cpu and 1GB RAM as well as my new AMD PhenomII 940/4GB RAM machine. Both benefit from the clock and preemption change, so maybe if you have issues with responsiveness, you might want to try that?
Code: Select all
Processor type and features  --->
    Preemption Model --->
        ( ) No Forced Preemption (Server)
        (X) Voluntary Kernel Preemption (Desktop)
        ( ) Preemptible Kernel (Low-Latency Desktop)
    Timer frequency --->
        ( ) 100 HZ
        ( ) 250 HZ
        (X) 300 HZ
        ( ) 1000 HZ

1000 Hz was a setting needed by some ID Software games if I remember well.
Did you try playing ET:QW or Quake 4 with one of your computer since you changed this preemption model and that timer frequency ??

Captain Newbie · Post by **Captain Newbie** » Sat May 30, 2009 4:32 am

Lepaca Kliffoth wrote:The issue is FIXED in 2.6.30-rc4. The IO scheduler you choose doesn't matter.

etc

*** IF YOU ARE REPORTING THIS FIXED ***

Please tell us your kernel and hardware configuration.

I can report things as 'subjectively better' in 2.6.30-rc7-git3 (sys-kernel/git-sources). IOWait still loads up to high levels (84%) and processes report as diskbound in top (status D), but firefox keeps chugging happily away as does VLC.

Hardware:

Intel Core 2 Quad Q9300 on Intel P35 Express
8GB DDR2-800
Seagate 1TB Serial ATA hard drive

DigitalCorpus · Post by **DigitalCorpus** » Sat May 30, 2009 8:18 am

That sounds like great news. I'll move over to 2.6.30 when it becomes stable.

Out of curiosity, where can we find details to exactly what the problem was?

Elv13 · Post by **Elv13** » Mon Jun 01, 2009 12:28 am

I got it again after all, but it is much, much better.

Kernel 2.6.30-rc5, nvidia nforce 680i, Intel C2Q Q6600, EVGA motherboard

poly_poly-man · Post by **poly_poly-man** » Mon Jun 01, 2009 12:32 am

last week, on 2.6.28, had a nearly unusable system with nothing more than a torrent in the background. Now, on .30_rc7, bittorrent seems to screw with my network card a little, but that, installing ubuntu in a vm, etc. are all fine.

forsaken1 · Post by **forsaken1** » Mon Jun 01, 2009 12:02 pm

DigitalCorpus wrote:That sounds like great news. I'll move over to 2.6.30 when it becomes stable.

Out of curiosity, where can we find details to exactly what the problem was?

http://lwn.net/Articles/328363/

It's focused on ext3, but alot of the fixes affects pretty much all fs.

Elv13 · Post by **Elv13** » Fri Jun 05, 2009 4:44 am

Interesting, anybody know how to change the size from 512k to 64k like expressed in the article, I might give it a try if it make things even better than they are now.

fangorn · Post by **fangorn** » Fri Jun 05, 2009 6:52 am

poly_poly-man wrote:last week, on 2.6.28, had a nearly unusable system with nothing more than a torrent in the background. Now, on .30_rc7, bittorrent seems to screw with my network card a little, but that, installing ubuntu in a vm, etc. are all fine.

I am doing video conversion a lot. So I am moving hundreds of GB from one disk to the another on a daily basis.

I am using 2.6.28 atm and had some minor advancements changing I/O schedulers, assuming that only one disk I/O stream is working. Whenever two or more disk I/O streams interfere the system is totally unusable. It takes _minutes_ to refresh the display of an already running firefox after a desktop switch. Also I/O slows down to a crawl (less than 10% of normal data rate) instead of approx. 50 % per stream that I would have expected. It is a little better if one of the disks involved is a share mounted over NFS, but not much.

I have two machines that I would like to upgrade to 2.6.30. How is the support for this version in NVIDIA and FGLRX drivers?

kernelOfTruth · Post by **kernelOfTruth** » Fri Jun 05, 2009 8:39 am

nvidia-support for 2.6.30 is great, fglrx is (as usual) non-existant

fangorn · Post by **fangorn** » Sat Jun 06, 2009 4:47 am

OK, tried vanilla-sources-2.6.30-rc8 on my cruncher.

System is more responsive, but whenever I copy data the workload (two video encodings in parallel) on the cpu (Core i7 920) drops from 100 % on all 8 (virtual) cores to approx. 30 %.

Increased responsiveness could also result from the change from preemptile to voluntary preemptile and from 1000 Hz to 250 Hz.

So, I cannot confirm that 2.6.30 has changed too much.

Here is my .config http://nopaste.info/98d98c1e5d.html

Edit: for me anticipatory i/o scheduler made better results than cfq. That's why it is default in my config.

DigitalCorpus · Post by **DigitalCorpus** » Tue Jun 09, 2009 9:20 am

fangorn, which filesystems are you using and what mount options?

fangorn · Post by **fangorn** » Tue Jun 09, 2009 11:55 am

using xfs for video storage (internal, on USB or eSATA external harddiscs), ext3 or reiserfs (v3) for system storages, nfs for network storages (with xfs on the server machine).

Edit: Using mainly default(,noatime) options IIRC, but will check this evening.

lagalopex · Post by **lagalopex** » Tue Jun 09, 2009 1:35 pm

They perhaps found something: http://bugzilla.kernel.org/show_bug.cgi?id=12309#c360

AMD64 system slow/unresponsive during disk access...

vanilla or gentoo kernel sources