Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
AMD64 system slow/unresponsive during disk access (Part 2)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo on AMD64
View previous topic :: View next topic  
Author Message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Mon Sep 13, 2010 6:27 pm    Post subject: Reply with quote

Awesome! 2.6.36 is on my radar now.
Back to top
View user's profile Send private message
darklegion
Guru
Guru


Joined: 14 Nov 2004
Posts: 467

PostPosted: Tue Sep 14, 2010 10:43 am    Post subject: Reply with quote

Okay, I've set up 2.6.36-rc4, and it appears that the bug is completely gone. I'm currently copying 200GBs worth of files, many of which are 7GB or more, and there has not been a single pause. Tab switching and scrolling are very smooth. Hopefully I haven't spoken too soon again, but it really feels like it is fixed this time, at least on my system. I'm using CFQ, BTW.

Also, CFS appears to have improved quite a bit recently (last time I tried was with 2.6.33), to the point that I don't think I need BFS anymore. No more long pauses in Oblivion.

One issue I had was with mounting cifs shares, which resulted in a kernel null pointer dereference, although I managed to fix it by starting a new kernel .config from scratch.
Back to top
View user's profile Send private message
d-fens
Tux's lil' helper
Tux's lil' helper


Joined: 09 Jan 2004
Posts: 88

PostPosted: Wed Sep 15, 2010 9:51 am    Post subject: Reply with quote

seems to be good news :)

i'm just a bit puzzled - whats the recommended kernel to test all the goodies and is it in portage or an overlay?
Back to top
View user's profile Send private message
darklegion
Guru
Guru


Joined: 14 Nov 2004
Posts: 467

PostPosted: Wed Sep 15, 2010 11:01 am    Post subject: Reply with quote

d-fens wrote:
seems to be good news :)

i'm just a bit puzzled - whats the recommended kernel to test all the goodies and is it in portage or an overlay?


I'm just using vanilla 2.6.36-rc4 (from www.kernel.org) and installing it manually.

CFS isn't quite as impressive as I thought, though. I'm getting the occasional audio skip and somewhat jerky scrolling when under high cpu load. Has anyone got BFS working on 2.6.36-rc4 yet? I tried it out, but got failed chunks.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5709
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Sep 17, 2010 12:55 pm    Post subject: Reply with quote

for even less latencies - try out the following patchset:

[PATCH 0/8] Reduce latencies and improve overall reclaim efficiency v2
(the thread-view got screwed up but I'm confident that you'll find the patches)

alternate view (from marc): http://marc.info/?t=128455382700010&r=1&w=2

I patched my kernel with several improvements including the mentioned patchset and using most of Con Koliva's patches from his ck-patchset

so my system is responsive in 99,9% of situations :D
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
JuNix
Apprentice
Apprentice


Joined: 05 Mar 2003
Posts: 204
Location: Sheffield

PostPosted: Fri Sep 17, 2010 3:09 pm    Post subject: Reply with quote

I was having terrible problems on my system, mouse pointer freezing, system being unresponsive during heavy I/O, until I found what was wrong.

Months ago, powertop recommended I change /proc/sys/vm/dirty_writeback_centisecs to 1500.

I reverted it back to default today and all my latency problems are solved.
Certainly with xfs (logbufs=8) it is not a good idea to increase this parameter.

I noticed this because performing a bonnie++ benchmark on a freshly booted system caused heavy swapping, as the cache grew, but pending writes were delayed by this parameter, causing memory pressure. Now with dirty_writeback_centisecs back to the default 500, this no longer occurs.

This fixes the responsiveness problems I've been having for months.

This might help someone out there(?)
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5709
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Sep 17, 2010 3:39 pm    Post subject: Reply with quote

JuNix wrote:
I was having terrible problems on my system, mouse pointer freezing, system being unresponsive during heavy I/O, until I found what was wrong.

Months ago, powertop recommended I change /proc/sys/vm/dirty_writeback_centisecs to 1500.

I reverted it back to default today and all my latency problems are solved.
Certainly with xfs (logbufs=8) it is not a good idea to increase this parameter.

I noticed this because performing a bonnie++ benchmark on a freshly booted system caused heavy swapping, as the cache grew, but pending writes were delayed by this parameter, causing memory pressure. Now with dirty_writeback_centisecs back to the default 500, this no longer occurs.

This fixes the responsiveness problems I've been having for months.

This might help someone out there(?)


try the following when your system has to do some swapping:
Quote:
echo "10" > /proc/sys/vm/page-cluster
# default: 3
# for swapping (?) and page-faults, higher value might speed up linear readings significantly (!) - something like readahead
# recommended to be set to 0 for desktops or systems without swap


I have experienced best results with 10 when doing the counterintuitive approach (instead of 0)

that way it swaps way faster and the system almost doesn't slow down during swapping :)
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
d-fens
Tux's lil' helper
Tux's lil' helper


Joined: 09 Jan 2004
Posts: 88

PostPosted: Sun Sep 19, 2010 7:37 pm    Post subject: Reply with quote

just installed vanilla 2.6.36-rc4 and no additional patches: eix-sync, emerge hugin, dd-ing 5Gigs at the same time and no stuttering!
nice!
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Tue Nov 16, 2010 5:08 pm    Post subject: Reply with quote

Did anyone try the latest Mike patch for the scheduler? Its like 200 lines of patch and supposedly biggest desktop interactivity change since....whatever.

http://marc.info/?l=linux-kernel&m=128978361700898&w=2
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3739

PostPosted: Tue Nov 16, 2010 6:03 pm    Post subject: Reply with quote

Quote:
Did anyone try the latest Mike patch for the scheduler?


yes and it made quite a lot of a difference in my eeepc, firefox for example feels much more useable now if i have sth. compiling in background.
Not sure if it will make such a big difference in a powerfull box though.

And it even looks like Ingo is willing to push this in the next merge request, cool.

cheers
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Tue Nov 16, 2010 6:14 pm    Post subject: Reply with quote

gringo wrote:
Quote:
Did anyone try the latest Mike patch for the scheduler?


yes and it made quite a lot of a difference in my eeepc, firefox for example feels much more useable now if i have sth. compiling in background.
Not sure if it will make such a big difference in a powerfull box though.

And it even looks like Ingo is willing to push this in the next merge request, cool.

cheers
Did the patch apply cleanly against 2.6.36? If I read the emails correctly, its scheduled for inclusion in 2.6.38 at this time.
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3739

PostPosted: Tue Nov 16, 2010 6:39 pm    Post subject: Reply with quote

Quote:
Did the patch apply cleanly against 2.6.36?


dont know, i tried with current git, which is 2.6.37-rc2-something.

Quote:
If I read the emails correctly, its scheduled for inclusion in 2.6.38 at this time.


yeah, merge window for 2.6.37 is closed already, we probably will have to wait for 2.6.38 and see if Trovalds is willing to definitely accept this.

cheers
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Tue Nov 16, 2010 7:51 pm    Post subject: Reply with quote

the patch requires the use of cgroups, which carries a penalty of around 5% in the throughput. But that seems small enough to try it.
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3739

PostPosted: Tue Nov 16, 2010 8:21 pm    Post subject: Reply with quote

Quote:
the patch requires the use of cgroups, which carries a penalty of around 5% in the throughput


dont know about the real impact of using cgroups, i have it enabled already because im also playing around with systemd.

cheers
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5709
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Tue Nov 16, 2010 10:33 pm    Post subject: Reply with quote

the following patchset should reduce i/o wait / or latency during heavy i/o and memory operations *A LOT*:

http://marc.info/?l=linux-kernel&m=129001098801700&w=2

also:

http://lkml.org/lkml/2010/10/28/405

and adjust

+int inactive_file_ratio = 50;

to

+int inactive_file_ratio = 20;

which should lower page faults
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Fri Nov 19, 2010 5:01 am    Post subject: Reply with quote

Have a look at this one: http://www.webupd8.org/2010/11/alternative-to-200-lines-kernel-patch.html

Interesting! Just do the stuff in user space and get the same benefits as the kernel patch!
Back to top
View user's profile Send private message
tallica
Apprentice
Apprentice


Joined: 27 Jul 2007
Posts: 152
Location: Lublin, POL

PostPosted: Fri Nov 19, 2010 8:28 am    Post subject: Reply with quote

devsk wrote:
Have a look at this one: http://www.webupd8.org/2010/11/alternative-to-200-lines-kernel-patch.html

Interesting! Just do the stuff in user space and get the same benefits as the kernel patch!


hmm I don't have /sys/fs/cgroup/cpu

cgroup is enabled:
Code:
grep CGROUP /usr/src/linux/.config
CONFIG_CGROUPS=y
CONFIG_CGROUP_DEBUG=y
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
# CONFIG_CGROUP_MEM_RES_CTLR is not set
CONFIG_CGROUP_SCHED=y
CONFIG_BLK_CGROUP=m
# CONFIG_DEBUG_BLK_CGROUP is not set


Any ideas?
_________________
Gentoo ~AMD64 | Audacious
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1394
Location: Bardowick, Germany

PostPosted: Fri Nov 19, 2010 12:05 pm    Post subject: Reply with quote

1st idea: You forgot to install the kernel/reboot ;) (okay, only kidding!)
2nd idea: maybe its CONFIG_CPUSETS ?
_________________
Boycott systemd!
Back to top
View user's profile Send private message
tallica
Apprentice
Apprentice


Joined: 27 Jul 2007
Posts: 152
Location: Lublin, POL

PostPosted: Fri Nov 19, 2010 12:11 pm    Post subject: Reply with quote

Yamakuzure wrote:
1st idea: You forgot to install the kernel/reboot ;) (okay, only kidding!)
2nd idea: maybe its CONFIG_CPUSETS ?

Code:
zgrep CPUSETS /proc/config.gz
CONFIG_CPUSETS=y


so it's not that

EDIT:
Anyway I'm going to try that magic 200 lines patch.
_________________
Gentoo ~AMD64 | Audacious
Back to top
View user's profile Send private message
isolationism
Tux's lil' helper
Tux's lil' helper


Joined: 01 Nov 2004
Posts: 127

PostPosted: Fri Nov 19, 2010 3:58 pm    Post subject: Reply with quote

Michal. wrote:
hmm I don't have /sys/fs/cgroup/cpu


Neither do I. :( Wish I did 'cause I'm not willing to switch kernels right now just to apply the patch (am running 2.6.36) and have massively parallel processors (2x6 core Xeons with hyperthreading = 24 virtual processors) -- hiccup problem is REALLY bad even when doing relatively mundane tasks like starting/stopping virtual machines (e.g. mouse pointer goes dead for ~1 sec).

Anyone know why the /sys/fs/cgroup/cpu node might be missing?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Fri Nov 19, 2010 4:31 pm    Post subject: Reply with quote

I don't have cgroups at all. So, I can't test it, but can u see if you have /dev/cgroups?
Back to top
View user's profile Send private message
isolationism
Tux's lil' helper
Tux's lil' helper


Joined: 01 Nov 2004
Posts: 127

PostPosted: Fri Nov 19, 2010 4:45 pm    Post subject: Reply with quote

I ended up following the "Ubuntu" method where you create the device node in /etc/conf.d/local.start and all seems to be working fine -- no kernel patches involved. I'm definitely seeing lower latency on the "big" IO tasks like when starting up a virtual machine, but I'm also seeing lots of little lags even as I type this where every 10-20 seconds it seems like a letter takes an extra 250ms or something to show up. It's kind of weird.

Anyway the nice thing with the quick-hit patch is that I can just edit ~/.bashrc and effectively disable the thing without having to dick around (any more) in the kernel. Wonder if the guys who have recommended this have tested it out on something with as many virtual processors as I have in this machine; maybe the performance penalty for the patch becomes increasingly expensive the more processors you add.

Poor responsiveness while compiling has never has never been a problem for me on this machine; I do everything with make -j26 and you'd never know it's compiling -- yet it builds a kernel in about 45 seconds.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5709
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sat Nov 20, 2010 11:42 pm    Post subject: Reply with quote

and yet another approach:


[PATCH v4] sched: automated per session task groups:

http://marc.info/?l=linux-kernel&m=129028174609578&w=2
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Sun Nov 21, 2010 12:00 am    Post subject: Reply with quote

kernelOfTruth wrote:
and yet another approach:


[PATCH v4] sched: automated per session task groups:

http://marc.info/?l=linux-kernel&m=129028174609578&w=2
which is same as http://forums.gentoo.org/viewtopic-p-6490539.html?sid=eec90b9e66f9f07c36645c3656ead483#6490539 but with the patch modified in a couple of lines for clean up requested by Ingo.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Mon Nov 22, 2010 8:24 am    Post subject: Reply with quote

OK, version V4 of the kernel patch does something really different. It makes groupings based on session (based on processe group leaders, created by system call setsid) and not TTY. So, this does address the issue people raised and wanted to use systemd for it. Well, guess what! Not needed. The in-kernel solution works better and smoother without any user intervention. What took this thing so long!

The autogroup creates 58 unique groups on my machine. See below:

Code:
# cat /proc/`pidof firefox`/autogroup
/autogroup-46 nice 0
# for i in /proc/*/autogroup;do if [[ "`cat $i`" == "/autogroup-46 nice 0" ]] ;then cat `dirname $i`/comm;fi;done
kdeinit4
klauncher
kded4
kglobalaccel
ksmserver
kwin
knotify4
plasma-desktop
krunner
konsole
knotes
kgpg
polkit-kde-auth
kmix
pidgin
conky
gnome-power-man
set_dpms
VBoxXPCOMIPCD
VirtualBox
firefox
plugin-containe
plugin-containe
GoogleTalkPlugi
So, the whole desktop runs in one group. And yes, the programs started by shells do get their own groups. So, nothing kills anything. I just did the make -j64 test and resumed a VM while the loadavg was 63. VM resumed just like normal and XP running inside it had no lag. Repeat zero lag! Its not letting a group of programs run away with the CPU.

So, the thing works! And works well! Great job Mike!

PS: It doesn't address the IO related hangs but it does reduce that problem to a bearable state (by slowing down the IO hogger, which has to go back to CPU when one of its IO returns or when it has to hunt free pages).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo on AMD64 All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
Page 4 of 7

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum