| View previous topic :: View next topic |
| Author |
Message |
pijalu
Guru
Joined: 04 Oct 2004
Posts: 365
|
 Posted: Sun May 14, 2006 12:59 pm Post subject: [HOWTO] Fun with io priorities aka ionice |
 |
|
Sometimes, when my sweet gentoo do some heavy background tasks on my laptop (eg: big fat emerge, update-db, prelink, etc...), my io usage becomes annoying (no cpu use... just hd is out for lunch)...and this, even if the tasks are niced
==> My desktop starts to lags... opening a console takes way too much time....
If you already experience this, here comes a tip to keep a pleasant desktop: messing with io priorities...
As usual, this is not garanteed, if you break something, get hacked, loose your job or even start to miss windows, don't complain to me  .... This won't make your desktop go faster with IO... just reschedule things to keep a snappy desktop
So, let's start !
Check kernel settings:
* Be sure you have a PREEMT enabled kernel
| Code: |
$ zcat /proc/config.gz | grep PREEMT
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
|
(normal or voluntary should do)
* be sure CFQ iosched is enabled
| Code: |
$ zcat /proc/config.gz | grep IOSCHED
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_IOSCHED="anticipatory"
|
* if CONFIG_IOSCHED_CFQ is set but CONFIG_DEFAULT_IOSCHED is not CFG (like here)
add to /etc/conf.d/local.start:
| Code: |
for drive in /sys/block/*/queue/scheduler
do
echo cfq > $drive
done
|
(restart local)
* install ionice
a) get the source from kernel docs:
| Code: |
sed -n -e "/snip ionice.c/,/snip ionice.c/ p" /usr/src/linux/Documentation/block/ioprio.txt | sed "s/.\+snip ionice.\+//g" > ionice.c
|
If you want all users to have access to ionice, it will need to bet set uid root ==> this mean security hazard..... the following patch should avoid normal user setting higher ioprio than default and, last but not least, avoid ionice to start programs as root.... This is not bullet proof and can lead to security problems... jump to point e and skip f if you plan to use ionice as root only)
b) Patch it => create a file called ionice.patch and copy the following content
| Code: |
--- ionice.c.ori 2006-05-14 13:59:53.000000000 +0200
+++ ionice.c 2006-05-14 13:56:32.000000000 +0200
@@ -1,5 +1,3 @@
-
-
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
@@ -99,14 +97,23 @@
printf("%s: prio %d\n", to_prio[ioprio_class], ioprio);
}
} else {
+ if (getuid()!=0 && (ioprio_class < IOPRIO_CLASS_BE || ioprio < 4) ) {
+ fprintf(stderr,"User cannot set higher ioprio\n");
+ return 1;
+ }
if (ioprio_set(IOPRIO_WHO_PROCESS, pid, ioprio | ioprio_class << IOPRIO_CLASS_SHIFT) == -1) {
perror("ioprio_set");
return 1;
}
- if (argv[optind])
+ if (argv[optind]) {
+ if (seteuid(getuid())!=0 || setegid(getgid())!=0) {
+ perror("Error setting real user");
+ return 1;
+ }
execvp(argv[optind], &argv[optind]);
}
+ }
return 0;
}
|
c) check if patch is fine
| Code: |
patch --dry-run -l < ionice.patch
|
d) if fine, apply
| Code: |
patch -l < ionice.patch
|
e) compile
| Code: |
gcc -o ionice ionice.c
|
f) set rights (only if you patched ionice...):
| Code: |
# chown root:root ionice && chmod u+s,-rw ionice
|
g) copy to /usr/bin (/usr/sbin for a root only version)
| Code: |
# mv ionice /usr/bin
|
* Play with it:
I use ionice on every heavy io cron jobs.... Simply change the cron job scripts (/etc/cron.*) to prefix the calls to program with ionice...( do some backups first... )
Some infos:
* Can be used with nice
* 3 prio:
RT (real time) ==> -c1 (take it no matter what)
BE (BEST EFFORT)==> -c2 (default)
IDLE ==> -c3 (disk will be used only if "free")
* Level: available for RT and BE, use via -nX
Levels allows you a finer tunning of the class (prio between same class) - this is not used for idle...
Default prio are BE level 4
Check /usr/src/linux/Documentation/block/ioprio.txt for more info
Examples
eg: Start an emerge sync as low priorities
| Code: |
# ionice -c3 emerge --sync
|
Not annoying cron slocate (update-db) - /etc/cron.daily/slocate :
| Code: |
#! /bin/sh
if [ -x /usr/bin/updatedb ]
then
if [ -f /etc/updatedb.conf ]
then
/usr/bin/ionice -c3 nice /usr/bin/updatedb
else
/usr/bin/ionice -c3 nice /usr/bin/updatedb -f proc
fi
chown root:locate /var/lib/slocate/slocate.db
fi
|
Have fun.... |
|
| Back to top |
|
 |
johntash
n00b
Joined: 30 Apr 2005
Posts: 20
Location: KS
|
| Posted: Wed May 17, 2006 3:35 am Post subject: |
|
|
Nice.. I think I'll try this on my gentoo desktop later this week. Looks good  |
|
| Back to top |
|
|
JuNix
Apprentice
Joined: 05 Mar 2003
Posts: 224
Location: Sheffield
|
| Posted: Wed May 17, 2006 8:58 am Post subject: |
|
|
| Nice post! |
|
| Back to top |
|
|
JuNix
Apprentice
Joined: 05 Mar 2003
Posts: 224
Location: Sheffield
|
| Posted: Fri May 19, 2006 11:10 am Post subject: |
|
|
Just to follow up on this, this really works, very nicely indeed. So so useful
So I ran a benchmark
Completely idle disks
| Code: | gatekeeper ~ # hdparm -T -t /dev/md2
/dev/md2:
Timing cached reads: 2012 MB in 2.00 seconds = 1004.70 MB/sec
Timing buffered disk reads: 282 MB in 3.02 seconds = 93.47 MB/sec
|
Lots of normally prioritised diskio using chkfrag.pl
| Code: | gatekeeper ~ # hdparm -T -t /dev/md2
/dev/md2:
Timing cached reads: 2188 MB in 2.00 seconds = 1092.19 MB/sec
Timing buffered disk reads: 40 MB in 3.13 seconds = 12.77 MB/sec
|
So you can immediately see that while chkfrag.pl is running, I'm only getting 12.77 MB/s, so I have really degraded disk io bandwidth available, lots of expensive head seeks
and finally, chkfrag.pl is invoked with ionice -c3 ....
| Code: | gatekeeper ~ # hdparm -T -t /dev/md2
/dev/md2:
Timing cached reads: 2732 MB in 2.00 seconds = 1365.29 MB/sec
Timing buffered disk reads: 310 MB in 3.02 seconds = 102.81 MB/sec
|
and for the curious
chkfrag.pl
| Code: | #!/usr/bin/perl -w
#this script search for frag on a fs
use strict;
#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;
#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");
$/ = "\0";
while (defined (my $file = <FILES>)) {
PrivoxyWindowOpen(FRAG, "-|", "filefrag", $file);
my $res = <FRAG>;
if ($res =~ m/.*:\s+(\d+) extents? found/) {
my $fragment = $1;
$fragments += $fragment;
if ($fragment > 1) {
$fragfiles++;
}
$files++;
} else {
print ("$res : not understand for $file.\n");
}
close (FRAG);
}
close (FILES);
print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
|
|
|
| Back to top |
|
|
hans0r
Tux's lil' helper
Joined: 02 Jan 2005
Posts: 122
Location: Germany
|
| Posted: Tue Jun 13, 2006 7:34 pm Post subject: |
|
|
| great howto! works like a charm |
|
| Back to top |
|
|
pussi
l33t
Joined: 08 May 2004
Posts: 727
Location: Finland
|
| Posted: Mon Jul 10, 2006 2:17 pm Post subject: |
|
|
| ionice is also avalibe from portage from sys-apps/util-linux-2.13_pre7, which is currently hard masked though. |
|
| Back to top |
|
|
pijalu
Guru
Joined: 04 Oct 2004
Posts: 365
|
| Posted: Mon Jul 10, 2006 3:24 pm Post subject: |
|
|
| pussi wrote: | | ionice is also avalibe from portage from sys-apps/util-linux-2.13_pre7, which is currently hard masked though. |
Yep, but i didn't suceeded to compile this package on my system => the reason why I choose to take it directly from the kernel doc |
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Thu Aug 10, 2006 12:42 pm Post subject: |
|
|
Isn't there any other scheduler supporting io priorities? The CFQ scheduler produces EXTREME lags on the desktop. For example, when editing within vi, the response can take up to a minute or more.... so that solution is far from optimal, but better than nothing . M;aybe that is due to ext3's incapability to maintain a high total throughput rate in high disk load situations (unlike XFS)?
If you are using fcrontab, you should consider running your jobs via:
| Code: | %hourly * rm -f /var/spool/cron/lastrun/cron.hourly
%daily * 5-23 rm -f /var/spool/cron/lastrun/cron.daily
%weekly * 5-23 rm -f /var/spool/cron/lastrun/cron.weekly
%monthly * * * rm -f /var/spool/cron/lastrun/cron.monthly
@ 10 if /usr/bin/test -x /usr/bin/ionice; then /usr/bin/ionice -c3 /usr/sbin/run-crons; else /usr/sbin/run-crons; fi
|
BTW: the ionice utility is also available via my scripting tools package: https://stier.dynu.com/~myportage/sys-libs/mscript-tools/ |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Thu Sep 14, 2006 2:01 pm Post subject: |
|
|
| ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly. |
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Thu Sep 14, 2006 2:08 pm Post subject: |
|
|
| neuron wrote: | | ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly. |
Uhm, isn't that what -c3 says? To delay any io operations as long as there are higher-prio IOs? 99% iowait tells you that there is IO going on, so that seems perfect to me (as long as there actually IS some non -c3 IO). |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Thu Sep 14, 2006 2:19 pm Post subject: |
|
|
| Cinquero wrote: | | neuron wrote: | | ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly. |
Uhm, isn't that what -c3 says? To delay any io operations as long as there are higher-prio IOs? 99% iowait tells you that there is IO going on, so that seems perfect to me (as long as there actually IS some non -c3 IO). |
yes, but it's a tad bit extreme.
Like in I can copy stuff in about 100kb/sec, while downloading something at 1mb/sec. It should not go anywhere near that slow. And ionice -c2 -p7 gives me 20-30mb/sec instead |
|
| Back to top |
|
|
bludger
Guru
Joined: 09 Apr 2003
Posts: 389
|
| Posted: Sat Sep 16, 2006 6:12 am Post subject: |
|
|
| This looks great. Any idea when it will be unmasked in portage? |
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Sun Sep 17, 2006 9:25 pm Post subject: |
|
|
Install my scripting tools package. It includes the ionice binary. Don't wait for Gentoo. You have no guarantee that they will ever support it.
https://stier.dynu.com/~myportage/sys-libs/mscript-tools/
Question:
does anyone else experience the following problem?
1.) ionice -c3 emerge ... some pkg
2.) rsync / target (in parallel to 1)
3.) some GUI (ie. tvtime) with regular io and cpu priorities hangs as long as rsync builds the filelist... that is really ANNOYING and definitely a bug. |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Mon Sep 18, 2006 12:11 am Post subject: |
|
|
| Cinquero wrote: | | Install my scripting tools package. It includes the ionice binary. Don't wait for Gentoo. You have no guarantee that they will ever support it. |
it's in sys-apps/util-linux-2.13_pre7 |
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Mon Sep 18, 2006 12:15 am Post subject: |
|
|
| neuron wrote: | | it's in sys-apps/util-linux-2.13_pre7 |
Not unmasked. |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Mon Sep 18, 2006 12:23 am Post subject: |
|
|
| Cinquero wrote: | | neuron wrote: | | it's in sys-apps/util-linux-2.13_pre7 |
Not unmasked. |
no but "You have no guarantee that they will ever support it.", that's fairly unlikely |
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Mon Sep 18, 2006 12:29 am Post subject: |
|
|
| neuron wrote: | | no but "You have no guarantee that they will ever support it.", that's fairly unlikely |
Conversation considered useless. |
|
| Back to top |
|
|
fgrosshans
n00b
Joined: 28 Apr 2004
Posts: 11
Location: Munich
|
| Posted: Tue Sep 19, 2006 4:14 pm Post subject: Thanks for the trick... |
|
|
Thanks a lot. It makes emerge-by-working practical on my laptop !
It just lacks the way to (io)renice an already started process. For a started emerge, for exemple, you can find the relevant PID by
| Code: | # ps -A | grep emerge
22066 tty2 00:00:19 emerge
|
and then ...
_________________
Frédéric Grosshans |
|
| Back to top |
|
|
pijalu
Guru
Joined: 04 Oct 2004
Posts: 365
|
| Posted: Tue Sep 19, 2006 6:44 pm Post subject: |
|
|
Something like this script may help
| Code: |
#!/bin/bash
cmd="$@"
if [ -z "$cmd" ]
then
echo "feed me something..."
exit 0
fi
# get all options
params=`echo $@ | sed -e "s/ /\n/g" | sed -n "/^-/p"`
# get rid of extra \n
params=`echo $params`
# set a def val if needed
if [ -z "$params" ]
then
params="-c3"
fi
# for each given prog name
for item in `echo $@ | sed "s/-.*//g" | sed "s/\ /\n/g"`
do
# for each process hitting this
for pid in `ps -A | grep $item | sed "s/[^ 0-9].*//g"`
do
echo "ionicing $item ($pid) with $params"
ionice $params -p$pid
done
done
|
usage: iorenice [partial prog name] [ionice options]
Note: This will take ALL hit to partial prog name (multiple should work) and apply the give class/prio (-c3 by def) |
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Tue Sep 19, 2006 6:59 pm Post subject: |
|
|
Uhm, my mscript-tools script allows to nice a complete process hierarchy. Just give it the parent process id:
mscript-tools NiceAll `pidof emerge`
You just need to replace nice with ionice in it. |
|
| Back to top |
|
|
MaratIK1
n00b
Joined: 21 Sep 2006
Posts: 4
|
|
| Back to top |
|
|
Cinquero
Apprentice
Joined: 24 Jun 2004
Posts: 249
|
| Posted: Sat Sep 30, 2006 6:50 pm Post subject: |
|
|
| Summary of my experience with the CFQ scheduler: H-O-R-R-I-B-L-E. Just plain useless. When I do not use io prios, a simple "rm -rf /var/tmp/ccache" may lock the entire desktop until it has finished.... even NFS transfers have stalled completely. I'm switching back to AS. |
|
| Back to top |
|
|
bludger
Guru
Joined: 09 Apr 2003
Posts: 389
|
| Posted: Mon Oct 02, 2006 9:58 am Post subject: Re: ebuild |
|
|
According to this link, ionice is included in schedutils 1.5.0, which appears to be already in portage. Has anyone tried this out?
Last edited by bludger on Mon Oct 02, 2006 10:10 am; edited 1 time in total |
|
| Back to top |
|
|
bludger
Guru
Joined: 09 Apr 2003
Posts: 389
|
| Posted: Mon Oct 02, 2006 10:09 am Post subject: |
|
|
| Cinquero wrote: | | Summary of my experience with the CFQ scheduler: H-O-R-R-I-B-L-E. Just plain useless. When I do not use io prios, a simple "rm -rf /var/tmp/ccache" may lock the entire desktop until it has finished.... even NFS transfers have stalled completely. I'm switching back to AS. |
I haven't tried it out myself, so I can't really judge, but it seems contrary to the theory of CFQ as far as I understand it. According to this link, io should be distributed fairly between all processes. Can someone else confirm this phenomenon? I find it strange that Redhat have made this the default scheduler, if it really is so bad. Can you confirm that everything is running with IOPRIO_CLASS_BE (Best effort) and that nothing is running with IOPRIO_CLASS_RT (Real time)?
Maybe I'll have to try it out for myself. BTW if I reboot with the boot flag "elevator=cfq" will that select cfq as default? I would rather not recompile my kernel just for this test. |
|
| Back to top |
|
|
bludger
Guru
Joined: 09 Apr 2003
Posts: 389
|
| Posted: Mon Oct 02, 2006 11:12 am Post subject: |
|
|
| bludger wrote: | | Maybe I'll have to try it out for myself. BTW if I reboot with the boot flag "elevator=cfq" will that select cfq as default? I would rather not recompile my kernel just for this test. |
Just found this: /usr/src/linux/Documentation/block/switching-sched.txt. It is possible to change the scheduler on the fly with:
echo cfq > /sys/block/<device>/queue/scheduler |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
