[HOWTO] Fun with io priorities aka ionice

Message

pijalu · Post by **pijalu** » Sun May 14, 2006 12:59 pm

Sometimes, when my sweet gentoo do some heavy background tasks on my laptop (eg: big fat emerge, update-db, prelink, etc...), my io usage becomes annoying (no cpu use... just hd is out for lunch)...and this, even if the tasks are niced
==> My desktop starts to lags... opening a console takes way too much time....

If you already experience this, here comes a tip to keep a pleasant desktop: messing with io priorities...
As usual, this is not garanteed, if you break something, get hacked, loose your job or even start to miss windows, don't complain to me .... This won't make your desktop go faster with IO... just reschedule things to keep a snappy desktop

So, let's start !
Check kernel settings:
* Be sure you have a PREEMT enabled kernel

Code: Select all

$ zcat /proc/config.gz | grep PREEMT
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set

(normal or voluntary should do)
* be sure CFQ iosched is enabled

Code: Select all

$ zcat /proc/config.gz | grep IOSCHED
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_IOSCHED="anticipatory"

* if CONFIG_IOSCHED_CFQ is set but CONFIG_DEFAULT_IOSCHED is not CFG (like here)
add to /etc/conf.d/local.start:

Code: Select all

	for drive in  /sys/block/*/queue/scheduler 
	do
		echo cfq > $drive
	done

(restart local)

* install ionice
a) get the source from kernel docs:

Code: Select all

sed -n -e "/snip ionice.c/,/snip ionice.c/ p" /usr/src/linux/Documentation/block/ioprio.txt | sed "s/.\+snip ionice.\+//g" > ionice.c

If you want all users to have access to ionice, it will need to bet set uid root ==> this mean security hazard..... the following patch should avoid normal user setting higher ioprio than default and, last but not least, avoid ionice to start programs as root.... This is not bullet proof and can lead to security problems... jump to point e and skip f if you plan to use ionice as root only)
b) Patch it => create a file called ionice.patch and copy the following content

Code: Select all

--- ionice.c.ori	2006-05-14 13:59:53.000000000 +0200
+++ ionice.c	2006-05-14 13:56:32.000000000 +0200
@@ -1,5 +1,3 @@
-
-
 #include <stdio.h>
 #include <stdlib.h>
 #include <errno.h>
@@ -99,14 +97,23 @@
 			printf("%s: prio %d\n", to_prio[ioprio_class], ioprio);
 		}
 	} else {
+		if (getuid()!=0 && (ioprio_class < IOPRIO_CLASS_BE || ioprio < 4)  ) {
+			fprintf(stderr,"User cannot set higher ioprio\n");
+			return 1;
+		}
 		if (ioprio_set(IOPRIO_WHO_PROCESS, pid, ioprio | ioprio_class << IOPRIO_CLASS_SHIFT) == -1) {
 			perror("ioprio_set");
 			return 1;
 		}
 
-		if (argv[optind])
+		if (argv[optind]) {
+		    if (seteuid(getuid())!=0 || setegid(getgid())!=0) {
+			perror("Error setting real user");
+			return 1;
+		    }
 			execvp(argv[optind], &argv[optind]);
 	}
+	}
 
 	return 0;
 }

c) check if patch is fine

Code: Select all

patch --dry-run -l < ionice.patch

d) if fine, apply

Code: Select all

patch -l < ionice.patch

e) compile

Code: Select all

gcc -o ionice ionice.c

f) set rights (only if you patched ionice...):

Code: Select all

# chown root:root ionice && chmod u+s,-rw ionice

g) copy to /usr/bin (/usr/sbin for a root only version)

Code: Select all

# mv ionice /usr/bin

* Play with it:
I use ionice on every heavy io cron jobs.... Simply change the cron job scripts (/etc/cron.*) to prefix the calls to program with ionice...( do some backups first... )

Some infos:
* Can be used with nice
* 3 prio:
RT (real time) ==> -c1 (take it no matter what)
BE (BEST EFFORT)==> -c2 (default)
IDLE ==> -c3 (disk will be used only if "free")

* Level: available for RT and BE, use via -nX
Levels allows you a finer tunning of the class (prio between same class) - this is not used for idle...
Default prio are BE level 4

Check /usr/src/linux/Documentation/block/ioprio.txt for more info

Examples
eg: Start an emerge sync as low priorities

Code: Select all

# ionice -c3 emerge --sync

Not annoying cron slocate (update-db) - /etc/cron.daily/slocate :

Code: Select all

#! /bin/sh

if [ -x /usr/bin/updatedb ]
then
	if [ -f /etc/updatedb.conf ]
	then
		/usr/bin/ionice -c3 nice /usr/bin/updatedb
	else
		/usr/bin/ionice -c3 nice /usr/bin/updatedb -f proc
	fi
	chown root:locate /var/lib/slocate/slocate.db
fi

Have fun....

johntash · Post by **johntash** » Wed May 17, 2006 3:35 am

Nice.. I think I'll try this on my gentoo desktop later this week. Looks good

JuNix · Post by **JuNix** » Wed May 17, 2006 8:58 am

Nice post!

JuNix · Post by **JuNix** » Fri May 19, 2006 11:10 am

Just to follow up on this, this really works, very nicely indeed. So so useful

So I ran a benchmark

Completely idle disks

Code: Select all

gatekeeper ~ # hdparm -T -t /dev/md2

/dev/md2:
 Timing cached reads:   2012 MB in  2.00 seconds = 1004.70 MB/sec
 Timing buffered disk reads:  282 MB in  3.02 seconds =  93.47 MB/sec

Lots of normally prioritised diskio using chkfrag.pl

Code: Select all

gatekeeper ~ # hdparm -T -t /dev/md2

/dev/md2:
 Timing cached reads:   2188 MB in  2.00 seconds = 1092.19 MB/sec
 Timing buffered disk reads:   40 MB in  3.13 seconds =  12.77 MB/sec

So you can immediately see that while chkfrag.pl is running, I'm only getting 12.77 MB/s, so I have really degraded disk io bandwidth available, lots of expensive head seeks

and finally, chkfrag.pl is invoked with ionice -c3 ....

Code: Select all

gatekeeper ~ # hdparm -T -t /dev/md2

/dev/md2:
 Timing cached reads:   2732 MB in  2.00 seconds = 1365.29 MB/sec
 Timing buffered disk reads:  310 MB in  3.02 seconds = 102.81 MB/sec

and for the curious

chkfrag.pl

Code: Select all

#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");

$/ = "\0";

while (defined (my $file = <FILES>)) {
        PrivoxyWindowOpen(FRAG, "-|", "filefrag", $file);
        my $res = <FRAG>;
        if ($res =~ m/.*:\s+(\d+) extents? found/) {
                my $fragment = $1;
                $fragments += $fragment;
                if ($fragment > 1) {
                        $fragfiles++;
                }
                $files++;
        } else {
                print ("$res : not understand for $file.\n");
        }
        close (FRAG);
}
close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");

hans0r · Post by **hans0r** » Tue Jun 13, 2006 7:34 pm

great howto! works like a charm

pussi · Post by **pussi** » Mon Jul 10, 2006 2:17 pm

ionice is also avalibe from portage from sys-apps/util-linux-2.13_pre7, which is currently hard masked though.

pijalu · Post by **pijalu** » Mon Jul 10, 2006 3:24 pm

pussi wrote:ionice is also avalibe from portage from sys-apps/util-linux-2.13_pre7, which is currently hard masked though.

Yep, but i didn't suceeded to compile this package on my system => the reason why I choose to take it directly from the kernel doc

Cinquero · Post by **Cinquero** » Thu Aug 10, 2006 12:42 pm

Isn't there any other scheduler supporting io priorities? The CFQ scheduler produces EXTREME lags on the desktop. For example, when editing within vi, the response can take up to a minute or more.... so that solution is far from optimal, but better than nothing

. M;aybe that is due to ext3's incapability to maintain a high total throughput rate in high disk load situations (unlike XFS)?

If you are using fcrontab, you should consider running your jobs via:

Code: Select all

%hourly *        rm -f /var/spool/cron/lastrun/cron.hourly
%daily * 5-23       rm -f /var/spool/cron/lastrun/cron.daily
%weekly * 5-23      rm -f /var/spool/cron/lastrun/cron.weekly
%monthly * * *   rm -f /var/spool/cron/lastrun/cron.monthly
@ 10 if /usr/bin/test -x /usr/bin/ionice; then /usr/bin/ionice -c3 /usr/sbin/run-crons; else /usr/sbin/run-crons; fi

BTW: the ionice utility is also available via my scripting tools package: https://stier.dynu.com/~myportage/sys-l ... ipt-tools/

neuron · Post by **neuron** » Thu Sep 14, 2006 2:01 pm

ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly.

Cinquero · Post by **Cinquero** » Thu Sep 14, 2006 2:08 pm

neuron wrote:ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly.

Uhm, isn't that what -c3 says? To delay any io operations as long as there are higher-prio IOs? 99% iowait tells you that there is IO going on, so that seems perfect to me (as long as there actually IS some non -c3 IO).

neuron · Post by **neuron** » Thu Sep 14, 2006 2:19 pm

Cinquero wrote:
neuron wrote:ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly.
Uhm, isn't that what -c3 says? To delay any io operations as long as there are higher-prio IOs? 99% iowait tells you that there is IO going on, so that seems perfect to me (as long as there actually IS some non -c3 IO).

yes, but it's a tad bit extreme.

Like in I can copy stuff in about 100kb/sec, while downloading something at 1mb/sec. It should not go anywhere near that slow. And ionice -c2 -p7 gives me 20-30mb/sec instead

bludger · Post by **bludger** » Sat Sep 16, 2006 6:12 am

This looks great. Any idea when it will be unmasked in portage?

Cinquero · Post by **Cinquero** » Sun Sep 17, 2006 9:25 pm

Install my scripting tools package. It includes the ionice binary. Don't wait for Gentoo. You have no guarantee that they will ever support it.

https://stier.dynu.com/~myportage/sys-l ... ipt-tools/

Question:

does anyone else experience the following problem?

1.) ionice -c3 emerge ... some pkg

2.) rsync / target (in parallel to 1)

3.) some GUI (ie. tvtime) with regular io and cpu priorities hangs as long as rsync builds the filelist... that is really ANNOYING and definitely a bug.

neuron · Post by **neuron** » Mon Sep 18, 2006 12:11 am

Cinquero wrote:Install my scripting tools package. It includes the ionice binary. Don't wait for Gentoo. You have no guarantee that they will ever support it.

it's in sys-apps/util-linux-2.13_pre7

Cinquero · Post by **Cinquero** » Mon Sep 18, 2006 12:15 am

neuron wrote:it's in sys-apps/util-linux-2.13_pre7

Not unmasked.

neuron · Post by **neuron** » Mon Sep 18, 2006 12:23 am

Cinquero wrote:
neuron wrote:it's in sys-apps/util-linux-2.13_pre7
Not unmasked.

no but "You have no guarantee that they will ever support it.", that's fairly unlikely

Cinquero · Post by **Cinquero** » Mon Sep 18, 2006 12:29 am

neuron wrote:no but "You have no guarantee that they will ever support it.", that's fairly unlikely

Conversation considered useless.

fgrosshans · Post by **fgrosshans** » Tue Sep 19, 2006 4:14 pm

Thanks a lot. It makes emerge-by-working practical on my laptop !

It just lacks the way to (io)renice an already started process. For a started emerge, for exemple, you can find the relevant PID by

Code: Select all

 # ps -A | grep emerge
22066 tty2     00:00:19 emerge

and then ...

Code: Select all

 ionice -c3 -p2206

pijalu · Post by **pijalu** » Tue Sep 19, 2006 6:44 pm

Something like this script may help

Code: Select all

#!/bin/bash

cmd="$@"
if [ -z "$cmd" ]
 then
    echo "feed me something..."
    exit 0
fi

# get all options
params=`echo $@ | sed -e "s/ /\n/g" | sed -n "/^-/p"`
# get rid of extra \n
params=`echo $params`

# set a def val if needed
if [ -z "$params" ]
then
    params="-c3"
fi

# for each given prog name
for item in `echo $@ | sed "s/-.*//g" | sed "s/\ /\n/g"`
do
  # for each process hitting this
  for pid in `ps -A | grep $item | sed "s/[^ 0-9].*//g"`
  do
      echo "ionicing $item ($pid) with $params"
      ionice $params -p$pid
  done
done

usage: iorenice [partial prog name] [ionice options]
Note: This will take ALL hit to partial prog name (multiple should work) and apply the give class/prio (-c3 by def)

Cinquero · Post by **Cinquero** » Tue Sep 19, 2006 6:59 pm

Uhm, my mscript-tools script allows to nice a complete process hierarchy. Just give it the parent process id:

mscript-tools NiceAll `pidof emerge`

You just need to replace nice with ionice in it.

MaratIK1 · Post by **MaratIK1** » Thu Sep 21, 2006 4:52 pm

Request for ebuild added:
https://bugs.gentoo.org/show_bug.cgi?id=148519

Cinquero · Post by **Cinquero** » Sat Sep 30, 2006 6:50 pm

Summary of my experience with the CFQ scheduler: H-O-R-R-I-B-L-E. Just plain useless. When I do not use io prios, a simple "rm -rf /var/tmp/ccache" may lock the entire desktop until it has finished.... even NFS transfers have stalled completely. I'm switching back to AS.

bludger · Post by **bludger** » Mon Oct 02, 2006 9:58 am

MaratIK1 wrote:Request for ebuild added:
https://bugs.gentoo.org/show_bug.cgi?id=148519

According to this link, ionice is included in schedutils 1.5.0, which appears to be already in portage. Has anyone tried this out?

bludger · Post by **bludger** » Mon Oct 02, 2006 10:09 am

Cinquero wrote:Summary of my experience with the CFQ scheduler: H-O-R-R-I-B-L-E. Just plain useless. When I do not use io prios, a simple "rm -rf /var/tmp/ccache" may lock the entire desktop until it has finished.... even NFS transfers have stalled completely. I'm switching back to AS.

I haven't tried it out myself, so I can't really judge, but it seems contrary to the theory of CFQ as far as I understand it. According to this link, io should be distributed fairly between all processes. Can someone else confirm this phenomenon? I find it strange that Redhat have made this the default scheduler, if it really is so bad. Can you confirm that everything is running with IOPRIO_CLASS_BE (Best effort) and that nothing is running with IOPRIO_CLASS_RT (Real time)?

Maybe I'll have to try it out for myself. BTW if I reboot with the boot flag "elevator=cfq" will that select cfq as default? I would rather not recompile my kernel just for this test.

bludger · Post by **bludger** » Mon Oct 02, 2006 11:12 am

bludger wrote:Maybe I'll have to try it out for myself. BTW if I reboot with the boot flag "elevator=cfq" will that select cfq as default? I would rather not recompile my kernel just for this test.

Just found this: /usr/src/linux/Documentation/block/switching-sched.txt. It is possible to change the scheduler on the fly with:

echo cfq > /sys/block/<device>/queue/scheduler

[HOWTO] Fun with io priorities aka ionice

[HOWTO] Fun with io priorities aka ionice

Thanks for the trick...

ebuild

Re: ebuild