How can I tell which process is causing disk activity?

Message

Charles Alexander · Post by **Charles Alexander** » Thu Mar 30, 2006 8:20 pm

I'm getting a lot of spurious disk activity; I'm not using more than a few KB of swap so I don't that is it. I'm wondering if there is a tool which can act like "top" does for processor time to basically tell me which processes are using the most disk time. I've found lsof and dstat, but neither of these apps seems to help me. lsof just tells me which processes have files open--I can't find which ones are actively reading/writing to those files. dstat seems to be just an overall disk usage monitor.

sschlueter · Post by **sschlueter** » Fri Mar 31, 2006 12:16 am

I think none of the common tools like top, vmstat, iostat, mpstat, lsof, ... can display disc acticity by process id.

Filemon from sysinternals.com can display file system activity in real time, but as far as I can see this tool isn't available from their home page any more. While I'm sure that one could still find a download location for this tool, I guess that it wouldn't work with recent kernels anyway.

Unfortunately, I haven't found an alternative to Filemon yet.

There just doesn't seem to exist a simple real time file system monitor, although it should be easy to create one because the required kernel support seems to exist. In fact, there are several implementations for the kernel. But I don't know if they also report the process id that causing the file access.

There is inotify, which is part of current vanilla kernels. Beagle is the most poplar software package that uses inotify.

There is fschange http://stefan.buettcher.org/cs/fschange/index.html
This one seems particularly easy to use, perhaps tail -f /proc/fschange is sufficient to get an impression which process is causing most disk activity. Doesn't use inotify, needs kernel patching.

And there is Dazuko http://www.dazuko.org/
Since it can also block access to certain files, it it used by several virus scanners. Uses its own kernel module.

If a tool can collect all file system activity, it would then be easy to generate some statistics like the percentage of disk activity per process id.

So, if anybody knows a simple file system activity monitor that uses one of the mentioned kernel modules, please tell us about it.

Charles Alexander · Post by **Charles Alexander** » Fri Mar 31, 2006 7:38 pm

Is there anything to tell perhaps which processes are currently blocking on IO?

Earthwings · Post by **Earthwings** » Fri Mar 31, 2006 7:54 pm

lm-profiler from app-laptop/laptop-mode-tools should be what you're looking for. See http://www.gentoo.org/doc/en/power-mana ... #doc_chap5

sschlueter · Post by **sschlueter** » Fri Mar 31, 2006 8:48 pm

I have found http://pyinotify.sourceforge.net/ which implements a small monitor that uses inotify. But I'm afraid to tell you that this tool does not do what I want. It recognizes when files are opened and closed. But it does not tell the pid. And it does not recognize modification of files that are opened and remain open. I have tried this tool on a server that runs a busy mysqld and pyinotify did not report any mysql related activity

sschlueter · Post by **sschlueter** » Sat Apr 01, 2006 6:44 am

I have now installed Dazuko. The example application tells the pids, but just like pyinotify it just seems to report when a file is being opened, it doesn't report changes to an already opened file.

sschlueter · Post by **sschlueter** » Sat Apr 01, 2006 6:49 am

Ah, now I have finally found what I wanted and that is probably useful for Charles Alexander as well.

Real-time file-system activity monitor:

Activate:

Code: Select all

echo 1 > /proc/sys/vm/block_dump

Watch:

Code: Select all

while true; do dmesg -c | grep -v pdflush | grep -v reiserfs/0; sleep 0.5; done

This is a very raw view, it can be used to get a first impression, but it doesn't show any statistics like which process is causing the most disk activity or which process has been "waiting for io" for the longest time.

Earthwings · Post by **Earthwings** » Sat Apr 01, 2006 7:01 am

sschlueter wrote:Ah, now I have finally found what I wanted and that is probably useful for Charles Alexander as well.

Real-time file-system activity monitor:

Activate:
Code: Select all
echo 1 > /proc/sys/vm/block_dump
Watch:
Code: Select all
while true; do dmesg -c | grep -v pdflush | grep -v reiserfs/0; sleep 0.5; done
This is a very raw view, it can be used to get a first impression, but it doesn't show any statistics like which process is causing the most disk activity or which process has been "waiting for io" for the longest time.

And this is what lm-profiler does for you, see my post above

sschlueter · Post by **sschlueter** » Sat Apr 01, 2006 3:35 pm

Earthwings wrote: And this is what lm-profiler does for you, see my post above

I've now run lm-profiler. As far as I understand this tool has a different focus. It tries to figure out which services could be disabled in order to optimize notebook hard disk spindown intervals.

For instance, a service that periodicaly reads/writes from/to disk every two minutes (even if the amount of data is small) is bad in this respect.

But this is not relevant for a desktop or server where the hard disk is constantly running.

Charles Alexander wants to know

1) percentage of total disk activity by process id (aka "which process is causing the most disk activity")

2) percentage of total "waiting for io" by process id (aka "which process suffers the most from hard disk performance bottleneck")

I still don't know any tool that could offer this information.