Disk I/O locks up system

chaoscommander · Tux's lil' helper Joined: 15 Oct 2012 Posts: 106

I often get lockups/hangs that last for a few seconds to several minutes, whenever a process has to do a lot of disk I/O. I took a look with iotop to see if it's one specific process hogging the disk. Sometimes it's a regular update process, like mlocate, makewhatis, baloo etc., but sometimes (especially when waking up from hibernation) it's a bunch of userspace processes (firefox, libreoffice, kde itself, whatever has been hibernating) doing whatever it is they are doing with the complete HDD bandwidth.

I first suspected it was due to my machine being a little older, but it's always the disk, not the CPU that is at 100% capacity. It also happens with a fairly fast ThinkPad that has a SSD in it (albeit more rarely, so far only when large packages are being merged). Is there a way to reconfigure the IO scheduler so it will allocate bandwidth more equally instead of letting single threads block the entire disk? I'm using CFQ I/O scheduler.
Or am I looking for the problem in an entirely wrong direction?

eccerr0r · Posted: Tue Mar 10, 2015 2:00 pm Post subject:

If your machine completely hangs, I'd say it's a hardware issue and perhaps it's time the machine needs to be replaced. My machines can take 100% disk i/o load just fine. Is it just Gentoo that exhibits this failure mode? When you run 'dmesg' if it comes back from hanging, does it report something?

If it's just "so slow that there's virtually no forward progress" that's a different case. There's ionice or you can make cgroups to throttle but you can't use this on swapping and kernel (resume from disk) so if it's crashing during either of these, see above.

Make sure you're using the right kernel disk io driver too, or try a different one (use specific driver instead of generic, or vice versa).
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

chithanh · Posted: Tue Mar 10, 2015 2:50 pm Post subject:

The system should not lock up fully, even during disk I/O.

Try to enable voluntary or full kernel preemption. Also in my experience, the deadline scheduler behaves better with heavy I/O load and interactive processes.

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2006

I suspect the problem is swapping. Using firefox and libreoffice on my (1 Gb RAM) laptop shows typical symptoms: you can use one or the other continuously, but changing between them causes the disk light to come on hard, and a minute or so before it the system starts to respond again.

I have a suspicion that Javascript is implicated somewhere in firefox - my laptop frequently slows right down on some web pages that appear pretty innocuous, so I wonder if there are memory leaks or alternatively wasteful algorithms that run fast on machines with lots of RAM but slow where RAM is constrained.
_________________
Greybeard

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Cyker · Veteran Joined: 15 Jun 2006 Posts: 1746

Also, check the power management setup for your Thinkpad; Some of them, esp. newer ones, have known issues with ASPM which can cause weird problems like this.

I had to disable ASPM completely because every time I plugged in an ExpressCard it would make the whole system 'pause' every couple of seconds.

(Mine is an X230 for reference)

Mr. M · Posted: Thu Mar 12, 2015 4:22 pm Post subject:

I'm also seeing lock-ups on a thinkpad t420s. This started happening after upgrading my system a few days ago (kernel from 3.14.33 to 3.18.7; I hadn't updated my system for a while). The system seems to hang freeze completely when e.g. emerging packages in a KDE terminal. The emerge job continues (I see the HD led blinking) but everything else is frozen (mouse pointer does not move, cannot switch windows using Alt-Tab, cannot switch VT using e.g. Alt-F1). The system does come back after a while (5-10 minutes).

I never had this problem before to this degree, so I wonder if it is just the IO scheduler (did it get worse for desktop use recently?). I will try using the BFQ scheduler. What is the best way for enabling this? Is it just USE=experimental for gentoo-sources and then enable it in the kernel config (where?)?

chaoscommander · Tux's lil' helper Joined: 15 Oct 2012 Posts: 106

Mr. M · Posted: Fri Mar 13, 2015 12:47 pm Post subject:

I noticed that the "baloo_file_extr" process was constantly running and creating a lot of IO. After disabling it, my system seems to be more responsive again.

You can disable it by adding "Indexing-Enabled=false" to ~/.kde4/share/config/baloofilerc

Mr. M · Posted: Mon Mar 16, 2015 6:35 pm Post subject:

Switching to the BFQ scheduler made a huge difference in terms of responsiveness

.

chaoscommander · Tux's lil' helper Joined: 15 Oct 2012 Posts: 106

I second that. It still isn't perfect, but much, much better. Waking up from hibernation now is about 5 times faster. (1 minute instead of 5).

chaoscommander · Tux's lil' helper Joined: 15 Oct 2012 Posts: 106

Hm.. it still hangs massively on big emerges, though.

I just found this in dmesg, about 20sec after wakeup from hibernation, but I'm not sure if it's related.

chaoscommander · Tux's lil' helper Joined: 15 Oct 2012 Posts: 106

That trace dump has not recurred -> seems unrelated.
Still: my responsiveness issues persist, albeit to a lesser extent than before. Does anyone have any other ideas?

Yamakuzure · Posted: Fri May 22, 2015 9:53 am Post subject:

kernelOfTruth · Posted: Fri May 22, 2015 7:04 pm Post subject:

shazeal · Posted: Fri May 22, 2015 8:17 pm Post subject:

shazeal · Posted: Fri May 22, 2015 8:26 pm Post subject:

Found the solution to the ZFS issue! Its not BFQ's fault at all.

https://bbs.archlinux.org/viewtopic.php?id=196439

kernelOfTruth · Posted: Fri May 22, 2015 9:54 pm Post subject:

shazeal · Posted: Sat May 23, 2015 7:05 am Post subject:

energyman76b · Advocate Joined: 26 Mar 2003 Posts: 2048 Location: Germany

kernelOfTruth · Posted: Mon May 25, 2015 9:42 pm Post subject:

energyman76b · Advocate Joined: 26 Mar 2003 Posts: 2048 Location: Germany

noop
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.

Yamakuzure · Posted: Wed May 27, 2015 8:38 am Post subject:

Strange. I will try with noop, then. But why is CFQ working just fine?

Has anybody ever compared using noop versus injecting using bfq to zfs_vdev_scheduler?

Edit: KernelOfTruth wrote: "when using BFQ, switching the vdev-scheduler to BFQ also should significantly reduce latency: echo bfq > /sys/module/zfs/parameters/zfs_vdev_scheduler"

So, better than noop? Riddles...
_________________
Important German:

"Aha" - German reaction to pretend that you are really interested while giving no f*ck.
"Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.

haarp · Guru Joined: 31 Oct 2007 Posts: 535

Also, if you're using an SSD and ext4 wit hthe discard option, deleting lots of files can cause considerable delays as they get TRIMed from the disk.

krinn · Watchman Joined: 02 May 2003 Posts: 7470

Or maybe you are blaming the wrong thing?

While scheduler should switch to each task and give it an amount of time to work, if the scheduler is unable to switch, you can use whatever scheduler you wish, it won't switch.

Why it couldn't switch, if your hardware is busy and block interrupt, scheduler (the one you wish) must wait the hardware to stop doing that.
What you should look at? Stuff like enabling MSI, use APIC instead of PIC, irq conflict for older hw (one using PIC only)...
If you have IRQ conflict and two devices try to work, one device may hide the other device request and worst may re-run the first one.
Say a mouse and hdd, when your hdd is writing and the mouse is moved, the mouse movement may get just ignore because every time the mouse move, the conflict might gives hand to the hdd again. Result: hdd is busy writing, mouse move, hdd is ask to continue while mouse is ignore. And user may see hdd doing its work, while mouse get no answer at all, in the mean time, poor scheduler is waiting the hdd to end before it can work.
Of course a mouse have no IRQ, but usb mouse are using the usb controller irq, and lame motherboard are used to shared usb controller with hdd controller.
You should also look at broken apic/pic option in kernel (CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS), and maybe using hpet that would gave a finer timer resolution and may help scheduler balancing.
If you lack memory and the task was swap, there's nothing you can do, the hdd must read the swap to restore the memory because the scheduler is switching to it, in this case, i suppose if noop scheduler is doing less switching, then it would be the best to use.

For your information, i just never had any problem like this using deadline scheduler, however i don't think the scheduler type makes any difference there (as long as memory is not an issue).