AMD64 system slow/unresponsive during disk access (Part 2)

[yoshi314]

so, did anybody happen to be early adopter of these patches?

http://www.phoronix.com/scan.php?page=n ... &px=ODQ3Mw ( or direct LKML : http://lkml.org/lkml/2010/8/1/40 )

i don't have an amd64 system atm, so can't test. i wonder how much of a difference does it make.

[Jorgo]

i tried with gentoo-sources-2.6.35 but the second patch part fails ...
so restored the original file.

[kernelOfTruth]

i tried with gentoo-sources-2.6.35 but the second patch part fails ...
so restored the original file.

yeah, it's failing because naiming of items / stuff changed:

where @line 1337 (interesting line number eh ? )

it was looking for:

--- mm/vmscan.c 2010-07-20 11:21:08.000000000 +0800
+++ mm/vmscan.c 2010-08-01 16:47:52.000000000 +0800
@@ -1337,14 +1378,8 @@

nr_reclaimed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);

- /*
- * If we are direct reclaiming for contiguous pages and we do
- * not reclaim everything in the list, try again and wait
- * for IO to complete. This will stall high-order allocations
- * but that should be acceptable to the caller
- */
- if (nr_reclaimed < nr_taken && !current_is_kswapd() &&
- sc->lumpy_reclaim_mode) {
+ /* Check if we should syncronously wait for writeback */
+ if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
congestion_wait(BLK_RW_ASYNC, HZ/10);

/*

it's should be something like (I changed this manually so it would be better if you changed it manually - a real diff will follow later if it was successful):

--- mm/vmscan.c 2010-07-20 11:21:08.000000000 +0800
+++ mm/vmscan.c 2010-08-01 16:47:52.000000000 +0800
@@ -1244,14 +1356,8 @@

nr_freed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);

- /*
- * If we are direct reclaiming for contiguous pages and we do
- * not reclaim everything in the list, try again and wait
- * for IO to complete. This will stall high-order allocations
- * but that should be acceptable to the caller
- */
- if (nr_freed < nr_taken && !current_is_kswapd() &&
- sc->lumpy_reclaim_mode) {
+ /* Check if we should syncronously wait for writeback */
+ if (should_reclaim_stall(nr_taken, nr_freed, priority, sc)) {
congestion_wait(BLK_RW_ASYNC, HZ/10);

/*

confusing:
nr_reclaimed changed into nr_freed

should_reclaim_stall <-- didn't change


edit:
don't forget the 2nd patch !

[kernelOfTruth]

here you go:

desktop-responsiveness_2.6.35_fix.patch

Code: Select all

--- /usr/src/sources/kernel/zen-upstream/mm/vmscan.c	2010-07-21 17:01:20.911512995 +0200
+++ mm/vmscan.c	2010-08-04 22:11:43.663379966 +0200
@@ -1113,6 +1113,47 @@
 }
 
 /*
+ * Returns true if the caller should wait to clean dirty/writeback pages.
+ *
+ * If we are direct reclaiming for contiguous pages and we do not reclaim
+ * everything in the list, try again and wait for writeback IO to complete.
+ * This will stall high-order allocations noticeably. Only do that when really
+ * need to free the pages under high memory pressure.
+ */
+static inline bool should_reclaim_stall(unsigned long nr_taken,
+					unsigned long nr_freed,
+					int priority,
+					struct scan_control *sc)
+{
+	int lumpy_stall_priority;
+
+	/* kswapd should not stall on sync IO */
+	if (current_is_kswapd())
+		return false;
+
+	/* Only stall on lumpy reclaim */
+	if (!sc->lumpy_reclaim_mode)
+		return false;
+
+	/* If we have relaimed everything on the isolated list, no stall */
+	if (nr_freed == nr_taken)
+		return false;
+
+	/*
+	 * For high-order allocations, there are two stall thresholds.
+	 * High-cost allocations stall immediately where as lower
+	 * order allocations such as stacks require the scanning
+	 * priority to be much higher before stalling.
+	 */
+	if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
+		lumpy_stall_priority = DEF_PRIORITY;
+	else
+		lumpy_stall_priority = DEF_PRIORITY / 3;
+
+	return priority <= lumpy_stall_priority;
+}
+
+/*
  * shrink_inactive_list() is a helper for shrink_zone().  It returns the number
  * of reclaimed pages
  */
@@ -1202,15 +1243,8 @@
 		nr_scanned += nr_scan;
 		nr_freed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);
 
-		/*
-		 * If we are direct reclaiming for contiguous pages and we do
-		 * not reclaim everything in the list, try again and wait
-		 * for IO to complete. This will stall high-order allocations
-		 * but that should be acceptable to the caller
-		 */
-		if (nr_freed < nr_taken && !current_is_kswapd() &&
-		    sc->lumpy_reclaim_mode) {
-			congestion_wait(BLK_RW_ASYNC, HZ/10);
+		/* Check if we should syncronously wait for writeback */
+		if (should_reclaim_stall(nr_taken, nr_freed, priority, sc)) {
 
 			/*
 			 * The attempt at page out may have made some

kudos to Wu Fengguang and KOSAKI Motohiro

[kernelOfTruth]

I'm not sure if I'm already that tired to perceive everything as blazing fast

but it really seems to make a difference (I'm updating around 700 GBs of data via rsync right now):

all apps almost instantly launch

[Jorgo]

Thanks a lot for your work, but it is not working for me.

I'm using gentoo-sources 2.6.35. You seems to use zen-sources ...

I tried to adopt but i get a reject error.

Code: Select all

mm/vmscan.c |   51 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 43 insertions(+), 8 deletions(-)
--- mmotm.orig/mm/vmscan.c	2010-07-20 11:21:08.000000000 +0800
+++ mmotm/mm/vmscan.c	2010-08-01 16:47:52.000000000 +0800
@@ -1113,6 +1113,47 @@ static noinline_for_stack void update_is
 }
 
 /*
+ * Returns true if the caller should wait to clean dirty/writeback pages.
+ *
+ * If we are direct reclaiming for contiguous pages and we do not reclaim
+ * everything in the list, try again and wait for writeback IO to complete.
+ * This will stall high-order allocations noticeably. Only do that when really
+ * need to free the pages under high memory pressure.
+ */
+static inline bool should_reclaim_stall(unsigned long nr_taken,
+               unsigned long nr_freed,
+               int priority,
+               struct scan_control *sc)
+{
+   int lumpy_stall_priority;
+
+   /* kswapd should not stall on sync IO */
+   if (current_is_kswapd())
+      return false;
+
+   /* Only stall on lumpy reclaim */
+   if (!sc->lumpy_reclaim_mode)
+      return false;
+
+   /* If we have relaimed everything on the isolated list, no stall */
+   if (nr_freed == nr_taken)
+      return false;
+
+   /*
+    * For high-order allocations, there are two stall thresholds.
+    * High-cost allocations stall immediately where as lower
+    * order allocations such as stacks require the scanning
+    * priority to be much higher before stalling.
+    */
+   if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
+      lumpy_stall_priority = DEF_PRIORITY;
+   else
+      lumpy_stall_priority = DEF_PRIORITY / 3;
+
+   return priority <= lumpy_stall_priority;
+}
+
+/*
  * shrink_inactive_list() is a helper for shrink_zone().  It returns the number
  * of reclaimed pages
  */
@@ -1202,15 +1243,8 @@
       nr_scanned += nr_scan;
       nr_freed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);
 
-      /*
-       * If we are direct reclaiming for contiguous pages and we do
-       * not reclaim everything in the list, try again and wait
-       * for IO to complete. This will stall high-order allocations
-       * but that should be acceptable to the caller
-       */
-      if (nr_freed < nr_taken && !current_is_kswapd() &&
-          sc->lumpy_reclaim_mode) {
-         congestion_wait(BLK_RW_ASYNC, HZ/10);
+      /* Check if we should syncronously wait for writeback */
+      if (should_reclaim_stall(nr_taken, nr_freed, priority, sc)) {
 
          /*
           * The attempt at page out may have made some

Code: Select all

patching file mm/vmscan.c
Hunk #1 succeeded at 1194 (offset 81 lines).
patch unexpectedly ends in middle of line
Hunk #2 FAILED at 1243.
1 out of 2 hunks FAILED -- saving rejects to file mm/vmscan.c.rej

vmscan.c.rej:

Code: Select all

--- mm/vmscan.c 2010-07-20 11:21:08.000000000 +0800
+++ mm/vmscan.c 2010-08-01 16:47:52.000000000 +0800
@@ -1243,15 +1284,8 @@
       nr_scanned += nr_scan;
       nr_freed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);
 
-      /*
-       * If we are direct reclaiming for contiguous pages and we do
-       * not reclaim everything in the list, try again and wait
-       * for IO to complete. This will stall high-order allocations
-       * but that should be acceptable to the caller
-       */
-      if (nr_freed < nr_taken && !current_is_kswapd() &&
-          sc->lumpy_reclaim_mode) {
-         congestion_wait(BLK_RW_ASYNC, HZ/10);
+      /* Check if we should syncronously wait for writeback */
+      if (should_reclaim_stall(nr_taken, nr_freed, priority, sc)) {
 
          /*
 
mm/vmscan.c.rej lines 1-20/20 (END) 

[SlashBeast]

http://paste.pocoo.org/raw/r59CezZil6xEbcKH5laK/

Apply cleanly on 2.6.35 vanilla with tuxonice and linux-phc (only 2 hunks).

[Shining Arcanine]

For any ignorant people like me that are not well versed in the use of patch, you can do the following:

Code: Select all

wget -O desktop-responsiveness_2.6.35_fix.patch http://paste.pocoo.org/raw/r59CezZil6xEbcKH5laK/
cd /usr/src/linux
patch -p6 < $OLDPWD/desktop-responsiveness_2.6.35_fix.patch

This assumes that you eselected the appropriate kernel.

By the way, I am using sys-kernel/vanilla-sources-2.6.35 and patch is emitting a minor warning:

# patch -p6 < /root/desktop-responsiveness_2.6.35_fix.patch
patching file mm/vmscan.c
Hunk #1 succeeded at 1112 (offset -1 lines).
patch unexpectedly ends in middle of line
Hunk #2 succeeded at 1242 with fuzz 1 (offset -1 lines).

Anyway, I am recompiling my kernel now. I did not know that there was a problem, but I am happy to have it fixed.

Edit: I have an Intel X25-M SSD. Someone at Phoronix claimed dd if=/dev/zero of=test bs=1M count=5024 && rm test -f would bring unpatched systems to a crawl, but I have run it on my system in both a patched and unpatched state and I am having difficulty seeing an improvement. Minor lag is still there when opening programs with that command running. The system can also be unresponsive when switching between open programs with kwin while the command is running. Some lag also occurs in closing programs. The lag does not last more than few seconds, but nevertheless, it is there, regardless of whether or not my system is patched. On the other hand, Google Chromium feels more responsive. I am not sure if it is because my system has not been running long enough for there to be a slow-down (occasionally, I can notice tearing due to repaints), but so far, those issues have not appeared.

[SlashBeast]

Someone at Phoronix claimed dd if=/dev/zero of=test bs=1M count=5024 && rm test -f would bring unpatched systems to a crawl, but I have run it on my system in both a patched and unpatched state and I am having difficulty seeing an improvement.

Orly? 'piotr' at phoronix is me. And I only said, that with this patch I have lags when doing much disk activity. I haven't said that without it is better or worst, I see no improvement.

[NaterGator]

I patched a few minute ago, and copying large (1.3GB) video files across two disks definitely resulted in a GUI slowdown/near halt before the patch. With the patch things are markedly improved, but not perfect.

Looking at IO wait in top still shows some significant stalling, but apparently the patch did make a difference. FWIW I noticed the issue the most when performing IO operations with particularly slow devices, like old USB sticks.

[Shining Arcanine]

Someone at Phoronix claimed dd if=/dev/zero of=test bs=1M count=5024 && rm test -f would bring unpatched systems to a crawl, but I have run it on my system in both a patched and unpatched state and I am having difficulty seeing an improvement.

Orly? 'piotr' at phoronix is me. And I only said, that with this patch I have lags when doing much disk activity. I haven't said that without it is better or worst, I see no improvement.

Sorry, I misread your post at Phoronix.

[kernelOfTruth]

I patched a few minute ago, and copying large (1.3GB) video files across two disks definitely resulted in a GUI slowdown/near halt before the patch. With the patch things are markedly improved, but not perfect.

Looking at IO wait in top still shows some significant stalling, but apparently the patch did make a difference. FWIW I noticed the issue the most when performing IO operations with particularly slow devices, like old USB sticks.

this

I've ZFS on top of dm / luks on an old USB harddrive and there are no interruptions in the GUI noticable anymore !

besides those 2 there are 5 additional patches but I don't have the time to port those since it's somewhat more tricky to do those - hopefully the zen devs will be able to port them

on non-slow volumes it's sometimes better, sometimes worse - overall it's better than before

[Jorgo]

This patch for 2.6.35 applies with without error but when i compile:

Code: Select all

 CC      mm/vmscan.o
  CC      kernel/sysctl.o
mm/vmscan.c:1163: Fehler: Redefinition von »should_reclaim_stall«
mm/vmscan.c:1122: Anmerkung: Vorherige Definition von »should_reclaim_stall« war hier
mm/vmscan.c:1204: Fehler: Redefinition von »should_reclaim_stall«
mm/vmscan.c:1163: Anmerkung: Vorherige Definition von »should_reclaim_stall« war hier
make[1]: *** [mm/vmscan.o] Fehler 1
make: *** [mm] Fehler 2

https://bugzilla.kernel.org/attachment.cgi?id=27314

EDIT: File seems to be patched before by gentoo-patchset.
Working with file from vanilla sources.

[kernelOfTruth]

This patch for 2.6.35 applies with without error but when i compile:

Code: Select all

 CC      mm/vmscan.o
  CC      kernel/sysctl.o
mm/vmscan.c:1163: Fehler: Redefinition von »should_reclaim_stall«
mm/vmscan.c:1122: Anmerkung: Vorherige Definition von »should_reclaim_stall« war hier
mm/vmscan.c:1204: Fehler: Redefinition von »should_reclaim_stall«
mm/vmscan.c:1163: Anmerkung: Vorherige Definition von »should_reclaim_stall« war hier
make[1]: *** [mm/vmscan.o] Fehler 1
make: *** [mm] Fehler 2

https://bugzilla.kernel.org/attachment.cgi?id=27314

that's the one from the zen-kernel devs:
git.zen-kernel.org

[Jorgo]

Playing around for 1 hour now with patched system.
Still some hick-ups but not so bad as before.
So if there are some other patches i think they found the first part of the issue but not all ...

[lagalopex]

While the original patch on lkml did not remove the "congestion_wait" call, it is removed in all the posted patches in here.
And the patch was not for the mainline 2.6, its for the "-mm tree of the moment"-kernel.

[kernelOfTruth]

While the original patch on lkml did not remove the "congestion_wait" call, it is removed in all the posted patches in here.
And the patch was not for the mainline 2.6, its for the "-mm tree of the moment"-kernel.

there you go !

that explains why it's so different from the vanilla-kernel (I first had suspected that it was from linux-next)

hopefully someone will backport the rest of the patches or the devs create a bunch of those

[DaggyStyle]

any known source has this patch?

[devsk]

Couple of question folks:

1. Which patch has all the fixes?
2. Has anybody seen any side effects like reduced throughput or copy speed?

[yoshi314]

i'm guessing zen-sources is the way to go atm. not sure if it contains the complete patch.

[kernelOfTruth]

i'm guessing zen-sources is the way to go atm. not sure if it contains the complete patch.

it does

the 2.6.35 branch is the way to go - only the first 2 patches are needed for that because it already includes the additional ones

for 2.6.34 you need a lot more but it's also included with zen-stable

[darklegion]

It seems that using the deadline i/o scheduler is still useful with some systems. I'm using 2.6.35-zen2 (which includes the vmscan patches) and certain large file copies results in the dreaded multiple second pauses with cfq and bfq. Using ionice helps with bfq and cfq, but at the cost of greatly reduced throughput. With deadline the pauses are gone, and throughput seems to be fine. I realise that this may not help with all systems, but it certainly helped in my case.

EDIT: Never mind, the pauses are still there. Reduced somewhat, but still there.

[kernelOfTruth]

It seems that using the deadline i/o scheduler is still useful with some systems. I'm using 2.6.35-zen2 (which includes the vmscan patches) and certain large file copies results in the dreaded multiple second pauses with cfq and bfq. Using ionice helps with bfq and cfq, but at the cost of greatly reduced throughput. With deadline the pauses are gone, and throughput seems to be fine. I realise that this may not help with all systems, but it certainly helped in my case.

EDIT: Never mind, the pauses are still there. Reduced somewhat, but still there.

2.6.36 in that regard is much better

[devsk]

It seems that using the deadline i/o scheduler is still useful with some systems. I'm using 2.6.35-zen2 (which includes the vmscan patches) and certain large file copies results in the dreaded multiple second pauses with cfq and bfq. Using ionice helps with bfq and cfq, but at the cost of greatly reduced throughput. With deadline the pauses are gone, and throughput seems to be fine. I realise that this may not help with all systems, but it certainly helped in my case.

EDIT: Never mind, the pauses are still there. Reduced somewhat, but still there.

2.6.36 in that regard is much better

Does 2.6.36 include the fix from Wu Fengguang?

[kernelOfTruth]

It seems that using the deadline i/o scheduler is still useful with some systems. I'm using 2.6.35-zen2 (which includes the vmscan patches) and certain large file copies results in the dreaded multiple second pauses with cfq and bfq. Using ionice helps with bfq and cfq, but at the cost of greatly reduced throughput. With deadline the pauses are gone, and throughput seems to be fine. I realise that this may not help with all systems, but it certainly helped in my case.

EDIT: Never mind, the pauses are still there. Reduced somewhat, but still there.

2.6.36 in that regard is much better

Does 2.6.36 include the fix from Wu Fengguang?

you mean the posted 2 patches which also were written about on phoronix.com ?

sure, they went in pretty early from the start