| View previous topic :: View next topic |
| Author |
Message |
mr_friend
n00b
Joined: 12 Dec 2007
Posts: 4
|
 Posted: Wed Dec 12, 2007 10:09 pm Post subject: PE 2650 aacraid custom kernel very SLOW Write performance |
 |
|
Hello Everybody,
I have been pulling my hair out for the last few days trying to get the system running normally. This is not my first custom gentoo kernel...
From the liveCD 2007.0 the write performance is good(afaik):
| Code: |
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 40 MBs | 0.4 s | 102.049 MB/s | 0.0 % | 326.6 % |
| Random Write 16 MBs | 0.6 s | 26.419 MB/s | 0.0 % | 60.9 % |
| Read 40 MBs | 0.0 s | 2173.204 MB/s | 217.3 % | 652.0 % |
| Random Read 16 MBs | 0.0 s | 2073.656 MB/s | 0.0 % | 0.0 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.030 ms | 0.081 ms | 0.00000 | 0.00000 |
| Random Write | 0.020 ms | 0.074 ms | 0.00000 | 0.00000 |
| Read | 0.007 ms | 0.064 ms | 0.00000 | 0.00000 |
| Random Read | 0.006 ms | 0.024 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.017 ms | 0.081 ms | 0.00000 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
|
But, from my kernel 2.6.23-gentoo-r3 it is SLOW
| Code: |
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 40 MBs | 12.4 s | 3.223 MB/s | 0.3 % | 4.9 % |
| Random Write 16 MBs | 18.8 s | 0.831 MB/s | 0.0 % | 2.0 % |
| Read 40 MBs | 0.0 s | 2000.000 MB/s | 0.0 % | 600.0 % |
| Random Read 16 MBs | 0.0 s | 1562.344 MB/s | 0.0 % | 400.0 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.015 ms | 40.001 ms | 0.00000 | 0.00000 |
| Random Write | 0.005 ms | 10.000 ms | 0.00000 | 0.00000 |
| Read | 0.004 ms | 10.000 ms | 0.00000 | 0.00000 |
| Random Read | 0.003 ms | 10.001 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.008 ms | 40.001 ms | 0.00000 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
|
It seems ODD that only the write performance is suffering... everything slows down to a crawl...
Here is the kernel .config 2.6.23-gentoo-r3
Here is my older kernel config that used to work fine: .config 2.6.17-gentoo-r4
Here are the modules that get loaded:
| Code: |
rtc 14232 0
ata_generic 8708 0
pata_serverworks 10624 0
evdev 10880 0
tg3 102164 0
ohci_hcd 30860 0
usbcore 136016 2 ohci_hcd
8250_pci 23040 0
8250 32900 1 8250_pci
serial_core 22296 1 8250
serverworks 8068 0 [permanent]
sworks_agp 10784 0
agpgart 33288 1 sworks_agp
|
Here is the lspci -v output.
Here is the dmesg output, with all kinds of debug info.
Somebody, PLEASE HELP ME. I am going crazy...
I will paypal $50 to whoever gets this resolved for me  as a token of appreciation!!!.
_________________
Regards, mr_friend
PS: let's see, I know who made my computer(DELL  ), then, who made the computer inside my head? |
|
| Back to top |
|
 |
mr_friend
n00b
Joined: 12 Dec 2007
Posts: 4
|
| Posted: Sat Dec 15, 2007 3:31 pm Post subject: |
|
|
Hello everybody,
I got ~100 views of the thread but, no replies...
Here's how I resolved the issue... take a quess
I went back to the 2.6.19-gentoo-r5 kernel, which has an older version of the aacraid driver...
_________________
Regards, mr_friend
PS: let's see, I know who made my computer(DELL ), then, who made the computer inside my head? |
|
| Back to top |
|
|
mendo
n00b
Joined: 12 Jun 2005
Posts: 3
Location: Odense, Denmark
|
| Posted: Thu Mar 06, 2008 1:55 pm Post subject: aacraid slow |
|
|
I'm having the exact same problem. I have an Adaptec 31205 SAS raid controller and are using the aacraid driver. I test the speed by just unpacking the linux kernel source and delete it again. I have found that the change happened between linux-2.6.22 and linux-2.6.23.
2.6.22:
Unpack: 22.3 seconds.
Delete: 1.8 seconds.
2.6.23:
Unpack: 327 seconds.
Delete: 258 seconds.
The times include flush to disk with the sync command.
I have just tested again with gentoo-sources-2.6.24-r3. That's the same as with 2.6.23.
I have noticed the following difference in the log:
2.6.22:
Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] 2925506560 512-byte hardware sectors (1497859 MB)
Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] Assuming Write Enabled
Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
2.6.24:
Mar 6 13:55:01 dexter kernel: Driver 'sd' needs updating - please use bus_type methods
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] 2925506560 512-byte hardware sectors (1497859 MB)
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Write Protect is off
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
My controller is installed in a dual opteron dualcore machine. What about yours?
What firmware version do you have on your controller? I have 5.2-0 (15323).
I'm going to try vanilla-sources too. |
|
| Back to top |
|
|
mendo
n00b
Joined: 12 Jun 2005
Posts: 3
Location: Odense, Denmark
|
| Posted: Mon Mar 10, 2008 11:08 am Post subject: |
|
|
I have now tried linux kernel 2.6.25-rc4 (vanilla from kernel.org) to be sure that it hasn't anything with gentoo to do. I also upgraded my controller firmware to build 15611 (the latest from Adaptec) but nothing helped.
I have however found out something else. I'm using the xfs filesystem and I found that if i mount the filesystem with the "nobarrier" option the performance is back to normal. It could seem that the barrier thing in xfs is relying on a feature which has disappeared or got broken in later versions of aacraid.
Actually I should have thought of this earlier. For some time ago I worked with a new iSCSI based raid array. Here I ran into the same problem.
It seems to be a problem with many harddrive controllers. Even on my laptop I have run into this problem. I believe the problem first appeared in kernel 2.6.17. At that time I didn't know about the nobarrier option but now I just tried it and it was in fact that problem again.
I think I can see in your dmesg output that you're also using xfs. Try the nobarrier option! |
|
| Back to top |
|
|
Poseidon
n00b
Joined: 18 Apr 2002
Posts: 18
Location: New Hampshire, US
|
| Posted: Thu Feb 19, 2009 3:45 pm Post subject: Thank you, Thank you, Thank you!!! |
|
|
FFS  , I have two poweredge 2650 servers and have done everything in my power to get them working up to speed. I was sure the PERC controller or memory on it was fried at this point. I of course use XFS with Raid-5 for all our media / file server operations for performance reasons. It slowed down awhile ago (prob when I upgraded the kernel), but couldn't figure it out.
You just saved me two servers. I wish I'd read this sooner or you'd have saved me a lot of diagnosing time and hours of sleep.
 |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
