Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
PE 2650 aacraid custom kernel very SLOW Write performance
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
mr_friend
n00b
n00b


Joined: 12 Dec 2007
Posts: 4

PostPosted: Wed Dec 12, 2007 10:09 pm    Post subject: PE 2650 aacraid custom kernel very SLOW Write performance Reply with quote

Hello Everybody,

I have been pulling my hair out for the last few days trying to get the system running normally. This is not my first custom gentoo kernel...

From the liveCD 2007.0 the write performance is good(afaik):
Code:

Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write          40 MBs |    0.4 s | 102.049 MB/s |   0.0 %  | 326.6 % |
| Random Write   16 MBs |    0.6 s |  26.419 MB/s |   0.0 %  |  60.9 % |
| Read           40 MBs |    0.0 s | 2173.204 MB/s | 217.3 %  | 652.0 % |
| Random Read    16 MBs |    0.0 s | 2073.656 MB/s |   0.0 %  |   0.0 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.030 ms |        0.081 ms |  0.00000 |   0.00000 |
| Random Write |        0.020 ms |        0.074 ms |  0.00000 |   0.00000 |
| Read         |        0.007 ms |        0.064 ms |  0.00000 |   0.00000 |
| Random Read  |        0.006 ms |        0.024 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.017 ms |        0.081 ms |  0.00000 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'


But, from my kernel 2.6.23-gentoo-r3 it is SLOW
Code:

Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write          40 MBs |   12.4 s |   3.223 MB/s |   0.3 %  |   4.9 % |
| Random Write   16 MBs |   18.8 s |   0.831 MB/s |   0.0 %  |   2.0 % |
| Read           40 MBs |    0.0 s | 2000.000 MB/s |   0.0 %  | 600.0 % |
| Random Read    16 MBs |    0.0 s | 1562.344 MB/s |   0.0 %  | 400.0 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.015 ms |       40.001 ms |  0.00000 |   0.00000 |
| Random Write |        0.005 ms |       10.000 ms |  0.00000 |   0.00000 |
| Read         |        0.004 ms |       10.000 ms |  0.00000 |   0.00000 |
| Random Read  |        0.003 ms |       10.001 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.008 ms |       40.001 ms |  0.00000 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'


It seems ODD that only the write performance is suffering... everything slows down to a crawl...

Here is the kernel .config 2.6.23-gentoo-r3

Here is my older kernel config that used to work fine: .config 2.6.17-gentoo-r4

Here are the modules that get loaded:
Code:

rtc                    14232  0
ata_generic             8708  0
pata_serverworks       10624  0
evdev                  10880  0
tg3                   102164  0
ohci_hcd               30860  0
usbcore               136016  2 ohci_hcd
8250_pci               23040  0
8250                   32900  1 8250_pci
serial_core            22296  1 8250
serverworks             8068  0 [permanent]
sworks_agp             10784  0
agpgart                33288  1 sworks_agp


Here is the lspci -v output.
Here is the dmesg output, with all kinds of debug info.

Somebody, PLEASE HELP ME. I am going crazy...

I will paypal $50 to whoever gets this resolved for me :-) as a token of appreciation!!!.
_________________
Regards, mr_friend
PS: let's see, I know who made my computer(DELL :-P), then, who made the computer inside my head?
Back to top
View user's profile Send private message
mr_friend
n00b
n00b


Joined: 12 Dec 2007
Posts: 4

PostPosted: Sat Dec 15, 2007 3:31 pm    Post subject: Reply with quote

Hello everybody,

I got ~100 views of the thread :-) but, no replies...

Here's how I resolved the issue... take a quess

I went back to the 2.6.19-gentoo-r5 kernel, which has an older version of the aacraid driver... :-)
_________________
Regards, mr_friend
PS: let's see, I know who made my computer(DELL :-P), then, who made the computer inside my head?
Back to top
View user's profile Send private message
mendo
n00b
n00b


Joined: 12 Jun 2005
Posts: 5

PostPosted: Thu Mar 06, 2008 1:55 pm    Post subject: aacraid slow Reply with quote

I'm having the exact same problem. I have an Adaptec 31205 SAS raid controller and are using the aacraid driver. I test the speed by just unpacking the linux kernel source and delete it again. I have found that the change happened between linux-2.6.22 and linux-2.6.23.

2.6.22:
Unpack: 22.3 seconds.
Delete: 1.8 seconds.

2.6.23:
Unpack: 327 seconds.
Delete: 258 seconds.

The times include flush to disk with the sync command.
I have just tested again with gentoo-sources-2.6.24-r3. That's the same as with 2.6.23.

I have noticed the following difference in the log:
2.6.22:
Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] 2925506560 512-byte hardware sectors (1497859 MB)
Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] Assuming Write Enabled
Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through

2.6.24:
Mar 6 13:55:01 dexter kernel: Driver 'sd' needs updating - please use bus_type methods
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] 2925506560 512-byte hardware sectors (1497859 MB)
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Write Protect is off
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00
Mar 6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA

My controller is installed in a dual opteron dualcore machine. What about yours?
What firmware version do you have on your controller? I have 5.2-0 (15323).

I'm going to try vanilla-sources too.
Back to top
View user's profile Send private message
mendo
n00b
n00b


Joined: 12 Jun 2005
Posts: 5

PostPosted: Mon Mar 10, 2008 11:08 am    Post subject: Reply with quote

I have now tried linux kernel 2.6.25-rc4 (vanilla from kernel.org) to be sure that it hasn't anything with gentoo to do. I also upgraded my controller firmware to build 15611 (the latest from Adaptec) but nothing helped.
I have however found out something else. I'm using the xfs filesystem and I found that if i mount the filesystem with the "nobarrier" option the performance is back to normal. It could seem that the barrier thing in xfs is relying on a feature which has disappeared or got broken in later versions of aacraid.

Actually I should have thought of this earlier. For some time ago I worked with a new iSCSI based raid array. Here I ran into the same problem.
It seems to be a problem with many harddrive controllers. Even on my laptop I have run into this problem. I believe the problem first appeared in kernel 2.6.17. At that time I didn't know about the nobarrier option but now I just tried it and it was in fact that problem again.

I think I can see in your dmesg output that you're also using xfs. Try the nobarrier option!
Back to top
View user's profile Send private message
Poseidon
n00b
n00b


Joined: 18 Apr 2002
Posts: 18
Location: New Hampshire, US

PostPosted: Thu Feb 19, 2009 3:45 pm    Post subject: Thank you, Thank you, Thank you!!! Reply with quote

FFS 8O , I have two poweredge 2650 servers and have done everything in my power to get them working up to speed. I was sure the PERC controller or memory on it was fried at this point. I of course use XFS with Raid-5 for all our media / file server operations for performance reasons. It slowed down awhile ago (prob when I upgraded the kernel), but couldn't figure it out.

You just saved me two servers. I wish I'd read this sooner or you'd have saved me a lot of diagnosing time and hours of sleep.

8)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum