Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
RAID5 write performance - Any tips?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
Cyker
Veteran
Veteran


Joined: 15 Jun 2006
Posts: 1703

PostPosted: Sun Mar 09, 2008 11:21 pm    Post subject: RAID5 write performance - Any tips? Reply with quote

Write performance on my software RAID5 is less than a third of the read performance and has very high IOwait.

I'm looking into ways of mitigating this with an eye to improving performance - Any suggestions out there?

The average disk buffer is 2GB of RAM and the array is using 64k stripes and a 4k block size with the recommended stride value.

I have done only minor tweaking with blockdev and hdparm, but they just shift the performance optimizations for certain types of load (i.e. small files vs big files), whereas I'm aiming for something more generic.
(blockdev --setra for md0 composite is 8192 and for sd? components is 2048)

I've tried setting various combinations of IO scheduler on both sd? and md0 devices, and CFQ seems to do the best job on average.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Mon Mar 10, 2008 8:27 pm    Post subject: Reply with quote

Take a look at /sys/block/mdX/md/stripe_cache_active and if it is being maxed out during writes increse stripe_cache_size.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
Cyker
Veteran
Veteran


Joined: 15 Jun 2006
Posts: 1703

PostPosted: Mon Mar 10, 2008 10:32 pm    Post subject: Reply with quote

In the immortal words of Lister.... SMEGGING HELL!

I ran
Quote:
echo 8192 > /sys/block/md0/md/stripe_cache_size
and the sustained write performance has jumped up by almost 300%!!!!

Thanks dude, that is fantastic tip!!! :D

This totally needs to go into the wiki! (Well, when it comes back :( )
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Tue Mar 11, 2008 6:19 pm    Post subject: Reply with quote

If anyone wonders what this does. It will reduce the amount of times that the os will need to read stripes from disk when writing part of a stripe. The reason for the reading is that when all stripes (including partial) are written the parity for the entire stripe needs to be updated.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
Cyker
Veteran
Veteran


Joined: 15 Jun 2006
Posts: 1703

PostPosted: Tue Mar 11, 2008 7:16 pm    Post subject: Reply with quote

So it really is Yet Another Cache, but lower level than the normal disk buffer because it's caching the RAID stripe components?
Given that's one of the most intensive parts of the whole process I can see why boosting this affects the write speed so much ;)

Question: Is there a way to make this 'dynamic' like the normal Linux disk buffer? So that it allocates as much RAM as it can, but shrinks down as 'real' programs need more memory?

Or do you think this would be a waste? (Is the 'diminishing returns' on this much worse than normal caching? My optimum seems to be 8192, but I tested it up to 32768 and it still displays a healthy performance boost, 'tho not as great as the jump to 8192.)

Also, are there any other nifty tweaks like this? :mrgreen:

Now I just need to hope they fix the sky2 driver more (Previously I was getting 40MB/s but it'd stop working randomly under heavy load. Now it always works but peaks at 25-30MB/s) and I'l be sorted! :D
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Tue Mar 11, 2008 7:42 pm    Post subject: Reply with quote

Quote:
So it really is Yet Another Cache, but lower level than the normal disk buffer because it's caching the RAID stripe components?


Yes, this is exactly what it does.

Quote:
Question: Is there a way to make this 'dynamic' like the normal Linux disk buffer? So that it allocates as much RAM as it can, but shrinks down as 'real' programs need more memory?


I am not sure I have not investigated this. I know you can dynamically change the size at any time you want though. And you can always use stripe_cache_active as a guide.

Quote:
Also, are there any other nifty tweaks like this?


None that effect performance as much. Besides the ones you have mentioned I have played around with /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs to try to force linux to hold on to the writes a little longer so that it writes in larger blocks.


Quote:
Now I just need to hope they fix the sky2 driver more

I had problems with that in the past. I believe a kernel change lessened the problem but I have not investigated this in a long time and I am not even sure what machine uses that driver. I do know the systems with I have tg3 gigabit nic drivers work the best. And the nvidia, sky2 and realtek Gbit nics had their problems...
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
richard.scott
Veteran
Veteran


Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK

PostPosted: Fri Oct 24, 2008 2:39 pm    Post subject: Reply with quote

drescherjm wrote:
Take a look at /sys/block/mdX/md/stripe_cache_active and if it is being maxed out during writes increse stripe_cache_size.


I don't have this file on my system?

Code:
 # cd /sys/block/md0/md/
# ls
array_state      component_size  dev-hdb1  metadata_version  raid_disks  reshape_position  suspend_hi   sync_completed  sync_speed_max
bitmap_set_bits  degraded        layout    mismatch_cnt      rd0         resync_start      suspend_lo   sync_max        sync_speed_min
chunk_size       dev-hda1        level     new_dev           rd1         safe_mode_delay   sync_action  sync_speed


and if I try to create it I get this:

Code:
# echo 8192 > stripe_cache_size
-su: stripe_cache_size: No such file or directory


I guess this isn't something I can take advantage of :?
Back to top
View user's profile Send private message
Cyker
Veteran
Veteran


Joined: 15 Jun 2006
Posts: 1703

PostPosted: Fri Oct 24, 2008 4:31 pm    Post subject: Reply with quote

Hmm... that's weird.

Hey, wait, what RAID are you using?
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Fri Oct 24, 2008 10:03 pm    Post subject: Reply with quote

NOTE: You will not have that on raid 0 or raid 1. If this is raid 5 or 6 I agree this is weird.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
richard.scott
Veteran
Veteran


Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK

PostPosted: Mon Oct 27, 2008 1:17 pm    Post subject: Reply with quote

drescherjm wrote:
NOTE: You will not have that on raid 0 or raid 1. If this is raid 5 or 6 I agree this is weird.


It's on RAID 1 :oops:
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Thu Dec 11, 2008 3:44 pm    Post subject: Reply with quote

BTW there are some benchmarks in the following thread to show the effects of this setting:
https://forums.gentoo.org/viewtopic-t-709075-highlight-.html
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
manaka
Apprentice
Apprentice


Joined: 23 Jul 2007
Posts: 178
Location: Spain

PostPosted: Fri Dec 12, 2008 7:14 pm    Post subject: Reply with quote

AFAIK, stripe_cache_size it's not dynamic in the current implementation, but can be changed at any time.

Another trick that can provide a little performance boost is aligning the data with the RAID chunks. In case you use ext3, have a look at the stride and stripe-width options of mke2fs. Other FS may provide similar options...
_________________
Javier Miqueleiz

"Listen to your heart. It knows all things, because it came from the Soul of the World, and it will one day return there."
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Mon Dec 15, 2008 3:56 am    Post subject: Reply with quote

Quote:
Another trick that can provide a little performance boost is aligning the data with the RAID chunks.


I found this to be absolutely essential on windows software raid because they do not have a stripe cache. I have not tried this on linux. I will try to remember that next time I set up an array. I generally do not use ext3 because of its extremely slow deletes on large files (compared to xfs). I believe ext4 solves this so maybe that will be a good choice.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
manaka
Apprentice
Apprentice


Joined: 23 Jul 2007
Posts: 178
Location: Spain

PostPosted: Mon Dec 15, 2008 8:54 pm    Post subject: Reply with quote

BTW, mkfs.xfs also has an option for data aligning with RAID stripe size. It's the -d sunit option. :)

drescherjm, could you provide more details about that option? (I'm not a Windows guy, but for the sake of curiosity...).
_________________
Javier Miqueleiz

"Listen to your heart. It knows all things, because it came from the Soul of the World, and it will one day return there."
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Mon Dec 15, 2008 9:09 pm    Post subject: Reply with quote

Quote:
drescherjm, could you provide more details about that option? (I'm not a Windows guy, but for the sake of curiosity...).


Hmm. I am a little confused at what you are asking but here goes:

On windows you have to partition the disks so that the starting sector is at 2048 instead of 63 or even better fully aligned depending on the number of disks. The following thread has a long discussion of this:


http://forums.storagereview.net/index.php?showtopic=25786

BTW, I use the same username there as here so the raid5 I was talking about there I ended up giving up and used single 750GB drives instead because performance under windows software raid (1, 10 and 5) with less than 5 drives was all slower than a single drive in reads and writes. In my opinion MSFT must have stock in some hardware raid controller company...
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Mon Dec 15, 2008 9:43 pm    Post subject: Reply with quote

Quote:
BTW, mkfs.xfs also has an option for data aligning with RAID stripe size. It's the -d sunit option.

Thanks. I will check into that.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Tue Jan 06, 2009 3:38 am    Post subject: Reply with quote

O.O I went from raid1 to raid5 today - and copying a 600mb file took ages - several minutes, with long pauses, when it was moving, it was around 3mb/sec...
I incresed stripe_cache_size from 256 to 2048 and it went over in a couple of seconds! Wow! Thanks for the hint. 4096 halfed it again - now to 5 seconds ....

all I need is a good place to put this settings somewhere. local.start? but it is the last one called in boot...
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2778
Location: Pittsburgh, PA, USA

PostPosted: Tue Jan 06, 2009 3:42 am    Post subject: Reply with quote

I put them in /etc/conf.d/local.start
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum