Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
The Filesystem choice thread
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... , 29, 30, 31  Next  
This topic is locked: you cannot edit posts or make replies.    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Jul 06, 2008 3:32 pm    Post subject: Reply with quote

Quote:
Realized agcount=8 isn't optimal for me, agcount=4 is actually better, even though I got 4 cpu's in this computer.


yeah, the agcount-thingy is pretty tricky: if you have too much performance deteriorates a lot, it also worsens application loading-time a lot (seemingly) - at least a few: e.g. okular, kpdf take 2-3 times longer to load ...

I currently have agcount=20 (for a 20 gigabyte partition) and acroread takes 30 seconds to load :lol: (acroread specific ?), oowriter loads in just a few seconds,

at least data throughput is nice:

a little comparison between reiserfs & xfs (extraction of a 4.3 gigabyte stage4-tarball (including 3 kernel-sources):

Quote:
reiserfs:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda6 20G 14G 6.5G 68% /mnt/gentoo

xfs (tweaked):
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda6 20G 15G 5.1G 75% /mnt/gentoo

extraction-time:

reiserfs: 20 Minutes (mount -o noatime,nodiratime,data=writeback,commit=120)
xfs (un-tweaked): 100 Minutes (space-usage equal to reiserfs) (mount -o noatime,nodiratime,logbufs=8)
xfs (tweaked): 21-22 Minutes (mount -o logbsize=256k,nobarrier,logbufs=8,allocsize=512m)

xfs tweak-options (20 Gb-volume):
mkfs.xfs -l size=128m,version=2,lazy-count=1 -d agcount=20 -i attr=2 -n size=16k
mount -o logbsize=256k,nobarrier,logbufs=8,allocsize=512m


all in all xfs seems to be a nice filesystem, will keep it for a while ... :)

dunno why the defaults during fs-creation are that bad, it doesn't have the abysmal performance a lot people pretend it to have ...
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Sun Jul 06, 2008 3:41 pm    Post subject: Reply with quote

and how fast would be a tweaked reiserfs?
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Jul 06, 2008 3:52 pm    Post subject: Reply with quote

energyman76b wrote:
and how fast would be a tweaked reiserfs?


still faster than an über-tweaked xfs :wink:

it took 20 minutes to extract the system on reiserfs; 22 minutes on that tweaked xfs

I don't know how fast xfs really is, but I'll see in the following weeks :idea:

reiserfs so far always has been faster than ext3, xfs or jfs :P (at least for me)

if you want numbers you have to do the benchmarking on your own :wink: - I need this puppy for work & feel comfy (still!) with xfs
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Sun Jul 06, 2008 4:00 pm    Post subject: Reply with quote

kernelOfTruth wrote:
energyman76b wrote:
and how fast would be a tweaked reiserfs?


still faster than an über-tweaked xfs :wink:

it took 20 minutes to extract the system on reiserfs; 22 minutes on that tweaked xfs

I don't know how fast xfs really is, but I'll see in the following weeks :idea:

reiserfs so far always has been faster than ext3, xfs or jfs :P (at least for me)

if you want numbers you have to do the benchmarking on your own :wink: - I need this puppy for work & feel comfy (still!) with xfs


I did ;)

but it is nice to see other results ;)
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Jul 06, 2008 5:21 pm    Post subject: Reply with quote

just found an interesting article on reiser4 and how it functions "under the hood", unfortunately it's only available in German:

http://www.linux-magazin.de/layout/set/print/content/view/full/4459

you can use google-translate to read it in english:

translated article: Fourth Symphony by Reiser

this should explain why reiser4 was developed newly from the ground and is superior to reiserfs v3.6 :idea: (especially why it's much more silent during it's file-operations on harddrives)

xattr + quotas plugins development FTW ! :P

we really need some dudes programming the xattr + quotas plugins, cleaning up the code a little and programming the repacker, then it should be ready for mainline-inclusion 8) (... yeah, this won't happen so fast ... :( )
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Sun Jul 06, 2008 5:53 pm    Post subject: Reply with quote

kernelOfTruth wrote:
just found an interesting article on reiser4 and how it functions "under the hood", unfortunately it's only available in German:

http://www.linux-magazin.de/layout/set/print/content/view/full/4459

you can use google-translate to read it in english:

translated article: Fourth Symphony by Reiser

this should explain why reiser4 was developed newly from the ground and is superior to reiserfs v3.6 :idea: (especially why it's much more silent during it's file-operations on harddrives)

xattr + quotas plugins development FTW ! :P

we really need some dudes programming the xattr + quotas plugins, cleaning up the code a little and programming the repacker, then it should be ready for mainline-inclusion 8) (... yeah, this won't happen so fast ... :( )


Edward has a list of 'issues' that have to be fixed first. Of course - if not for Andrew Morton the same people who attacked reiser4 for its 'unclean' structure would have done the same with ext4. And xfs has still hooks for proprietary, outside code (cxfs). But reiser4 is evil. Yeah ....
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Jul 06, 2008 6:34 pm    Post subject: Reply with quote

energyman76b wrote:


Edward has a list of 'issues' that have to be fixed first. Of course - if not for Andrew Morton the same people who attacked reiser4 for its 'unclean' structure would have done the same with ext4. And xfs has still hooks for proprietary, outside code (cxfs). But reiser4 is evil. Yeah ....


WTF ? 8O :(

they prefer evil code with hooks for proprietary stuff instead of revolutionary new opensource code ? :evil:

now THAT's some really schizophrenic behavior :? :lol:
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sun Jul 06, 2008 7:14 pm    Post subject: Reply with quote

Just tried a different kind of test, I copied a 1gb file to a tmpfs volume, thinking I'd copy that to the device in one test, and then copy it from the device to the device.

Just having that mounted (it turned out to 800mb, I got 2gb memory in this computer right now) dropped my performance on both xfs and ext3 quite severly.

Simply unmounting the tmpfs brings the performance back up again. I do "sync; echo 3 > /proc/sys/vm/drop_caches;" between each test, and I can see differences between tests, is a simple dd really that memory hungry? Whats going on here?

xfs from:
WRITE : 285 MB/s
READ1 : 281 MB/s
READ2 : 278 MB/s
to:
WRITE : 238 MB/s
READ1 : 146 MB/s
READ2 : 161 MB/s


ext3 from:
WRITE : 205 MB/s
READ1 : 262 MB/s
READ2 : 262 MB/s
to:
WRITE : 176 MB/s
READ1 : 158 MB/s
READ2 : 164 MB/s
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Sun Jul 06, 2008 7:43 pm    Post subject: Reply with quote

seems that your 'test' doesn't make sure that the stuff hits the platter and (partially) tests buffer performance ;)
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sun Jul 06, 2008 8:03 pm    Post subject: Reply with quote

energyman76b wrote:
seems that your 'test' doesn't make sure that the stuff hits the platter and (partially) tests buffer performance ;)


Apperantly so, not a big deal for me either way though, as it should be consistently just as wrong in every test :p

I'll have to do some proper tests as well, not sure what kind of read/write performance I should get with 5 drives in raid5, but 161mb/sec read speed seem pretty low to me.

Hm, its odd that the test would affect read speeds, as the cache is dropped between each test I'd think that couldn't be affected at all :/
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Tue Jul 08, 2008 6:49 am    Post subject: Reply with quote

Couldn't go with an agcount of 4, because it misaligned something on the real raid, went with default, which was 32 (after some small tests, but I couldn't do very accurate tests, because I had 4 raid disks in raid5 with a missing parity drive), after taking the backup off the missing drive, bonnie dropped from 270mb/sec read/write to 160mb/sec write, 215mb/sec read :(

I did my latest bonnie tests on the testing raid booting with mem=512, and 2gb test files, so it should have been very accurate. No clue why the real thing is so much slower :/, I'd expect some change in performance, but I was hoping for statistical noise only.

When I checked my logs, going from agcount=4 to agcount=16 dropped performance by around 7-8mb/sec on both read/write, but latency went down considerably. I dont think I should be seing a drop this big :/

//edit, ARGH, something is seriously wrong here. Doing copies now I see iowait at around 2-3%, a ton of kcryptd processes, all around 10%, and rsync on around 20%. Nothing is even close to maxing out io or cpu, yet the transfer speed is crap :/
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Tue Jul 08, 2008 6:16 pm    Post subject: Reply with quote

A friend of mine has offered to loan me enough harddrives to backup the full volume, so I figure now I'll do some testing again, and this time I'll do it right.

Gonna run bonnie overnight, booting a kernel with mem=2-300m, crond disabled etc. Also gonna get some graphs done, not entirely sure how, as I'll be doing a LOT of tests, I'll probably split it up based on md chunksize/encryption etc.

mkfs:
I'll do tests with these options, everything will always be mounted noatime,nodiratime:
ext3 stock
ext3 -b 4096
xfs stock
xfs -l internal,size=128m -d agcount=2 4 8 16 32 and 40 atleast
xfs -l internal,size=128m /dev/md2 - For automatic agcount
xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 with sunit=X for logging parameter, to stripe the log (although I'm not entirely sure how that works yet)
reiserfs stock
jfs stock
reiser4 stock
reiser4 -o create=ccreg40,compress=lzo1

mounts:
ext3 mounts will be done with:
noatime,nodiratime,commit=60
noatime,nodiratime

xfs mounts will be done with:
noatime,nodiratime,logbufs=8,logbsize=256k
noatime,nodiratime,logbufs=8
noatime,nodiratime

Raid:
Will vary chunksize 64,128,1024

Encryption:
Will always have twofish-lrw-benbi:sha256 256bit key under the raid, building the raid on top of it, it should be the fastest cipher. Will also do tests with RAW access, ie without encryption at all.
echo "8192" > /sys/block/md0/md/stripe_cache_size

IO Schedulers:
Really unsure here, will try with cfq and deadline atleast, probably my pre tweaked anticipatory that I got elsewhere aswell.

I'll also do tests with these kernel patches (if I can get them applied properly to a recent kernel):
http://lwn.net/Articles/249450/
http://lwn.net/Articles/242559/

Bonnie:
I'll probably base the whole thing around bonnie, but use different sizes (one generic size for the full test, but different sizes for the main filesystem test). REALLY unsure what options to run with here, I've never toyed much with bonnie, pretty much just ran with -S<twice my ram> :P

How to present all this and how to script something to do all the tests overnight is still on my todo list :p, also not entirely sure how to present the data properly. If anyone has any suggestions on additional tests to do, or any suggestions on kernel patches etc, feel free :). Due to time issues I will not run every filesystem test on every different raid chunksize, I'll run one batch test first of all different filesystems on the raid configuration I believe will be the fastest, and then I'll run with the best filesystem optimizations on the different chunk sizes most likely.


Last edited by neuron on Tue Jul 08, 2008 6:52 pm; edited 2 times in total
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Tue Jul 08, 2008 6:31 pm    Post subject: Reply with quote

you're doing really an extensive test here, neuron, for the sake of completeness & also on-topic:

could you add reiser4 to this set ? reiser4 default and reiser4 w. cryptcompress plugin (lzo) would be interesting to know, too

that way we'll have some kind of general overview for all filesystems available for linux :idea: :)

the fs-creation options for r4 are:

Code:
mkfs.reiser4
(default)

Code:
mkfs.reiser4 -o create=ccreg40,compress=lzo1
(with lzo1 compression, on an advanced raid the default's probably better)

many thanks in advance :)

every filesystem should be mounted with at least:
Code:
noatime,nodiratime


bear in mind that reiser4 has extents & write barrier support enabled by default and could still be optimized even more (code-wise), atm it's doing 2 writes on commit whereas in lots of cases that could be reduced to 1 write
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Tue Jul 08, 2008 6:51 pm    Post subject: Reply with quote

kernelOfTruth wrote:
you're doing really an extensive test here, neuron, for the sake of completeness & also on-topic:

could you add reiser4 to this set ? reiser4 default and reiser4 w. cryptcompress plugin (lzo) would be interesting to know, too


Can't believe I didn't think of that one, yeah, I'll add reiser4 both with and without cryptcompress.
Back to top
View user's profile Send private message
dusanc
Apprentice
Apprentice


Joined: 19 Sep 2005
Posts: 248
Location: Serbia

PostPosted: Tue Jul 08, 2008 9:00 pm    Post subject: Reply with quote

kernelOfTruth wrote:

...
every filesystem should be mounted with at least:
Code:
noatime,nodiratime


nodiratime is a subset of noatime
Quote:

bear in mind that reiser4 has extents & write barrier support enabled by default and could still be optimized even more (code-wise), atm it's doing 2 writes on commit whereas in lots of cases that could be reduced to 1 write

Could you clarify it please?

@neuron
And about benchmarks, well some people would say that bonnie benchmark is only worth for people that only run bonnie and nothing else :)
So maybe try to reproduce some of your real life needs, and measure which fs is the best, besides running bonnie :)
btw do you mean bonie++?
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Tue Jul 08, 2008 9:06 pm    Post subject: Reply with quote

dusanc wrote:
@neuron
And about benchmarks, well some people would say that bonnie benchmark is only worth for people that only run bonnie and nothing else :)
So maybe try to reproduce some of your real life needs, and measure which fs is the best, besides running bonnie :)
btw do you mean bonie++?


bonnie++ is an improved version of bonnie, it's been around for a while.

And I'm open for suggestions when it comes to testing methods, I haven't started testing yet, and I wont until I'm 100% ready to let this run for a while. Having a hard time deciding what to use to graph this data as well, so I'd love some suggestions there too.
Back to top
View user's profile Send private message
dusanc
Apprentice
Apprentice


Joined: 19 Sep 2005
Posts: 248
Location: Serbia

PostPosted: Tue Jul 08, 2008 9:18 pm    Post subject: Reply with quote

Well I ran onto this recent test:
http://marc.info/?l=linux-kernel&m=121484256609183&w=2
And some directions from fs devel for one way of benchmarking:
1. a machine with >= 512M RAM and "/" formatted with some fs
supported by old kernels.
2. a spare partition.
3. enable ramfs (it seems it is enabled by default in most distros).
4. put a tarball-to-copy in some working directory (I had
linux-2.6.9.tar.gz)

Note, that some old -mm kernels are not compilable/bootable
(if so, pull the "hotfixes" patches from akpm's directory on kernel.org)

5. edit the attached patches in accordance with your configurations
6. build and boot the testing kernel with reiser4 debug disabled
(I think no needs to boot in single mode, or discard kde, etc..)
7. run "vmstat 2"
8. run ./prepare_copy.sh && ./ncopy on another console

And here's the files:
prepare_copy.sh
Code:
#!/bin/sh

mount -t ramfs none /ram
cd /ram
tar -zxf ~/linux-2.6.9.tar.gz
cd -

ncopy
Code:
#! /bin/bash

function prepare
{
    mkfs.reiser4 -y /dev/hdb7
    mount -t reiser4 /dev/hdb7 /mnt
}

function copy
{

    for x in `seq 20`
    do
        cp -R /ram /mnt/$x
   sync
    done
}

prepare
time copy
df
umount /mnt


PS. I'd set the number so that fs gets atleast 80% full. And ofcourse you can try the same test but with different files (large media - avi, small media - mp3, portage directory etc.).
Also you can find something interesting at http://www.phoronix-test-suite.com/
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Wed Jul 09, 2008 7:22 am    Post subject: Reply with quote

Hm, I want cpu stats aswell, so a copy test like that is pretty much out (and it might be a bit tricky to automate), but something that fills the partititon.... that's a good point, I can't do that on the main test, as the volume will be 2tb in size, and if I make smaller partititons to test, my data on stripe sizes and such will be invalid. I might end up with 2 tests, one testing a load of filesystems at high filesystem usage, and one testing raid sizes and such.

And phoronix-test-suite isn't a benchmark per say, which is why I didn't think about it, it's based around using benchmarks and graphing the resulting data, which I want to do, so I'll look into using it to graph my data, or atleast to see what frameworks it uses for it, you might have saved me some work there ;)
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Wed Jul 09, 2008 2:21 pm    Post subject: Reply with quote

emm,neuron, for fair results you should turn barriers on for all fs or turn them off.
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Wed Jul 09, 2008 2:37 pm    Post subject: Reply with quote

energyman76b wrote:
emm,neuron, for fair results you should turn barriers on for all fs or turn them off.


It'll all be on the raid with cryptsetup, so it'll be off for everything.
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Wed Jul 09, 2008 2:49 pm    Post subject: Reply with quote

neuron wrote:
energyman76b wrote:
emm,neuron, for fair results you should turn barriers on for all fs or turn them off.


It'll all be on the raid with cryptsetup, so it'll be off for everything.


afair it just makes barriers not 'get through' but still slow down everybody using them.

You could run ext3 with barriers on/off and compare results.
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Wed Jul 09, 2008 4:05 pm    Post subject: Reply with quote

energyman76b wrote:
neuron wrote:
energyman76b wrote:
emm,neuron, for fair results you should turn barriers on for all fs or turn them off.


It'll all be on the raid with cryptsetup, so it'll be off for everything.


afair it just makes barriers not 'get through' but still slow down everybody using them.

You could run ext3 with barriers on/off and compare results.


will do
Back to top
View user's profile Send private message
dusanc
Apprentice
Apprentice


Joined: 19 Sep 2005
Posts: 248
Location: Serbia

PostPosted: Thu Jul 10, 2008 3:57 pm    Post subject: Reply with quote

Here's some more recent nice benchmarks http://tservice.net.ru/~s0mbre/old/?section=projects&item=fs_contest2
dbench, iozone, postmark etc.
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2045
Location: Germany

PostPosted: Thu Jul 10, 2008 4:12 pm    Post subject: Reply with quote

except he 'optimized' xfs some, reiserfs/reiser4 not at all and got extX away without barriers - while all other fs use barriers.

So the results are in favour of extX. You can reduce the performance of them by 30%.

Also, instead of the 'reiser4 was taken from mm' crap he should have used the correct reiser4 patch that you can find on kernel.org.
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
dusanc
Apprentice
Apprentice


Joined: 19 Sep 2005
Posts: 248
Location: Serbia

PostPosted: Thu Jul 10, 2008 8:47 pm    Post subject: Reply with quote

energyman76b wrote:
except he 'optimized' xfs some, reiserfs/reiser4 not at all and got extX away without barriers - while all other fs use barriers.

So the results are in favour of extX. You can reduce the performance of them by 30%.

Also, instead of the 'reiser4 was taken from mm' crap he should have used the correct reiser4 patch that you can find on kernel.org.

Well AFAIK the benchmark was done before reiser4.patch could be found on kernel.org
Other conclusions are true, but atleast it was documented :)
Btw. here's explanation on dbench reiser4 problems from Edward:
"I know about it: such hangs take place since 2.6.23-mm1 (or so):
I think that something got changed in the vm subsystem"
And about file creation results:
"I think this is because of poor reiser4_sync() performance: it does a
lot of work,
and that "creation" benchmark calls sync() for every created file, i.e.
100000 times.."
I hope this helps.

Dushan
PS. Is it just me or reiser4 made some good results in this "unfair" competition?
Back to top
View user's profile Send private message
Display posts from previous:   
This topic is locked: you cannot edit posts or make replies.    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2, 3 ... , 29, 30, 31  Next
Page 30 of 31

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum