Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
New filesystem: Btrfs! (cont.)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 8, 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
lkraav
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2004
Posts: 129
Location: Estonia

PostPosted: Tue Jun 28, 2011 10:56 pm    Post subject: Reply with quote

i would like to note down here that 2.6.38.5-zen+ started to randomly black screen freeze on me after a few months of usage. none of the freezes had a readable kernel bug output and nothing ever made it syslog, but i suspect btrfs since it had crashed with visible output earlier.

going to 2.6.39-pf2 has behaved rock solid now for several weeks with no changes in kernel configuration. setup below:

Code:

/dev/mapper/luks-id1 / btrfs noatime,nodiratime,compress,ssd,device=/dev/mapper/luks-id1,device=/dev/mapper/luks-id2                                                                                           0 1
LABEL=HOME              /home           btrfs           noatime,nodiratime,compress,ssd,device=/dev/mapper/c3,device=/dev/mapper/c4     0 2
LABEL=SSDPUB            /pub            btrfs           noatime,nodiratime,ssd
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Wed Jun 29, 2011 12:57 am    Post subject: Reply with quote

I had also been running into silent deadlock issues in recent kernels (mostly 2.6.39.x).

I can't say it's the same issue you're encountering, but the symptoms were similar. The Btrfs volume would silently deadlock without issuing any errors to the log.

I had a test case that could replicate the problem consistently, and the problem has cleared up with the recent 3.0_rc commits. But I'm not sure exactly which patch addressed the issue.

I'm now using a 2.6.39.2 kernel with the latest btrfs commits merged in using git, and haven't had any more deadlock issues.

This is a really good time to use Btrfs only in places where you are robust to failure.
Back to top
View user's profile Send private message
lkraav
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2004
Posts: 129
Location: Estonia

PostPosted: Wed Jun 29, 2011 2:54 am    Post subject: Reply with quote

Dont Panic wrote:
I had also been running into silent deadlock issues in recent kernels (mostly 2.6.39.x).
I can't say it's the same issue you're encountering, but the symptoms were similar. The Btrfs volume would silently deadlock without issuing any errors to the log.
I had a test case that could replicate the problem consistently, and the problem has cleared up with the recent 3.0_rc commits. But I'm not sure exactly which patch addressed the issue.
I'm now using a 2.6.39.2 kernel with the latest btrfs commits merged in using git, and haven't had any more deadlock issues.
This is a really good time to use Btrfs only in places where you are robust to failure.


Hmmm, what mount parameters do you use? I mean doing stuff like ssd,compress etc like I am definitely triggers extra code right? I am also using BFS for scheduler.
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Wed Jun 29, 2011 3:19 am    Post subject: Reply with quote

lkraav wrote:
Hmmm, what mount parameters do you use? I mean doing stuff like ssd,compress etc like I am definitely triggers extra code right? I am also using BFS for scheduler.

I nearly always use either 'compress-force' or 'compress-force=lzo'. I've started trying to work 'autodefrag' in when I'm playing the the 3.0_rc code (mostly just checking for stability right now).

I've never used the 'ssd' option. I sometime use 'space_cache', but my benchmarks usually find 'space_cache' to be of marginal benefit, so I only mount with 'space_cache' when I want to test it's stability. Of course it's always on once you mount with it once, but I have several partitions that just never see 'space_cache'.

My kernels include a few options like aufs and fbcondecor, but I haven't used BFS in quite a while. Aside from that, my kernels are pretty plain.

Now that you mention it, lock-ups where one of the reasons I quit playing with BFS. But that was a while ago. I'm sure it's much more reliable now. :)
Back to top
View user's profile Send private message
lkraav
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2004
Posts: 129
Location: Estonia

PostPosted: Wed Jun 29, 2011 3:53 am    Post subject: Reply with quote

Dont Panic wrote:

Now that you mention it, lock-ups where one of the reasons I quit playing with BFS. But that was a while ago. I'm sure it's much more reliable now. :)


CK actually had quite a bit of problems initially with 2.6.39, random lockups also, it's all documented in his blog. But by BFS v406 he figured it out with a bunch of people helping test and I've been running with that with no apparent issues at all.
Back to top
View user's profile Send private message
BlueFusion
Guru
Guru


Joined: 08 Mar 2006
Posts: 371

PostPosted: Thu Aug 04, 2011 12:04 am    Post subject: Reply with quote

Does anybody else ever experience the system locking up while periodically with intense disk seeking? I am running one desktop with btrfs as my rootfs. It is RAID 0 across 2 160GB 7200RPM drives. Both drives are in good condition according to smartctl.

It is worst when I open Firefox and pages load. It will freeze for 10-15 seconds and then resume operation. Each time this happens, everything on the screen with the exception of the mouse freezes and the disk activity light is on. I used dstat to see how heavy the disk I/O is during this time and the read/write is just in the kilobytes the entire time, so it must be a massively slow seek.

I tried defragmenting the filesystem multiple times using find / -type f -exec btrfs file defrag {} \; which did not reduce this occurrence. It is getting worse over time.

I do run torrent software on here 24/7 which likely causes a lot of fragmentation issues but I would have expected it to go away once I defragement the filesystem. The filesystem is just 2/3 filled at 65% used (103GB free) so there should be plenty of space to find contiguous blocks.

I am wondering if the space_cache option is causing this. None of my other systems which only use btrfs as my portage subvolume and other volatile storage subvolumes are experiencing any speed problems.

Thoughts?

Code:
phoenix ~ # uname -a
Linux phoenix.richgannon.net 2.6.39-gentoo-r3 #1 SMP PREEMPT Sun Jul 17 17:24:22 EDT 2011 i686 Intel(R) Pentium(R) D CPU 3.20GHz GenuineIntel GNU/Linux
phoenix ~ # df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                298G  187G  103G  65% /
udev                   10M  256K  9.8M   3% /dev
/dev/sda2             298G  187G  103G  65% /
rc-svcdir             1.0M   96K  928K  10% /lib/rc/init.d
shm                  1013M     0 1013M   0% /dev/shm
/dev/sda2             298G  187G  103G  65% /usr/portage
/dev/sda2             298G  187G  103G  65% /home
/dev/sda2             298G  187G  103G  65% /music
none                  600M   50M  551M   9% /tmp
phoenix ~ # dmesg | grep -i btrfs
Btrfs loaded
btrfs: disk space caching is enabled
btrfs: use zlib compression
btrfs: free space inode generation (0) did not match free space cache generation (414914) for block group 22846373888
btrfs: free space inode generation (0) did not match free space cache generation (415048) for block group 22980591616
btrfs: free space inode generation (0) did not match free space cache generation (415103) for block group 27409776640
btrfs: free space inode generation (0) did not match free space cache generation (415195) for block group 27543994368

_________________
i7-940 2.93Ghz | ASUS P6T Deluxe (v.1) | 24GB Triple Channel RAM | nVidia GTX660
4x 4TB Seagate NAS HDD (Btrfs raid5) | 2x 120GB Samsung 850 EVO SSD (Btrfs raid1)
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Fri Aug 05, 2011 1:38 pm    Post subject: Reply with quote

There's been several reports on the Mailing List and on IRC of a "btrfs slow down" occurring after a day or so (+/-) of usage.

AFAIK, there has not been much progress on identifying the source of this issue.

If you're running a 3.0 kernel, it might be interesting to mount with the 'autodefrag' option, and see if that helps.
Back to top
View user's profile Send private message
eric2
n00b
n00b


Joined: 23 Sep 2011
Posts: 2

PostPosted: Fri Sep 23, 2011 7:52 pm    Post subject: Reply with quote

I've read that it is possible to mount subvolumes with different mount options, but somehow I can't make this work :(

I have 2 subvolumes, and only 1 should use compression, but whatever I do, the result is always either both use compression or none use compression, depending on which subvolume is mounted first.

This is what I did:
Code:

# create mountpoints
cd /mnt
mkdir btrfs-root comp norm

# create btrfs root
btrfs.mkfs -L btrfs-root /dev/sdb1

# create subvolumes
mount /dev/sdb1 /mnt/btrfs-root -t btrfs -o defaults
cd /mnt/btrfs-root
btrfs subvolume create normal
btrfs subvolume create compressed
cd /mnt
umount /mnt/btrfs-root

# mount subvolumes
mount /dev/sdb1 /mnt/norm -t btrfs -o defaults,subvol=normal
mount /dev/sdb1 /mnt/comp -t btrfs -o defaults,subvol=compressed,compress


I've checked the compression by copying a big text file from outside the partition in to the subvolumes and execute a 'sync' and 'df' command to see the space used. I've tried with 'compress' and 'compress-force' also with '=zlib' or '=lzo' with the same result.

Am I doing something wrong?


Last edited by eric2 on Tue Sep 27, 2011 10:42 pm; edited 1 time in total
Back to top
View user's profile Send private message
eric2
n00b
n00b


Joined: 23 Sep 2011
Posts: 2

PostPosted: Tue Sep 27, 2011 10:41 pm    Post subject: Reply with quote

It looks like per subvolume compression is not possible:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg05552.html
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg09885.html

But I think they are working on it:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg12437.html
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sat Oct 08, 2011 7:37 am    Post subject: Hey, any BTRFS experts out there? Reply with quote

I have a testing system running BTRFS (2 device RAID1 data profile) as root filesystem. (Cheeky, I know.) It's ~amd64 v0.19-r3.

It backs up every night.

I'm getting a lot of these:
Code:
btrfs bad tree block start 0 177481867264
btrfs bad tree block start 0 177481867264
btrfs bad tree block start 18446462603027742720 177476366336
btrfs bad tree block start 18446462603027742720 177476366336
btrfs bad tree block start 0 177476386816
btrfs bad tree block start 0 177476386816
btrfs bad tree block start 0 177482096640
btrfs bad tree block start 0 177482096640
btrfs bad tree block start 0 177482096640
btrfs bad tree block start 0 177482096640
btrfs bad tree block start 0 177482104832
btrfs bad tree block start 0 177482104832
btrfs bad tree block start 0 177482100736
btrfs bad tree block start 0 177482100736
btrfs bad tree block start 0 177482108928
btrfs bad tree block start 0 177482108928
btrfs bad tree block start 0 177482121216
btrfs bad tree block start 0 177482121216
btree_readpage_end_io_hook: 8 callbacks suppressed
btrfs bad tree block start 0 177477054464
btrfs bad tree block start 0 177477054464
btrfs bad tree block start 0 177477062656
btrfs bad tree block start 0 177477062656
btrfs bad tree block start 0 177477046272
btrfs bad tree block start 0 177477046272
btrfs bad tree block start 0 177477058560
btrfs bad tree block start 0 177477058560
btrfs bad tree block start 0 177477066752
btrfs bad tree block start 0 177477066752
btree_readpage_end_io_hook: 29 callbacks suppressed
btrfs bad tree block start 0 177476694016
btrfs bad tree block start 0 177476694016
btrfs bad tree block start 0 177476689920
btrfs bad tree block start 0 177476689920
btrfs bad tree block start 0 177479421952
btrfs bad tree block start 0 177479421952
btrfs bad tree block start 0 177479438336
btrfs bad tree block start 0 177479471104
btrfs bad tree block start 0 177479426048
btrfs bad tree block start 0 177479475200
btrfs bad tree block start 0 177479479296
btrfs bad tree block start 0 177479483392
btrfs bad tree block start 0 177479487488
btrfs bad tree block start 0 177479426048
btrfs bad tree block start 0 177479438336
btrfs bad tree block start 0 177479438336
btree_readpage_end_io_hook: 16 callbacks suppressed
btrfs bad tree block start 0 177479311360
btrfs bad tree block start 0 177479311360
btrfs bad tree block start 0 177476644864
btrfs bad tree block start 0 177476644864
btrfs bad tree block start 0 177476632576
btrfs bad tree block start 0 177476632576

from DMESG during the backups.

Like I said, this is a 'just for fun' testing scenario because I really like the functional concept of BTRFS.

I'm not finding much around about BTRFS lately and the kernel.org crash isn't helping.

Do we know when we might have BTRFS repair tools? Finally, does anyone know where a developer thread is running?
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Sat Oct 08, 2011 7:39 pm    Post subject: Reply with quote

The Btrfs Mailing List is very active: http://vger.kernel.org/vger-lists.html#linux-btrfs

Also, The #btrfs IRC channel on freenode is very active.

The Btrfs developers are usually monitoring both.

Having said that, I would warn against bringing up btrfsck, unless you are irate and in the mood to start a flame war.

The lead developer who is working on btrfsck has made the mistake of repeatedly making promises with respect to btrfsck, and finding themselves unable to deliver on those promises.

While I believe the people working on btrfsck are putting in their best good-faith effort, the mood among a vocal set of affected users is beginning to turn ugly.

While many btrfs errors can be ignored, there is very little guidance to assist in assessing the severity of errors.

My recommendation is to back-up and restore to a freshly formatted partition.

If you can replicate your issue, please post to the Mailing List. The developers are pretty responsive when presented with issues that can be replicated.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sat Oct 08, 2011 7:50 pm    Post subject: Reply with quote

People still use BTRFS?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Sun Oct 09, 2011 2:14 am    Post subject: Reply with quote

devsk wrote:
People still use BTRFS?
What has come along to replace it?
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sun Oct 09, 2011 3:19 am    Post subject: Reply with quote

pjp wrote:
devsk wrote:
People still use BTRFS?
What has come along to replace it?
ext4 for rootfs and ZFS for everything else! ext4 is faster than BTRFS. And ZFS is better than BTRFS in every possible way!
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Sun Oct 09, 2011 4:00 am    Post subject: Reply with quote

afaik, ext4 doesn't offer any zfs/btrfs-like features?

Are you referring to zfs-fuse? I thought that was mainly for testing purposes.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sun Oct 09, 2011 5:27 am    Post subject: Reply with quote

pjp wrote:
afaik, ext4 doesn't offer any zfs/btrfs-like features?

Are you referring to zfs-fuse? I thought that was mainly for testing purposes.
Who needs advanced features for rootfs anyway? I need a stable fast rootfs and ext4 fits the bill.

No, I was referring to Native ZFS on Linux: http://zfsonlinux.org/

I have been running it for many months now without issues, something I could not say about BTRFS.
Back to top
View user's profile Send private message
jormartr
Apprentice
Apprentice


Joined: 02 Jan 2008
Posts: 174

PostPosted: Sun Oct 09, 2011 9:01 am    Post subject: Reply with quote

This is interesting.
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sun Oct 09, 2011 11:00 am    Post subject: BTRFS vs. ZFS vs. EXT4 Reply with quote

devsk wrote:
People still use BTRFS?

Is it a little chilly in here?

BTRFS has support in kernel and portage — ZFS doesn't.

BTRFS has fundamental native application of RAID0 / RAID1 — EXT4 doesn't.

How does one determine that EXT4 is faster than BTRFS in native RAID0 or RAID1 when a fundamental or native RAID on EXT4 is a non sequitur?
Code:
testserve # grep -i btrfs /usr/src/linux-3.0.6-gentoo/.config
CONFIG_BTRFS_FS=y

testserve / # grep -i zfs /usr/src/linux-3.0.6-gentoo/.config
testserve / #

I don't believe it's valid to operate a vital server root instance in a non-redundant environment.
ZFS cannot be supported by the linux kernel due to mutually exclusive licenses.
I'm operating test instances of BTRFS root filesystems with RAID1 data to optimistically promote such filesystems.
ZFS on Linux is interesting, however it doesn't seem to have a viable terminus as it can't be included in a non-patched Linux kernel, yet — if it depends on lawyers, I wouldn't wait around. I cannot say where Oracle is headed.
This test server hosts no vital or non-redundant data and backs up to an EXT4 partition on another hard drive.
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sun Oct 09, 2011 3:20 pm    Post subject: Re: BTRFS vs. ZFS vs. EXT4 Reply with quote

splurben wrote:
devsk wrote:
People still use BTRFS?

Is it a little chilly in here?

BTRFS has support in kernel and portage — ZFS doesn't.

BTRFS has fundamental native application of RAID0 / RAID1 — EXT4 doesn't.

How does one determine that EXT4 is faster than BTRFS in native RAID0 or RAID1 when a fundamental or native RAID on EXT4 is a non sequitur?
Code:
testserve # grep -i btrfs /usr/src/linux-3.0.6-gentoo/.config
CONFIG_BTRFS_FS=y

testserve / # grep -i zfs /usr/src/linux-3.0.6-gentoo/.config
testserve / #

I don't believe it's valid to operate a vital server root instance in a non-redundant environment.
ZFS cannot be supported by the linux kernel due to mutually exclusive licenses.
I'm operating test instances of BTRFS root filesystems with RAID1 data to optimistically promote such filesystems.
ZFS on Linux is interesting, however it doesn't seem to have a viable terminus as it can't be included in a non-patched Linux kernel, yet — if it depends on lawyers, I wouldn't wait around. I cannot say where Oracle is headed.
This test server hosts no vital or non-redundant data and backs up to an EXT4 partition on another hard drive.
:D:D

I burnt a BTRFS fanboi it looks like...:D My apologies for your lost data and time! I was fanboi as well. Read the thread back enough and you will find me burning as well...:D

Talk about what it is currently and not what it will become! ext4 is NOW and happening! ZFS is NOW and happening. And both are much more stable and usable than BTRFS.

PS: btw, I never claimed ext4 had native RAID0 and RAID1. When you have seen life long enough, you will realize that BTRFS's native RAID is nothing compared to ZFS's. Go, get some reading kid!
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Sun Oct 09, 2011 10:36 pm    Post subject: Reply with quote

devsk wrote:
Who needs advanced features for rootfs anyway? I need a stable fast rootfs and ext4 fits the bill.
Who needs more than 640k of RAM, etc., etc. I see no reason to use ext4 over 2 or 3. ZFS does offer some helpful features (which I hope are in or planned to be in btrfs).

devsk wrote:
No, I was referring to Native ZFS on Linux: http://zfsonlinux.org/
The only thing in the portage tree is fuse. I did find that site, but it didn't seem ready: 0.5.2 stable release does not yet support a mountable filesystem. I assume a filesystem is only useful if it is mountable? Plus, I don't see any information indicating when the stable and current versions were put out, which made it seem somewhat abandoned.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sun Oct 09, 2011 10:47 pm    Post subject: BTRFS Fanboi Twink Reply with quote

No data lost, no worries. Why so aggressive devsk?
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sun Oct 09, 2011 11:02 pm    Post subject: Reply with quote

pjp wrote:
devsk wrote:
Who needs advanced features for rootfs anyway? I need a stable fast rootfs and ext4 fits the bill.
Who needs more than 640k of RAM, etc., etc. I see no reason to use ext4 over 2 or 3. ZFS does offer some helpful features (which I hope are in or planned to be in btrfs).

devsk wrote:
No, I was referring to Native ZFS on Linux: http://zfsonlinux.org/
The only thing in the portage tree is fuse. I did find that site, but it didn't seem ready: 0.5.2 stable release does not yet support a mountable filesystem. I assume a filesystem is only useful if it is mountable? Plus, I don't see any information indicating when the stable and current versions were put out, which made it seem somewhat abandoned.


Cheers to pjp for a positive outlook and constructive commentary.
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Mon Oct 10, 2011 12:03 am    Post subject: Reply with quote

jormartr wrote:
This is interesting.
Indeed. Thanks for posting. 4-6 months is unfortunate, but at least it is recent info. Hopefully the progress will be significant.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Mon Oct 10, 2011 12:05 am    Post subject: Re: BTRFS Fanboi Twink Reply with quote

splurben wrote:
Why so aggressive devsk?
Because BTRFS burnt me badly and there was no way to deal with the issues. BTRFS remains the only FS to which I have lost data to. Simple test cases were generated and bug filed. Nobody cared! And developers looked clueless to me!

pjp: Pls give ZFS on Linux a try! When did Gentoo folks start becoming scared of something which is marked as RC5 or not in portage? ZFS is miles more stable than BTRFS on Linux. I have been running it as my main media and backup server for many months, and I love it!
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Mon Oct 10, 2011 12:40 am    Post subject: Reply with quote

btrfs remains under heavy development and is not yet intended for use where data loss is a problem or inconvenience. Using it at this point seems to be acceptance of such issues.

Anyway, as for trying ZFS, I've used ~arch way back when ('03 - '04) and had problems. I've decided to never do that again, so I only use stable. I'm also not interested in maintaining random overlays separately from portage (I find it to be a nuisance), and even less interested in maintaining custom ebuilds, especially of the critical variety. None of those scenarios are why I started using Gentoo, or continue using it.

Also, since licensing is an issue, it seems unlikely to ever get into the kernel, never mind gentoo-hardened*.


* Though if you start maintaining hardened with containers & ZFS, feel free to PM me! :D
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2, 3 ... 8, 9, 10, 11  Next
Page 9 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum