Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] /usr read-only file system
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Fri Aug 10, 2012 10:15 pm    Post subject: [SOLVED] /usr read-only file system Reply with quote

I ran into a big problem, so I'm hoping that someone can help me out.

I was updating my system and in the process there was a power cut (although it's a laptop, I didn't have the battery in). When I restarted, I found out that I cannot start X. So I decided to emerge it again, however it was unsuccessful. Upon investigation, it turned out that /usr, which is a separate partition is mounted as read only. Here's the output of a few commands:

Code:
#cat /etc/fstab
/dev/sda1   /boot   ext2   noauto,noatime 1 2
/dev/sda2   /none   swp   sw   0 0
/dev/sda5   /   ext3   noatime   0 1
/dev/sda6   /tmp   ext3   noatime   0 2
/dev/sda7   /var   ext3   noatime   0 2
/dev/sda8   /usr   ext3   noatime   0 2
/dev/sda9   /home   ext3   noatime 0 2


The permissions of the /usr folder look fine

Code:
# ls -l
drwxr-xr-x 14 root root 4096 Aug 10 22:22 usr


however I can't even create a test folder in /usr

Code:
#mkdir testfolder
mkdir: cannot create directory "testfolder": Read-only file system


So I'm really stuck because I can't run X and I can't really install any packages.

I found a few sources online, which suggest to use fsck after unmounting the partition. However a simple
Code:
# umount /usr
does not work, because it complains that the file system is in use. Unfortunately I don't have lsof installed so I tried killing all processes linked to that file system using
Code:
# fuser -km /usr

Unfortunately that logs me out and brings me back to the login prompt, so I cannot run fsck.

Any suggestions how to make /usr writable again? I'm really frustrated. Thanks.


Last edited by jopeto on Sun Aug 12, 2012 6:49 am; edited 4 times in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Fri Aug 10, 2012 10:22 pm    Post subject: Reply with quote

You could remount it read-write, but I suggest that you not do that until you understand why it mounted read-only. What is the output of dmesg?
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Fri Aug 10, 2012 10:25 pm    Post subject: Reply with quote

https://forums.gentoo.org/viewtopic-p-6863584.html#6863584

and you might wish have a look after to this one : https://forums.gentoo.org/viewtopic-t-901206.html
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Fri Aug 10, 2012 10:35 pm    Post subject: Reply with quote

Thanks for the reply. Unfortunately I'm writing from a different computer now and I don't have a usb stick to transfer the entire output of the dmesg command but the lines which look relevant seem to be the following:
Code:
EXT-fs (sda5): recovery required on readonly filesystem
EXT-fs (sda5): write access will be enabled during recovery
kjournald starting. Commit interval 5 seconds
EXT-fs (sda5): recovery complete
EXT-fs (sda5): mounted filesystem with ordered data mode
kjournald starting. Commit interval 5 seconds
EXT-fs (sda8): mounted filesystem with ordered data mode
system-udevd[15255]: starting version 187
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Fri Aug 10, 2012 10:45 pm    Post subject: Reply with quote

Thanks for your pointer. But I don't understand what it meams to add S to the end of the kernel command line in grub. Is it referring to the grub.conf file? Currently my grub.conf file has the line
Code:
kernel /boot/<kernel name> real_root=/dev/sda5


Should I change this to

Code:
kernel /boot/<kernel name> real_root=/dev/sda5 S
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Fri Aug 10, 2012 11:00 pm    Post subject: Reply with quote

I guess that's what it means. I followed the steps in the last post here
https://forums.gentoo.org/viewtopic-p-6863584.html#6863584
and now I'm able to install programs. Now I just have to figure out what to do from here.
I'm guessing
Code:
emerge -e world

?
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Fri Aug 10, 2012 11:08 pm    Post subject: Reply with quote

Actually I'll try first
Code:
revdep-rebuild
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sat Aug 11, 2012 12:23 am    Post subject: Reply with quote

OK, revdep-rebuild did the job. Thanks a lot for your help!
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sat Aug 11, 2012 12:51 am    Post subject: Reply with quote

Actually I'm reopening this thread. The functionality is fine, however I noticed that on boot when all the partitions are mounted I get the message
Code:
 * Checking local filesystems  ...
/dev/sda5: clean, 3682/131072 files, 50539/524288 blocks
/dev/sda1: clean, 41/32768 files, 8021/131072 blocks
/dev/sda6: clean, 13/131072 files, 25407/524288 blocks
/dev/sda7: clean, 23819/1310720 files, 205153/5242880 blocks
/dev/sda8 is mounted.
e2fsck: Cannot continue, aborting.


/dev/sda8 is mounted.
e2fsck: Cannot continue, aborting.


 * Operational error
 [ !! ]
 * Remounting root filesystem read/write ...
 [ ok ]
 * Updating /etc/mtab ...
 [ ok ]
 * Mounting local filesystems ...
mount: /dev/sda8 already mounted or /home busy
mount: according to mtab, /dev/sda8 is mounted on /usr
 * Some local filesystem failed to mount


I'm guessing this means that /dev/sda8 is permanently mounted? Here's my /etc/mtab

Code:
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=495371,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
/dev/sda5 / ext3 rw,noatime,errors=continue,barrier=1,data=ordered 0 0
/dev/sda8 /usr ext3 rw,noatime,errors=continue,barrier=1,data=ordered 0 0
tmpfs /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup_root /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
openrc /sys/fs/cgroup/openrc cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc 0 0
cpuacct /sys/fs/cgroup/cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/sda6 /tmp ext3 rw,noatime 0 0
/dev/sda7 /var ext3 rw,noatime 0 0
/dev/sda9 /home ext3 rw,noatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0


Any help on how I can get rid of that error? Thanks.
Back to top
View user's profile Send private message
rjtupas
n00b
n00b


Joined: 26 Jul 2003
Posts: 59
Location: Coronado, CA USA

PostPosted: Sat Aug 11, 2012 3:22 am    Post subject: Reply with quote

I'm experiencing similar issues at boot (/usr mounted ro) and resulting e2fsck errors. I'm running ~amd64, sys-fs/udev-187-r3, and sys-fs/lvm2-2.02.95-r4. When I shutdown, both /usr and /tmp complain:
Code:

  device-mapper: remove ioctl on  failed: Device or resource busy
...
  Unable to deactivate data-usr (254:0)
  device-mapper: remove ioctl on  failed: Device or resource busy
...
 Unable to deactivate data-tmp (254:4)
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sat Aug 11, 2012 10:09 pm    Post subject: Reply with quote

I was thinking that the solution to my problem might be to run fsck on /usr. However to do that I need to make sure that /usr is unmounted. Unfortunately I can't seem to do that. Here's what I do. First I go into runlevel 1
Code:
# init 1

Then I try to unmount /usr
Code:
# umount /usr

However I get an error that the file system is in use and cannot be unmounted. So I try
Code:
# umount -l /usr

and that seems to work. If I do ls /usr, I don't see anything. However when I run
Code:
# fsck /usr

I get an error that the file system is in use, so fsck cannot run. When examining with lsof, I saw that login is used by /usr. So as long as I'm logged in, I cannot unmount /usr. But I can't really do anything unless I'm loggin in. Strange...

So I'm thinking that maybe the the solution would be to boot from a live CD and run fsck on the file system from it, since otherwise it seems to be in use. Any ideas whether this is the right approach (i would try it right away, but I'll have to find a CD and a computer to burn it on first).

Also any idea about the problem with the error of /usr on boot up? I've shown them before, but here's the output of some commands:

Code:
$ mount
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=495371,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
/dev/sda5 on / type ext3 (rw,noatime,errors=continue,barrier=1,data=ordered)
/dev/sda8 on /usr type ext3 (rw,noatime,errors=continue,barrier=1,data=ordered)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda6 on /tmp type ext3 (rw,noatime)
/dev/sda7 on /var type ext3 (rw,noatime)
/dev/sda9 on /home type ext3 (rw,noatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)


Code:
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          2.0G  162M  1.8G   9% /
udev             10M     0   10M   0% /dev
/dev/sda5       2.0G  162M  1.8G   9% /
/dev/sda8        20G   11G  8.7G  54% /usr
tmpfs           1.9G  356K  1.9G   1% /run
cgroup_root      10M     0   10M   0% /sys/fs/cgroup
shm             1.9G     0  1.9G   0% /dev/shm
/dev/sda6       2.0G   68M  1.9G   4% /tmp
/dev/sda7        20G  389M   19G   3% /var
/dev/sda9       409G  8.8G  380G   3% /home


Code:
$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# noatime turns off atimes for increased performance (atimes normally aren't
# needed); notail increases performance of ReiserFS (at the expense of storage
# efficiency).  It's safe to drop the noatime options if you want and to
# switch between notail / tail freely.
#
# The root filesystem should have a pass number of either 0 or 1.
# All other filesystems should have a pass number of 0 or greater than 1.
#
# See the manpage fstab(5) for more information.
#

# <fs>         <mountpoint>   <type>      <opts>      <dump/pass>

# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
/dev/sda1      /boot      ext2      noauto,noatime   1 2
/dev/sda2      none      swap      sw      0 0
/dev/sda5      /      ext3      noatime      0 1
/dev/sda6      /tmp      ext3      noatime      0 2
/dev/sda7      /var      ext3      noatime      0 2
/dev/sda8      /usr      ext3      noatime      0 2
/dev/sda9      /home      ext3      noatime      0 2


/dev/cdrom      /mnt/cdrom   auto      noauto,ro   0 0
/dev/fd0      /mnt/floppy   auto      noauto      0 0


Code:
$ sudo fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0xa8a8a8a8

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1050623      524288   83  Linux
/dev/sda2         1050624    13633535     6291456   82  Linux swap / Solaris
/dev/sda3        13633536   976773167   481569816    5  Extended
/dev/sda5        13635584    17829887     2097152   83  Linux
/dev/sda6        17831936    22026239     2097152   83  Linux
/dev/sda7        22028288    63971327    20971520   83  Linux
/dev/sda8        63973376   105916415    20971520   83  Linux
/dev/sda9       105918464   976773167   435427352   83  Linux


On thing that I've noticed is that /usr seems to be mounted right after /, while /tmp, /var, and /home are mounted afterwards. Could that be the problem or is this normal? Thanks.
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sat Aug 11, 2012 10:54 pm    Post subject: Reply with quote

I found another post with the same problem, which however has not received any replies yet:
https://forums.gentoo.org/viewtopic-t-918740-start-0.html

So I tried reinstalling initramfs with the command
Code:
# genkernel --install initramfs

but that didn't change anything.
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sun Aug 12, 2012 12:07 am    Post subject: Reply with quote

Found yet other posts regarding the same issue:
http://www.gossamer-threads.com/lists/gentoo/user/253440
https://bbs.archlinux.org/viewtopic.php?id=143025
Seems like an issue with fsck, /usr, udev and initramfs and how they all play together. I'm guessing the sudden shutdown in my case caused a problem in the /usr partition, which necessitates a check with fsck, however since it is already mounted read/write in the beginning by initramfs, fsck cannot run. I currently have my initramfs generated by genkernel, maybe I'll have to try a different method.
The second link suggests replacing the 2 in the fstab file with 0. Is that a bad idea?
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sun Aug 12, 2012 12:34 am    Post subject: Reply with quote

OK, so I can confirm that it looks like that the problem stems from the fact that initramfs mounts the /usr directory read/write, which means that fsck cannot check it. The weird thing thing is that the root directory / can be checked. So maybe the root is mounted as read only.

I replaced the last number in the /etc/fstab file with 0 as opposed to 2, so that fsck does not check that partition, and the message on boot-up disappears. I experimented by editing the /etc/initramfs.conf file so that /var is also mounted by initramfs, and I got again the same error that fsck cannot check it. When replacing the last number in the /var line in /etc/fstab with 0, the message disappeared again.

So on the basis of that observation I have two questions.

1. Is it safe to have 0 as the last number in the lines of the /usr and /var directories? That basically means that these partitions will never be checked, which I guess might be bad.
2. How to make initramfs mount these two partitions as read only originally so that they can be checked by fsck and then be remounted as read/write after the initial boot is complete?

Thanks a lot.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Sun Aug 12, 2012 12:46 am    Post subject: Reply with quote

never use separate partition for /usr but i think i would set it ro in fstab and remount it rw after boot is complete, but the "filesystem is in use" is telling you someone record something in /usr and i suppose booting it in ro will bug that program (without knowing i would put my bet on udev).

did you check the link i gave first ? where user keep using separate /usr because of udev new rules i think you won't get anywhere if you wish keeping using separate /usr with a recent udev version (and you can thank systemd / ubuntu and family crappy changes)
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sun Aug 12, 2012 1:06 am    Post subject: Reply with quote

Thanks for your reply. Yes, I did read the link, but I guess that one referred to not using initramfs at all (if I understood it correctly). I guess I will give it a try now that I understand better what the problem is.

So would you recommend merging /usr and /?
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Sun Aug 12, 2012 2:34 am    Post subject: Reply with quote

Never really use /usr because i never get the need for.
So that advise is a bit weak, but if you don't NEED /usr sadly the udev/systemd dictators are telling you it's better to have /usr in /
So except if you have a real need, you should be safer with them tied or battle against udev :/
And this is kind of battle you might win, but at end, big lost while doing it, and you know the war would still be lost at end.
Back to top
View user's profile Send private message
jopeto
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jul 2012
Posts: 106

PostPosted: Sun Aug 12, 2012 6:49 am    Post subject: Reply with quote

Thanks a lot for your suggestion. I'm going to burn a live CD in the next few days, put /usr in /, merge the two partitions and edit the fstab to reflect that change.

I'm not very experienced, so when installing I read a source that said that it is better to have /usr in a separate partition, but I guess this is not the case, at least in the present case.

Thank you once again for your help, I'll mark this thread as solved again.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Sun Aug 12, 2012 5:56 pm    Post subject: Reply with quote

Some documentation does still advise that you put /usr separate. As far as I know, the only thing that does not handle this well is udev. Unfortunately, the udev maintainers have taken the attitude that they do not want to handle it well, and have been actively making changes that rely on not having a separate /usr. Many people are quite unhappy with this, because separate /usr worked for a long time and now does not.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Sun Aug 12, 2012 6:36 pm    Post subject: Reply with quote

for now we could live without systemd, but we couldn't anymore without udev that's why i said war will be lost at end.
what is bugging me bad is that i'm seeing those buggers coming with their big feet and their systemd crap.

udev itself is now inside systemd source, they said "blahblahblah nothing will change", but i might be the dark prophet saying it but what i see is "systemd for all distros !".

if they continue like that we will end with a linux livecd and all distros will just put their own stickers on it.
Back to top
View user's profile Send private message
sk3l
Tux's lil' helper
Tux's lil' helper


Joined: 14 Jul 2012
Posts: 78
Location: CT USA

PostPosted: Mon Aug 13, 2012 4:25 pm    Post subject: Good Info Reply with quote

I'm glad to have found this thread, because it is a conversation covering a questions/concerns that I myself have been grappling with in regards to The Right Way™ of pushing ahead in the face of the udev changes. Thank you jopeto, krinn and Hu for your input.

I use an LVM configuration containing everything but root. I had read some of the documentation spoken of here, and coupled with my past experiences, I had formed an option that partitioning/logical separation of file systems was a good thing as it relates to security, back up & restore of data, and so forth. Apparently this is a sentiment that may not be shared with certain key Gentoo "upstream" folks, or else there would be more sensitivity to this approach, in terms of udev support for it.

I found myself at a crossroads similar to where jopeto is now not long ago: do I simply abandon my partitioning scheme and glom /usr back onto to /, or should I press ahead with some method of early userspace mounting for a separate /usr, e.g. initramfs, patched udev, etc.? I elected to go with the initramfs path, compliments of genkernel. Overall, everything seems to function fine, but there are some small oddities that appear to crop up during boot/shutdown, as have been detailed here by jopeto and others, and I had chronicled in a separate thread here. Now, in reading some of the thoughts proffered in this thread, I'm beginning to perhaps rethink my decision. I still regard logical separation of concerns as it relates to file organization as prudent and useful, but if this a practice that is going to put me at loggerheads with the maintainers of Gentoo, then I am willing to relent as opposed to jump through hoops to keep up that practice. As it stands, needing to rebuild an initramfs with every new kernel or alternatively rolling my own initramfs init script is a chore I'm not relishing, and were it not for the new udev direction, there's no way I would be relying on initramfs for anything.

I think I'm going to consider long and hard whether I should just merge /usr and /, and maybe even /var. It takes some work to pull off that I really didn't want to invest time in, but then again, I could (hopefully) execute it one time and be done with this udev business for good and not need to fiddle about with initramfs. The question then becomes, what is the new best practice as it relates to partitioning and using LVM? The doc still mentions separating /usr and /var, although it has been updated to mention initramfs. Is this evolving, or will it remain status-quo?

Incidentally, it seems that this push, and Gentoo is not the only distro moving this way, to reunite / and /usr is not so much moving /usr back into /, but rather migrating the contents of / to /usr and leaving behind symlinks in / for historical reference. Please correct me if I'm mistaken in my understanding.

-Mike
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Mon Aug 13, 2012 10:49 pm    Post subject: Reply with quote

Your understanding is consistent with mine. It is unfortunate that the UsrMove project would kick off right as udev support for /usr took a nosedive, since the two are independent. The UsrMove project is relevant since migrating commands to /usr may further degrade udev's ability to run properly before /usr is mounted. However, aside from my distaste at seemingly unnecessary churn and at leaving behind lots of compatibility symlinks, I see nothing wrong with UsrMove.

Separating filesystems is often touted on the basis of improving security through various mount options and on the basis of isolating untrusted users from writing to filesystems that have system content on them. In this sense, combining / and /usr is not a problem, since you tend to have the same set of options, trusted/untrusted access, and so on, for both of them. Combining / and /var is different, since /var is normally safe to make noexec, but setting that on / or /usr would render the system unusable.

For my systems where I had enough free space, I made the unpleasant choice to migrate everything from / into /usr, rename that to /, and retire the old /. This saved me from continuing to work around the udev regressions and was relatively quick since / is small. It still feels wrong to do that just to maintain functionality that once worked with separate filesystems. For the machines where I cannot readily combine filesystems, I have masked all versions of udev which break with a separate /usr. The version installed already has some regressions in that area (notably, it tries to restore sound card settings before ALSA is available), but nothing important enough to move me to more drastic action.
Back to top
View user's profile Send private message
hujuice
Guru
Guru


Joined: 16 Oct 2007
Posts: 318
Location: Rome, Italy

PostPosted: Mon Nov 11, 2013 1:37 pm    Post subject: Reply with quote

There are many threads about this topic.
If you are interested, you can read about my very simple, not complete solution: https://forums.gentoo.org/viewtopic-p-7438296.html#7438296

Regards,
HUjuice
_________________
Who hasn't a spine, should have a method.
Chi non ha carattere, deve pur avere un metodo.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum