Filesystem errors while emerging... HELP!

Message

Drysh · Post by **Drysh** » Sat Sep 09, 2006 7:26 pm

I knew I was looking for trouble... But not that much trouble.

I'm running a ~x86 system, and I decided to update to gcc-4.1 and proflie 2006.1. I started yesterday afternoon, and when I woke up this morning I had some strange errors: portage could not copy anything to the final place. Trying to solve I find some other errors:

Code: Select all

marcelo@phoenix ~ $ cp test test2
cp: skipping file `test', as it was replaced while being copied

marcelo@phoenix ~ $ rm -r /var/tmp/portage/*
rm: WARNING: Circular directory structure.
This almost certainly means that you have a corrupted file system.
NOTIFY YOUR SYSTEM MANAGER.
The following directory is part of the cycle:
  `/var/tmp/portage/baselayout-1.12.4-r7/temp/logging'

I tried to delete manualy /var/tmp because I thought it was corrupted, and I finaly managed to (using several "rm -r /var/tmp/portage/*/*/*/*/*" and removing one * at a time), but that didn't help.

What I did:

I made a spring clean using "dep -s"
"emerge --sync; emerge -DNuv system; emerge -DNuv world" to update everything
"revdep-rebuild -vv -- -vp" to check if something was broken and it said gcc-4.1.1 needed to be rebuilt (I ignored)
"gcc-config" to change to gcc-4.1.1 and "eselect profile" to change to 2006.1

I changed some.. err.. a lot of flags on my system:

Code: Select all

USE="x86 X a52 aac acl acpi alsa audiofile bash-completion berkdb bitmap-fonts bluetooth bzip2 cairo caps cdr clamav cli crypt ctrlmenu cups dbus dlloader doc dri dv dvd elibc_glibc encode exif firefox flac flash flatfile font-server fortran ftp gcj gdbm gif gimp glep gtk gtk2 gtkhtml hash iconv icq imap input_devices_keyboard input_devices_mouse ipv6 isdnlog jabber javascript jikes jpeg kernel_linux ldap libg++ linguas_en linguas_pt_BR live lm_sensors mad matrox mikmod mime mmx mmxext mng modplug mp3 mpeg msn musepack mysql mysqlfriends mysqli nls nocd nowin nptl nptlonly nsplugin offensive ogg openal opengl oscar pam panel-plugin pcre pdf perl php png posix postgres ppds pppd profile python quicktime readline real reflection rtc scanner sdl session shorten smp sndfile speex spell spl sse sse2 ssl svg svga symlink tcltk tcpd tetex tga theora threads tiff tokenizer truetype truetype-fonts type1-fonts udev unicode usb userland_GNU userlocales vcd verbose video_cards_nv videos vorbis win32codecs wma wmf wxwindows xanim xface xfs xine xml xmlreader xmlwriter xorg xosd xpm xprint xv xvid xvmc yahoo zip zlib"

CFLAGS="-O2 -pipe -march=pentium4 -mmmx -mfpmath=sse -malign-double -m128bit-long-double"

"emerge gcc"

I ran Guenther Brunthaler's script to generate the package list to be emerged, ran the result script (the script is basicaly a list of emerges and shouldn't do anything odd). The list:

Code: Select all

item 1 =sys-kernel/linux-headers-2.6.17
item 2 =sys-libs/glibc-2.4-r3
item 3 =sys-devel/binutils-2.17
item 4 =sys-libs/zlib-1.2.3
item 5 =virtual/libiconv-0
item 6 =sys-devel/gettext-0.15
item 7 =sys-apps/diffutils-2.8.7-r1
item 8 =sys-libs/ncurses-5.5-r3
item 9 =virtual/libintl-0
item 10 =sys-apps/texinfo-4.8-r4
item 11 =sys-apps/groff-1.19.2-r1
item 12 =sys-devel/binutils-config-1.9-r1
item 13 =sys-devel/gnuconfig-20060702
item 14 =sys-libs/db-4.3.29-r2
item 15 =sys-libs/gdbm-1.8.3-r2
item 16 =sys-devel/libperl-5.8.8-r1
item 17 =dev-lang/perl-5.8.8-r2
item 18 =dev-libs/openssl-0.9.8c
item 19 =perl-core/Test-Harness-2.62
item 20 =app-shells/bash-3.1_p17
item 21 =app-admin/perl-cleaner-1.04.3
item 22 =perl-core/PodParser-1.34
item 23 =app-misc/ca-certificates-20050804
item 24 =dev-python/python-docs-2.4.3
item 25 =dev-libs/expat-2.0.0
item 26 =dev-lang/python-2.4.3-r3
item 27 =dev-python/python-fchksum-1.7.1
item 28 =sys-apps/sandbox-1.2.18.1
item 29 =app-portage/portage-manpages-1.2
item 30 =perl-core/Test-Simple-0.64
item 31 =virtual/perl-Test-Simple-0.64
item 32 =dev-perl/Locale-gettext-1.05
item 33 =sys-apps/help2man-1.36.4
item 34 =sys-devel/m4-1.4.6
item 35 =sys-devel/autoconf-2.60
item 36 =sys-devel/automake-1.5
item 37 =sys-devel/autoconf-2.13
item 38 =sys-devel/autoconf-wrapper-3.2-r2
item 39 =sys-devel/automake-1.8.5-r3
item 40 =sys-devel/automake-1.6.3
item 41 =sys-devel/automake-1.7.9-r1
item 42 =sys-devel/automake-1.4_p6
item 43 =sys-devel/automake-wrapper-1-r1
item 44 =sys-devel/automake-1.9.6-r2
item 45 =sys-apps/man-1.6d
item 46 =sys-devel/libtool-1.5.22
item 47 =dev-util/pkgconfig-0.21
item 48 =x11-misc/util-macros-1.1.0
item 49 =x11-proto/kbproto-1.0.3
item 50 =x11-proto/xextproto-7.0.2
item 51 =x11-proto/xf86bigfontproto-1.1.2
item 52 =x11-proto/inputproto-1.3.2
item 53 =x11-proto/xproto-7.0.7
item 54 =x11-libs/libXau-1.0.2
item 55 =x11-proto/bigreqsproto-1.0.2
item 56 =x11-libs/libXdmcp-1.0.2
item 57 =x11-proto/xcmiscproto-1.1.2
item 58 =x11-libs/xtrans-1.0.1
item 59 =x11-libs/libX11-1.0.3
item 60 =x11-libs/libICE-1.0.1
item 61 =x11-libs/libSM-1.0.1
item 62 =x11-libs/libXt-1.0.2
item 63 =x11-libs/libXext-1.0.1
item 64 =x11-libs/libXpm-3.5.5
item 65 =sys-libs/cracklib-2.8.9-r1
item 66 =sys-devel/flex-2.5.33-r1
item 67 =sys-libs/pam-0.78-r5
item 68 =net-libs/c-client-2004g
item 69 =app-admin/php-toolkit-1.0-r2
item 70 =media-libs/libpng-1.2.12-r1
item 71 =sys-apps/sed-4.1.5
item 72 =sys-libs/readline-5.1_p4
item 73 =sys-libs/timezone-data-2006k
item 74 =dev-libs/gmp-4.2.1
item 75 =sys-devel/bison-2.2
item 76 =dev-libs/mpfr-2.2.0_p14
item 77 =media-libs/libart_lgpl-2.3.17
item 78 =sys-apps/tcp-wrappers-7.6-r8
item 79 =net-nds/openldap-2.3.24-r2
item 80 =dev-db/libpq-8.1.4
item 81 =dev-libs/libmcrypt-2.5.7
item 82 =media-libs/jpeg-6b-r7
item 83 =app-arch/bzip2-1.0.3-r6
item 84 =sys-process/procps-3.2.7
item 85 =sys-apps/ed-0.2-r6
item 86 =dev-db/mysql-5.0.24-r1
item 87 =perl-core/Sys-Syslog-0.18
item 88 =dev-perl/Net-Daemon-0.39
item 89 =virtual/perl-Storable-2.15
item 90 =dev-perl/PlRPC-0.2018
item 91 =dev-perl/DBI-1.52
item 92 =dev-perl/DBD-mysql-3.0006
item 93 =dev-libs/libxml2-2.6.26
item 94 =net-mail/mailbase-1
item 95 =mail-mta/ssmtp-2.61-r2
item 96 =dev-lang/php-5.1.6-r2
item 97 =dev-lang/swig-1.3.29
item 98 =sys-libs/libcap-1.10-r9
item 99 =app-misc/pax-utils-0.1.13
item 100 =dev-php5/pecl-zip-1.7.2
item 101 =app-doc/php-docs-20050822
item 102 =dev-python/pycrypto-2.0.1-r5
item 103 =sys-apps/portage-2.1.1_rc1-r6
item 104 =sys-apps/sysvinit-2.86-r5
item 105 =sys-apps/attr-2.4.32
item 106 =sys-apps/acl-2.2.39
item 107 =sys-apps/coreutils-6.1
item 108 =sys-apps/baselayout-1.12.4-r7
item 109 =sys-apps/hotplug-base-20040401
item 110 =sys-fs/udev-099
...

It failed to emerge item 11 (groff), I skiped it and continued.

This morning it had reported a fail while emerging item 108 (baselayout). I tried to emerge.. Well... I can't emerge anything because it can't cp anything. I can still edit files, and things are running almost nice (I posted this message using X, xfce4, and firefox, so things are not that bad). Ideas? What should I do now?

PS: I may have done some other things yesterday (I was not following a list), those are what I remember and think is relevant.

EDIT: While writting this I noticed it was just after "emerge -1O coreutils" so... err.. How do I fix my coreutils without cp, rm, and emerge ???

How about not using sandbox in emerge, maybe it let's me re-emerge coreutils?

devsk · Post by **devsk** » Sat Sep 09, 2006 9:01 pm

you didn't post other characteristics of your system. What filesystem? what kernel? either your FS is trashed or your disk is going down. I will look at these two first.

from a livecd: did you run fsck on the filesystem to check if its sane?
Did you look at SMART data if your disk is smart capable?

and for future: If you find normal commands like cp misbehaving, try not to mess with filesystem anymore, boot into livecd and run fsck and other diagnostics. By doing the 'rm's that you did, you might have trashed the FS even further.

Phenax · Post by **Phenax** » Sat Sep 09, 2006 9:02 pm

Boot on a live disk and fsck all your fs..

Drysh · Post by **Drysh** » Sat Sep 09, 2006 10:59 pm

Ops... I'm using ext3 and gentoo-sources.

I have 4 partitions in my hd: sda1 (ext2 in /boot), sda2 (ext3 in /), sda3 (xfs in /data), sda4 as swap. The most important files are in /data/username and /data/share. This partion was unmounted before the test/update and I won't even try to mount it now. At worst, if I have to download a new install cd and re-install gentoo from the beggining, I'll loose my configs (/etc and /home/username are in the current partition - I use a symlink /home/username/data to /data/username).

I did a fsck. The odd thing is fsck is ok. It said there were some problems, corrected them and booted again. But rm and cp are still broken. I think the errors were from what I tried to change manualy. But that's not what's causing the problem. BTW: I didn't even had to boot with another disk: I marked the disk to be checked at next boot and rebooted. I was expecting fsck to fail, but it worked fine. My suspects are this is a problem with my coreutils. I would re-emerge them if I could, or even try to install a bin version of it.. But how can I do that without cp?

I'm starting to notice some other strange behaviors in my system: When I reboot, it says the superblock last edition date is in the future (everytime I reboot, but not when I boot after a few hours). If I were using windows I would already have used all anti-virus known to men... But this is Linux, so it's my fault, right?

Or should I suspect of a virus???

Phenax · Post by **Phenax** » Sat Sep 09, 2006 11:02 pm

Make sure your clock is dated correctly
use the

Code: Select all

date

command to make sure.

And yes, I seriously doubt this is a 'virus'

Drysh · Post by **Drysh** » Sat Sep 09, 2006 11:50 pm

I think it's an unrelated matter to the fs. I'm using ntp. Maybe the upgrade changed a config. I'm checking. Thanks.

Now... If anyone knows how to fix a system without cp and rm.

Ivar_Y · Post by **Ivar_Y** » Sun Sep 10, 2006 8:06 am

This bug might explain your superblock time in the future problem: http://bugs.gentoo.org/show_bug.cgi?id=142850

I have no idea how to solve your file system problem.

Ivar

Drysh · Post by **Drysh** » Sun Sep 10, 2006 3:25 pm

That will probably solve the problem with future times... But I can't try it now, because I can't copy files.

I'll try it as soon as fix my fs. Thanks.

Please, I need help with this. How do I fix my coreutils? I don't want to re-install my whole system.

devsk · Post by **devsk** » Sun Sep 10, 2006 4:01 pm

Drysh wrote:Please, I need help with this. How do I fix my coreutils? I don't want to re-install my whole system.

why do you want to fix coreutils when the problem is somewhere else? I don't see any error showing the coreutils is not installed properly or not working properly. In fact, cp and rm have told you that FS is corrupt. And you are not listening.

Did you boot into live cd? did you 'fsck -f /dev/<blah>'? Did you run smartctl -a -d ata /dev/<blah>? Did any of these give errors? Did you run disk vendor checks using their diagnostic cd? do other partitions on the disk work?

don't chase the wrong problem, you will never find the right solution.

Drysh · Post by **Drysh** » Mon Sep 11, 2006 3:36 am

devsk wrote:[Did you boot into live cd? did you 'fsck -f /dev/<blah>'? Did you run smartctl -a -d ata /dev/<blah>? Did any of these give errors? Did you run disk vendor checks using their diagnostic cd? do other partitions on the disk work?

Well, my CDs are in my the office, so I'll only be able to do most of these tests tomorrow night. But, so far I tried:

"touch /forcefsck" and reboot -- It gives an error, corrects it and boots again, then checks again and doesn't find any error. But it does that everytime: if repeat it just after doing that I get the same results.
Boot with the / mounted read-only and fsck -- No errors reported.

But the problem continues. I'll try again tommorow with the disks (live CD and HD diagnostic if I find it). Does it have any difference when I fsck using a CD or booting ro?
I didn't know of smartctrl, and it isn't installed in my system. What should I emerge to get smartctrl?

devsk · Post by **devsk** » Mon Sep 11, 2006 5:01 am

smartmontools