View previous topic :: View next topic |
Author |
Message |
Bio Apprentice
Joined: 17 Mar 2003 Posts: 197 Location: Geneva which should be in Switzerland...
|
Posted: Sun Apr 28, 2013 3:48 pm Post subject: Can't chroot back into an existing and valid install |
|
|
Hi all,
Yesterday I had to shutdown my gentoo server because of power outage. It's been up for more than 100 days. Once power was back and I started my server again I got the following error which prevented me from booting
Code: | CONFIG_DEVTMPFS=y is required in your kernel configuration
for this version of udev to run successfully
this requires immediate attention |
Not worried I took my favorite sysrescueCD and booted my box from the CD in order to build my kernel with the required option. But the problem is that, once booted from the rescue CD, I can't mount all my partitions and then chroot to my box.
FYI, I'm running a hardware raid V box where sda3 is my / partition, sda1 my /boot, then I have many LVM2 drives for several partitions such as /usr, /home, /var, /opt etc...
From the rescueCD command shell, I first mount sda3 as / and here's what I get
Code: | root@sysresccd /root % mount /dev/sda3 /mnt/gentoo
root@sysresccd /root % ls -ail /mnt/gentoo
total 8346950434
2 drwxr-xr-x 22 root root 4096 2012-08-21 11:36 .
11 drwxr-xr-x 10 root root 160 2013-04-28 15:40 ..
14940 drwxr-xr-x 2 root root 4096 2012-03-05 18:08 archive
29763 ---S--x--- 65294 1029803204 824758476 3720873972687735812 1915-02-10 14:55 bin
29761 d-ws--s--- 52718 2197998664 3542305932 1145097220 1908-09-24 03:47 boot
14883 drwxr-xr-x 2 root root 4096 2008-06-14 00:54 data
183 -rw------- 1 root root 0 2009-11-04 02:10 dead.letter
44643 ?-ws--sr-- 60877 4150254812 3959422967 3696159444 1902-10-24 11:37 dev
14884 drwxr-xr-x 80 root root 4096 2013-04-27 15:19 etc
29762 d-wx--sr-- 63928 18692 32029 2223308804 1942-04-15 06:01 home
29885 drwxr-xr-x 14 root root 12288 2013-03-22 09:03 lib
11 drwx------ 2 root root 16384 2008-06-14 00:28 lost+found
62 drwxr-xr-x 2 root root 4096 2011-02-01 13:31 media
64 drwxr-xr-x 8 root root 4096 2013-01-12 10:48 mnt
14882 drwxr-xr-x 2 root root 4096 2008-06-26 21:36 opt
71 drwxr-xr-x 2 root root 4096 2007-04-19 23:51 proc
72 drwxr-xr-x 15 root root 4096 2013-01-15 14:06 root
75 drwxr-xr-x 3 root root 4096 2012-06-11 16:02 run
30223 drwxr-xr-x 2 root root 4096 2013-03-22 09:05 sbin
30404 drwxr-xr-x 2 root root 4096 2007-04-19 23:54 sys
44642 drwS--S--- 64893 4154148073 4064255999 2825178314 1904-07-28 10:29 tmp
14881 drwxr-xr-x 2 root root 4096 2008-06-26 21:36 usr
44641 ?-ws-ws--- 59390 4282223276 2147221467 3435977932 2008-05-01 18:04 var
|
Notice that dev, bin and var are not directories... If i try to remove var or bin here's what I get
Code: | root@sysresccd /root % rm -rf /mnt/gentoo/bin /mnt/gentoo/var
rm: cannot remove `/mnt/gentoo/bin': Operation not permitted
rm: cannot remove `/mnt/gentoo/var': Operation not permitted |
Then I can't mount my var partition
Code: | root@sysresccd /root % mount /dev/vg/var /mnt/gentoo/var
mount: mount point /mnt/gentoo/var is not a directory |
And if I chroot
Code: | root@sysresccd /root % chroot /mnt/gentoo/bin/bash
chroot: cannot change root directory to /mnt/gentoo/bin/bash: Not a directory |
Obviously since my /mnt/gentoo/bin is not a directory...
So now I can't mount my old system back and build my kernel again. I don't know what happened to these strange var, bin, dev folders.
I have a backup of my system and could restore my / partition but before doing so I'd like to know what you think
Thanks for the help _________________ I'm all in ! |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9646 Location: almost Mile High in the USA
|
Posted: Sun Apr 28, 2013 4:35 pm Post subject: |
|
|
I'd probably just restore from backup :(
You could try fscking the disk but before doing that, make sure you copy everything that you may need onto yet another disk, just in case there's something on the drive that was modified after the backup was made.
I'm surprised there's that much damage to the filesystem after one unexpected shutdown but that's exactly the risk of unclean shutdowns, especially of RAID5...
BTW: what hardware raid5 is it, and did you see any disks kicked? all drives still up and running? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Bio Apprentice
Joined: 17 Mar 2003 Posts: 197 Location: Geneva which should be in Switzerland...
|
Posted: Sun Apr 28, 2013 5:29 pm Post subject: |
|
|
Well that's the thing as shutdown was clean. I have a UPS, and since power was off I was still able to SSH to my server and shut it down nicely
I ran a fsck.ext3 : no problems found
Aiming toward a restore of my / partition :/
My RAID5 is a 3ware 9650 4 ports, I can't query it from the rescue CD and get a status on my drives as the tw_cli is not available. But, before shutdown everything was running fine so I doubt there's a problem here _________________ I'm all in !
Last edited by Bio on Sun Apr 28, 2013 6:11 pm; edited 1 time in total |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9646 Location: almost Mile High in the USA
|
Posted: Sun Apr 28, 2013 6:04 pm Post subject: |
|
|
so fsck reported a clean shutdown? Tried a forced fsck?
Something caused the disk image to get corrupted somehow... May need to look into bad ram or perhaps bad RAID controller. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Bio Apprentice
Joined: 17 Mar 2003 Posts: 197 Location: Geneva which should be in Switzerland...
|
Posted: Mon Apr 29, 2013 3:14 pm Post subject: |
|
|
Forced fsck worked and my "weird" directories have just been removed which is ok for most of them as they are LVM mount points, but in the process I lost my /bin folder and my backup is unusable.
Any advice on how to proceed in order to get a clean and working /bin from scratch?
emerge -e world??? _________________ I'm all in ! |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9646 Location: almost Mile High in the USA
|
Posted: Mon Apr 29, 2013 9:09 pm Post subject: |
|
|
Sounds reasonable, grab files from http://tinderbox.dev.gentoo.org or just carefully copy the files from a stage3, and re-emerge everything again. Yeah this does look bad :(
And it goes without saying, even if you have a backup, it needs to be tested... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Bio Apprentice
Joined: 17 Mar 2003 Posts: 197 Location: Geneva which should be in Switzerland...
|
Posted: Mon Apr 29, 2013 9:26 pm Post subject: |
|
|
Yep I copied /bin from latest rescue CD and now I'm emerging everything from scratch. Looks like it should fix it. And now I'm gonna pay close attention to my RAID controller and disks _________________ I'm all in ! |
|
Back to top |
|
|
|