Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Can't chroot back into an existing and valid install
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
Bio
Apprentice
Apprentice


Joined: 17 Mar 2003
Posts: 197
Location: Geneva which should be in Switzerland...

PostPosted: Sun Apr 28, 2013 3:48 pm    Post subject: Can't chroot back into an existing and valid install Reply with quote

Hi all,

Yesterday I had to shutdown my gentoo server because of power outage. It's been up for more than 100 days. Once power was back and I started my server again I got the following error which prevented me from booting

Code:
CONFIG_DEVTMPFS=y is required in your kernel configuration
for this version of udev to run successfully
this requires immediate attention


Not worried I took my favorite sysrescueCD and booted my box from the CD in order to build my kernel with the required option. But the problem is that, once booted from the rescue CD, I can't mount all my partitions and then chroot to my box.

FYI, I'm running a hardware raid V box where sda3 is my / partition, sda1 my /boot, then I have many LVM2 drives for several partitions such as /usr, /home, /var, /opt etc...

From the rescueCD command shell, I first mount sda3 as / and here's what I get

Code:
root@sysresccd /root % mount /dev/sda3 /mnt/gentoo
root@sysresccd /root % ls -ail /mnt/gentoo

total 8346950434
    2 drwxr-xr-x    22 root       root                      4096 2012-08-21 11:36 .
   11 drwxr-xr-x    10 root       root                       160 2013-04-28 15:40 ..
14940 drwxr-xr-x     2 root       root                      4096 2012-03-05 18:08 archive
29763 ---S--x--- 65294 1029803204  824758476 3720873972687735812 1915-02-10 14:55 bin
29761 d-ws--s--- 52718 2197998664 3542305932          1145097220 1908-09-24 03:47 boot
14883 drwxr-xr-x     2 root       root                      4096 2008-06-14 00:54 data
  183 -rw-------     1 root       root                         0 2009-11-04 02:10 dead.letter
44643 ?-ws--sr-- 60877 4150254812 3959422967          3696159444 1902-10-24 11:37 dev
14884 drwxr-xr-x    80 root       root                      4096 2013-04-27 15:19 etc
29762 d-wx--sr-- 63928      18692      32029          2223308804 1942-04-15 06:01 home
29885 drwxr-xr-x    14 root       root                     12288 2013-03-22 09:03 lib
   11 drwx------     2 root       root                     16384 2008-06-14 00:28 lost+found
   62 drwxr-xr-x     2 root       root                      4096 2011-02-01 13:31 media
   64 drwxr-xr-x     8 root       root                      4096 2013-01-12 10:48 mnt
14882 drwxr-xr-x     2 root       root                      4096 2008-06-26 21:36 opt
   71 drwxr-xr-x     2 root       root                      4096 2007-04-19 23:51 proc
   72 drwxr-xr-x    15 root       root                      4096 2013-01-15 14:06 root
   75 drwxr-xr-x     3 root       root                      4096 2012-06-11 16:02 run
30223 drwxr-xr-x     2 root       root                      4096 2013-03-22 09:05 sbin
30404 drwxr-xr-x     2 root       root                      4096 2007-04-19 23:54 sys
44642 drwS--S--- 64893 4154148073 4064255999          2825178314 1904-07-28 10:29 tmp
14881 drwxr-xr-x     2 root       root                      4096 2008-06-26 21:36 usr
44641 ?-ws-ws--- 59390 4282223276 2147221467          3435977932 2008-05-01 18:04 var


Notice that dev, bin and var are not directories... If i try to remove var or bin here's what I get

Code:
root@sysresccd /root % rm -rf /mnt/gentoo/bin /mnt/gentoo/var
rm: cannot remove `/mnt/gentoo/bin': Operation not permitted
rm: cannot remove `/mnt/gentoo/var': Operation not permitted


Then I can't mount my var partition

Code:
root@sysresccd /root % mount /dev/vg/var /mnt/gentoo/var
mount: mount point /mnt/gentoo/var is not a directory


And if I chroot

Code:
root@sysresccd /root % chroot /mnt/gentoo/bin/bash
chroot: cannot change root directory to /mnt/gentoo/bin/bash: Not a directory


Obviously since my /mnt/gentoo/bin is not a directory...

So now I can't mount my old system back and build my kernel again. I don't know what happened to these strange var, bin, dev folders.

I have a backup of my system and could restore my / partition but before doing so I'd like to know what you think

Thanks for the help
_________________
I'm all in !
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9646
Location: almost Mile High in the USA

PostPosted: Sun Apr 28, 2013 4:35 pm    Post subject: Reply with quote

I'd probably just restore from backup :(

You could try fscking the disk but before doing that, make sure you copy everything that you may need onto yet another disk, just in case there's something on the drive that was modified after the backup was made.
I'm surprised there's that much damage to the filesystem after one unexpected shutdown but that's exactly the risk of unclean shutdowns, especially of RAID5...

BTW: what hardware raid5 is it, and did you see any disks kicked? all drives still up and running?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Bio
Apprentice
Apprentice


Joined: 17 Mar 2003
Posts: 197
Location: Geneva which should be in Switzerland...

PostPosted: Sun Apr 28, 2013 5:29 pm    Post subject: Reply with quote

Well that's the thing as shutdown was clean. I have a UPS, and since power was off I was still able to SSH to my server and shut it down nicely

I ran a fsck.ext3 : no problems found

Aiming toward a restore of my / partition :/

My RAID5 is a 3ware 9650 4 ports, I can't query it from the rescue CD and get a status on my drives as the tw_cli is not available. But, before shutdown everything was running fine so I doubt there's a problem here
_________________
I'm all in !


Last edited by Bio on Sun Apr 28, 2013 6:11 pm; edited 1 time in total
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9646
Location: almost Mile High in the USA

PostPosted: Sun Apr 28, 2013 6:04 pm    Post subject: Reply with quote

so fsck reported a clean shutdown? Tried a forced fsck?

Something caused the disk image to get corrupted somehow... May need to look into bad ram or perhaps bad RAID controller.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Bio
Apprentice
Apprentice


Joined: 17 Mar 2003
Posts: 197
Location: Geneva which should be in Switzerland...

PostPosted: Mon Apr 29, 2013 3:14 pm    Post subject: Reply with quote

Forced fsck worked and my "weird" directories have just been removed which is ok for most of them as they are LVM mount points, but in the process I lost my /bin folder and my backup is unusable.

Any advice on how to proceed in order to get a clean and working /bin from scratch?

emerge -e world???
_________________
I'm all in !
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9646
Location: almost Mile High in the USA

PostPosted: Mon Apr 29, 2013 9:09 pm    Post subject: Reply with quote

Sounds reasonable, grab files from http://tinderbox.dev.gentoo.org or just carefully copy the files from a stage3, and re-emerge everything again. Yeah this does look bad :(

And it goes without saying, even if you have a backup, it needs to be tested...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Bio
Apprentice
Apprentice


Joined: 17 Mar 2003
Posts: 197
Location: Geneva which should be in Switzerland...

PostPosted: Mon Apr 29, 2013 9:26 pm    Post subject: Reply with quote

Yep I copied /bin from latest rescue CD and now I'm emerging everything from scratch. Looks like it should fix it. And now I'm gonna pay close attention to my RAID controller and disks
_________________
I'm all in !
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum