[solved] initramfs - Kernel panic - BusyBox - Dm-Crypt

[mcimaster]

Hello,

I have installed Gentoo with LVM2 and Dm-Crypt. I followed this tutorial: http://en.gentoo-wiki.com/wiki/Root_fil ... t_and_RAID and I just skipped the RAID parts. Now I am getting an error message after typing the password and I can't find a solution. The system hangs up with the following message:

Code: Select all

  2 logical volume(s) in volume group "vg" now active
mount: mounting /dev/mapper/vg-root on /newroot failed: No such file or directory

BusyBox v1.15.3 (2010-09-30 15:49:47 CEST) multi-call binary

Usage: switch_root [-c /dev/console] NEW_ROOT NEW_INIT [ARGS]

Free initramfs and switch to another root fs:
chroot to NEW_ROOT, delete all in /, move NEW_ROOT to /,
execute NEW_INIT. PID must be 1. NEW_ROOT must be a mountpoint.

Options:
        -c DEV  Reopen stdio to DEV after switch

[   15.233541] Kernel panic - not syncing: Attempted to kill init!
[   15.233787] Pid: 1, comm: busybox Not tainted 2.6.35-gentoo-r9 #3
[   15.234030] Call Trace:
[   15.234288]  [<ffffffff81569116>] panic+0xa0/0x152

Here are my system settings:

Kernel: 2.6.35-gentoo-r9

Partitions:
/dev/sda1 --> Windows 7
/dev/sda2 --> Windows 7
/dev/sda3 --> boot (Gentoo)
/dev/sda4 --> system (Gentoo) The whole sda4 partition is a Volumegroup called VG and it is encrypted. This VG has two LVs (swap and root).

My fdisk looks like this:

Code: Select all

# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>

/dev/sda3               /boot           ext2            defaults,noatime        1 2
/dev/mapper/vg-root             /               ext3            noatime         0 1
/dev/mapper/vg-swap             none            swap            sw              0 0
/dev/cdrom              /mnt/cdrom      auto            noauto,ro       0 0

shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0

My grub.conf looks like that:

Code: Select all

default 0
timeout 3
splashimage=(hd0,2)/boot/grub/splash.xpm.gz

title Gentoo Linux 2.6.35-gentoo-r9
root (hd0,2)
kernel /boot/kernel-2.6.35-gentoo-r9
initrd /boot/initramfs


title Windows 7
rootnoverify (hd0,0)
makeactive
chainloader +1

My current init is:

Code: Select all

#!/bin/busybox sh

mount -t proc proc /proc
CMDLINE=`cat /proc/cmdline`

mount -t sysfs sysfs /sys

#wait a little to avoid trailing kernel output
sleep 3

#If you don't have a qwerty keyboard, uncomment the next line 
loadkmap < /etc/kmap-de

#If you have a msg, show it: 
cat /etc/msg

#dm-crypt
/bin/cryptsetup luksOpen /dev/sda4 vault

sleep 2

#lvm
#/bin/vgscan
/bin/vgchange -ay vg

sleep 2

#root filesystem
mount -r /dev/mapper/vg-root /newroot

#unmount pseudo FS
umount /sys
umount /proc
#umount /dev

#root switch
#exec /bin/busybox switch_root /newroot /sbin/init ${CMDLINE}
exec switch_root /newroot /sbin/init

The init script I tried before:

Code: Select all

#!/bin/sh

mount -t proc proc /proc
CMDLINE=`cat /proc/cmdline`

mount -t sysfs sysfs /sys

#wait a little to avoid trailing kernel output
sleep 3

#If you don't have a qwerty keyboard, uncomment the next line 
loadkmap < /etc/kmap-de

#If you have a msg, show it: 
#cat /etc/msg

#dm-crypt
/bin/cryptsetup luksOpen /dev/sda4 vault 

#lvm
#/bin/vgscan
/bin/vgchange -ay vg

#root filesystem
mount -r /dev/mapper/vg-root /newroot

#unmount pseudo FS
umount /sys
umount /proc

#root switch
exec /bin/busybox switch_root /newroot /sbin/init ${CMDLINE}

I hope you can give me some hints.

[Hu]

Do you have a /newroot directory in the initramfs for mount to place the filesystem on? If not, make one. If it still fails, modify your init script to drop you to an interactive shell so you can explore the initramfs. Start with checking the contents of /dev/mapper.

[mcimaster]

Thank you for the quick response.

Yes, I do have the /newroot directory. I created a new directory and changed the entry in the init file, but it is still the same error. He can't see the mountpoint.

How can I create an interactive shell in the init script?


EDIT: When I mount the root file system, I also use the same procedure.
1) cryptsetup luksOpen /dev/sda4 vault
2) vgchange -ay vg
3) mount /dev/mapper/vg-root /mnt/gentoo; mount /dev/sda3 /mnt/gentoo/boot;mount -t proc none /mnt/gentoo/proc;mount -o bind /dev /mnt/gentoo/dev;chroot /mnt/gentoo /bin/bash

And everything works like a charm. I don't understand, why initram gives me errors.

[mcimaster]

Okay, I slowed down the init process by adding some sleep commands into the init script.

Before I get the error message and the screen freezes, there is another part of the error:

Code: Select all

Unable to make device node for 'vg-root'
/dev/mapper/vg-root: open failed: No such file or directory

Does anybody know what that means?

[Hu]

It means that no directory was available to hold the device node when it attempted to create the device node vg-root. Did you forget to create /dev and/or /dev/mapper in the initramfs?

[mcimaster]

It means that no directory was available to hold the device node when it attempted to create the device node vg-root. Did you forget to create /dev and/or /dev/mapper in the initramfs?

No, I've created both in the initram. I checked also my LVM kernel settings, but all needed settings are included in the kernel. I really have no clue...

[Hu]

Modify the initramfs to start a shell just before that mount command would execute. When you get the shell, run ls -lR / and post the output.

[mcimaster]

Ok, I have installed a fresh system, but I still have the same error.

I wanted to do something like that and save the output on my pendrive:

Code: Select all

...
#lvm
#/bin/vgscan
/bin/vgchange -ay vg

sleep 5

mount -t devtmpfs none /dev
mount /dev/sdb1 /newroot

sleep 5

ls -lR / >> /newroot/log.log

sleep 30
umount /dev/sdb1
sleep 5

#root filesystem
mount -r /dev/mapper/vg-root /newroot
...

But I can't mount to any folder in the initramfs. Is there another possibility to save the output?

[Hu]

But I can't mount to any folder in the initramfs. Is there another possibility to save the output?

Pen and paper works well in these situations. You could also use a serial console or netconsole, but for a one-off, written text will be simpler.

[mcimaster]

But I can't mount to any folder in the initramfs. Is there another possibility to save the output?

Pen and paper works well in these situations. You could also use a serial console or netconsole, but for a one-off, written text will be simpler.

There is nothing better than the traditional way.

It's impossible for me to read the output, because it's gigantic. I included less into my initramfs, but it doesn't work. The display just printed out the whole "ls", and at the end I couldn't scroll up to the beginning of the output.

Code: Select all

ls -lR / | less

Are you looking for something specific?

[mcimaster]

I have tried to log the init messages via net: http://forums.gentoo.org/viewtopic-p-36 ... ml#3682124

Unfortunatly it doesn't work...

EDIT: I checked my router. It seems that my NIC is not ready, and doesn't have an IP-adress.

[Hu]

Are you looking for something specific?

Yes. I am trying to see why you get "No such file or directory." Since we now know we are chasing a failure to create the LVM device node, we primarily need to see inside /dev/mapper. My guess is that it is missing completely. An interactive shell would be much nicer here because you can save the output into the initramfs to view it piecemeal and because you can explore the initramfs to understand exactly why creation of the device node fails.

[mcimaster]

Are you looking for something specific?

Yes. I am trying to see why you get "No such file or directory." Since we now know we are chasing a failure to create the LVM device node, we primarily need to see inside /dev/mapper. My guess is that it is missing completely. An interactive shell would be much nicer here because you can save the output into the initramfs to view it piecemeal and because you can explore the initramfs to understand exactly why creation of the device node fails.

I don't know howto make an interactive shell. Do you have something ala quick and dirty? Otherwise I will have to consult Dr. Google...

I am not sure, if this is the info we are looking for, but before I mount -r /dev/mapper/vg-root /newroot, I do a ls in /dev/mapper and it says:

Code: Select all

control
vault
vg-root -> ../dm-2
vg-swap

Init look like that:

Code: Select all

...
#dm-crypt
/bin/cryptsetup luksOpen /dev/sda4 vault

#lvm
#/bin/vgscan
/bin/vgchange -ay vg

ls /dev/mapper
sleep 15

#root filesystem
mount -r /dev/mapper/vg-root /newroot
...

[mcimaster]

Allright, I am in the init shell, and I will write the process step by step down...

Code: Select all

/ # /bin/cryptsetup luksOpen /dev/sda4 vault
Enter passphrase for /dev/sda4:
/ # /bin/vgchange -ay vg
   Unable to make device node for 'vg-root'
   /dev/mapper/vg-root: open failed: No such file or directory
   2 logical volume(s) in volume group "vg" now active
/ #

So here we are. This the error. What shall I do know?

[Hu]

I don't know howto make an interactive shell. Do you have something ala quick and dirty? Otherwise I will have to consult Dr. Google...

A shell is interactive by default. Just start a shell at the right point in the script:

Code: Select all

mount -r /dev/mapper/vg-root /newroot || /bin/sh

I am not sure, if this is the info we are looking for, but before I mount -r /dev/mapper/vg-root /newroot, I do a ls in /dev/mapper and it says:

Code: Select all

control
vault
vg-root -> ../dm-2
vg-swap

This seems to conflict with your prior report of being unable to create vg-root.

Allright, I am in the init shell, and I will write the process step by step down...

Code: Select all

/ # /bin/vgchange -ay vg
   Unable to make device node for 'vg-root'
   /dev/mapper/vg-root: open failed: No such file or directory
   2 logical volume(s) in volume group "vg" now active
/ #

So here we are. This the error. What shall I do know?

Does /dev/mapper/vg-root exist despite the error message? If it is a symlink, does it point to something which exists? If you run the mount command, does it work?

[mcimaster]

Okay, I followed the whole init script. This is how it looks like:

Code: Select all

/ # /bin/cryptsetup luksOpen /dev/sda4 vault
Enter passphrase for /dev/sda4:
/ # /bin/vgchange -ay vg
   Unable to make device node for 'vg-root'
   /dev/mapper/vg-root: open failed: No such file or directory
   2 logical volume(s) in volume group "vg" now active
/ # /bin/vgchange -ay vg
   2 logical volume(s) in volume group "vg" now active
/ # mount -r /dev/mapper/vg-root /newroot
mount: mounting /dev/mapper/vg-root on /newroot failed: No such file or directory
/ # ls /dev/mapper/
control   vault   vg-root   vg-swap
/ # ls -la /dev/mapper/
control
vault
vg-root -> ../dm-2
vg-swap
/ # 

[mcimaster]

Allright, now I understand what you are saying...

No, this Link doesn't exist.

Code: Select all

vg-root -> ../dm-2

But I don't know why there is a link to dm-2. From where did I get it? What do I have to do know?

[Hu]

That is one way that device-mapped devices can be presented. It was probably created that way by either lvm2 or your dynamic device manager, if you have one. On my system, both the /dev/mapper node and the /dev/dm-* nodes are proper block devices, not symlinks.

What version of the various programs have you put in your initramfs?

[mcimaster]

I am not sure, if I know what you mean.

I put all the programs into the initramfs, which were mentioned in this tut: http://en.gentoo-wiki.com/wiki/Root_fil ... t_and_RAID

[mcimaster]

It looks like a never ending story

I have read that:

/dev/dm-* devices belong to the "device-mapper" subsystem, which is used to implement LVM, dm-multipath and software RAID in Linux 2.6.* kernel series.

The /dev/dm-* names are not persistent: they are allocated ascending order to whatever disk device-mapper is used with. There is no guarantee that /dev/dm-0 will still be /dev/dm-0 after a reboot. These device names should only be used for device-mapper debugging.

On the running system (via livecd) this looks like that:

Code: Select all

(chroot) livecd initram # ls -la /dev/dm*
brw-rw---- 1 root disk 253, 0 Oct 11 01:05 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Oct 11 01:05 /dev/dm-1
brw-rw---- 1 root disk 253, 2 Oct 11 01:05 /dev/dm-2

(chroot) livecd initram # ls -la /dev/mapper/
total 0
drwxr-xr-x  2 root root    120 Oct 11 01:05 .
drwxr-xr-x 15 root root   3540 Oct 11 01:05 ..
crw-rw----  1 root root 10, 57 Oct 11 00:53 control
lrwxrwxrwx  1 root root      7 Oct 11 01:05 vault -> ../dm-0
lrwxrwxrwx  1 root root      7 Oct 11 01:05 vg-root -> ../dm-2
lrwxrwxrwx  1 root root      7 Oct 11 01:05 vg-swap -> ../dm-1

Please correct me if I am wrong. When I copy the symlink of vg-root into the initramfs, I actually copy f.e. ../dm-2

How can I avoid this?

[mcimaster]

Finally I've solved the problem.

Hu mentioned it before, but my thinking was too slow.

The symlink vg-root -> ../dm-2 was dead. There was no dm-2 device in the initramfs. So I just copied all dm-* devices into the initramfs. And now everthing works fine.

Thank you for your patience and your great support.

I wish you all the best.
Good night.