symlinks in custom initramfs not resolved at boot [solved]

[triquetra]

I have been trying to create an initramfs with genkernel-next, but without success, as [topic=1009682]documented here[/topic].

In an attempt to troubleshoot the issue, I tried to create an initramfs with dracut, and even tried multiple permutations of a custom initramfs. None of them worked.

I have a booting system right now only because I am booting with an initramfs created with the old genkernel, but it seems I am unable to create a new functioning initramfs by any means.

I have banged my head against this problem for many hours over several days, and I am at a loss to figure out what I may be doing wrong. Any ideas?

[NeddySeagoon]

triquetra,

Luckily systemd has nothing to do with an initramfs, or I would not be posting here.

When the black magic of automatic initramfs lets you down, you ned to fall back on the manual controls.
About the only way te debug what is happening, is to sprinkle the init script in the initfamfs with commands to show mhats happening and pauses to let you read the console.

Please post the init script from your initrd and I'll suggest some debug additions.
If you made your own directory structure, that would be useful too.

[triquetra]

Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.

Code: Select all

#!/bin/busybox sh
 
rescue_shell() {
    echo $1
    busybox --install -s
    exec /bin/sh
}

# Mount the /proc and /sys filesystems.
mount -t devtmpfs none /dev  
mount -t proc none /proc
mount -t sysfs none /sys

# Assemble RAID:
( sleep 2
  mdadm --assemble --scan
  sleep 2
) &
wait

# Mount the root filesystem.
mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"
    
# Clean up.
umount /proc
umount /sys
umount /dev
     
# Boot the real thing.
exec switch_root /mnt/root /sbin/init

I have made my own directory structure as follows:

[code="su -p -c "find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-custom-3.18.2-gentoo""]
.
./bin
./bin/busybox
./lib64
./dev
./dev/null
./dev/tty
./dev/console
./dev/md126p5
./root
./proc
./mnt
./mnt/root
./sbin
./sbin/mdadm
./lib
./init
./etc
./etc/mdadm.conf
./sys
7 blocks
[/code]

Thank you in advance for your help.

[NeddySeagoon]

triquetra,

That looks "mostly harmless" and in line with the accepted wisdom.

I had similar pain a few months ago trying to get a remote box up. In the end I spent a few days debugging locally.
What I found was the busybox no longer runs its own applets without a full path and or symlinks.
This means your

Code: Select all

# Mount the /proc and /sys filesystems.
mount -t devtmpfs none /dev 
mount -t proc none /proc
mount -t sysfs none /sys 

all fail, then its steadily downhill from there.

Its not until you get to

Code: Select all

|| rescue_shell "Problem mounting root.  Dropping to shell" 

that the symlinks have a chance to get written. That may fail too as busybox is at /bin/busybox and no PATH is set.

Heres a few snippits of my init. Notice the full paths.

Code: Select all

#!/bin/busybox sh
rescue_shell() {
    echo "$@"
    echo "Something went wrong. Dropping you to a shell."
# have time to read the message
    /bin/sleep 20 
    /bin/busybox --install -s
    exec /bin/sh
}

and the commented out debug

Code: Select all

#echo "/bin"
#/bin/ls -l /bin
#echo "/sbin"
#/bin/ls -l /sbin

#/bin/sleep 20

#echo "Mount proc"
/bin/mount -t proc proc /proc
#/bin/busybox sleep 20

/bin/ls is a symlink to /bin/busybox that is created elsewhere, likewise /bin/sleep.
You will need symlinks for all of the busybox applets you want to use.
You can either set the PATH or use full path names. Setting the PATH was something I got wrong once, so I just left it in a mess.

[triquetra]

Hmmm ... no dice.

Here's my current init script:

Code: Select all

#!/bin/busybox sh
 
# Create applets
/bin/busybox --install -s

pause() {
    read -p "$*"
}

rescue_shell() {
    /bin/echo $1
    /bin/busybox --install -s
    exec /bin/sh
}

# Mount the /proc and /sys filesystems.
/bin/mount -t devtmpfs none /dev  
/bin/mount -t proc none /proc
/bin/mount -t sysfs none /sys
#echo /sbin/mdev > /proc/sys/kernel/hotplug
#mdev -s

# Assemble RAID:
( sleep 2
  /sbin/mdadm --assemble --scan
  sleep 2
) &
wait
pause "raid array just assembled.  Press Enter to continue"
# Mount the root filesystem.
/bin/mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"
    
# Clean up.
/bin/umount /proc
/bin/umount /sys
/bin/umount /dev
     
# Boot the real thing.
exec /sbin/switch_root /mnt/root /sbin/init

And here's what it gives me.

[NeddySeagoon]

triquetra,

I don't see whats happening with raid assembly but /dev/sdb unknown partition table looks odd.
I can't see what it says about /dev/sda

This suggests that you are using either fakeraid in raid0 across sda and sdb, in which case you only get a partition table on sda, or that you are using a partition table that is not supported by your kernel.

The kernel can see the HDD now, so thats progress.

[triquetra]

Yeah, some of the text is missing, and it scrolls by too fast for me to get it. Because of the stage that bootup is failing, I presume that there is no text file for me to obtain the missing text, right?

I am using mdadm with a fakeraid raid0 setup across sda and sdb. When the computer boots (with my old initramfs created with the old genkernel), it mounts at /dev/imsm0 (or something similar). Why would I get a partition table on sdb with genkernel initramfs, but not with the custom initramfs?

I'll try to post the genkernel-next boot failure screen for comparison a bit later.

[triquetra]

The change in my custom init script did not change the result. The kernel panic screen looks nearly identical, if not exactly so, in comparison to my previous custom initramfs.

Here's a pic of the genkernel-next failure for comparison. That seems to get farther along than the custom initramfs, but still no boot.

[triquetra]

Still no progress here. Any more ideas?

[s0be]

Is there any chance you're missing the devicemapper or raid modules in your kernel? If you compiled them as modules and put them in your init.d, you would probably need to manually insmod them.

[triquetra]

Devicemapper and raid modules are built into the kernel.

[juggling_ben]

Few things:

Can you post a ls -la from inside your init directory structure? Your image shows an error code of -40 when running init, which according to my reading of the kernel code, means it can't determine the binary format of /init

Also, in your listing of your initramfs, you seem to be missing any .so files. Are you compiling the tools with the 'static' use flag? Particularly busybox.

I'd be happy to share by init script and my little build script if you'd like. It's only slightly more complicated than yours (after assembling the raid, I have to decrypt, do lvm stuff, etc) but hasn't failed me in quite some time.

[s0be]

Comparing your initrd to mine (ignoring the structural differences), you don't specify a mount point when you mount the gentoo named partition:

Code: Select all

/bin/mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell" 

vs

Code: Select all

mount -o ro $(findfs "$dev") /mnt/root >>/tmp/initrdlog 2>&1

in mine

You can see my initrd stuff at:

https://github.com/s0be/modular_initrd/blob/master/init

[triquetra]

Few things:

Can you post a ls -la from inside your init directory structure? Your image shows an error code of -40 when running init, which according to my reading of the kernel code, means it can't determine the binary format of /init

Also, in your listing of your initramfs, you seem to be missing any .so files. Are you compiling the tools with the 'static' use flag? Particularly busybox.

I'd be happy to share by init script and my little build script if you'd like. It's only slightly more complicated than yours (after assembling the raid, I have to decrypt, do lvm stuff, etc) but hasn't failed me in quite some time.

Code: Select all

total 56K
drwxr-xr-x 13 root root 4.0K Feb 10 16:01 .
drwxr-xr-x 11 root root 4.0K Feb 20 10:14 ..
drwxr-xr-x  2 root root 4.0K Feb  8 16:32 bin
drwxr-xr-x  2 root root 4.0K Feb  9 14:16 dev
drwxr-xr-x  2 root root 4.0K Feb  9 13:56 etc
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 lib
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 lib64
drwxr-xr-x  3 root root 4.0K Feb  8 16:27 mnt
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 proc
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 root
drwxr-xr-x  2 root root 4.0K Feb  9 14:15 sbin
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 sys
drwxr-xr-x  4 root root 4.0K Feb 10 16:01 usr
-rwxr-xr-x  1 root root  786 Feb 22 18:03 init

Code: Select all

 * Found these USE flags for sys-apps/busybox-1.23.1-r1:
 U I
 - - debug         : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml
 + + ipv6          : Add support for IP version 6
 - - livecd        : !!internal use only!! DO NOT SET THIS FLAG YOURSELF!, used during livecd building
 - - make-symlinks : Create all the appropriate symlinks in /bin and /sbin.
 - - math          : Enable math support in gawk (requires libm)
 - - mdev          : Create the appropriate symlink in /sbin and install mdev.conf and support files
 + + pam           : Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
 - - savedconfig   : Use this to restore your config from /etc/portage/savedconfig ${CATEGORY}/${PN}. Make sure your USE flags allow for appropriate dependencies
 - - sep-usr       : Support a separate /usr without needing an initramfs by booting with init=/ginit
 + + static        : Make the system rescue shell (/bin/bb) static so you can recover even when glibc is broken
 - - syslog        : Enable support for syslog
 + + systemd       : Support systemd

Code: Select all

 * Found these USE flags for sys-fs/mdadm-3.3.2:
 U I
 + + static : !!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically

Please share your init and build script.

[triquetra]

@s0be

Thanks for the reply, and sharing your initramfs. I do usually find that problems I have with gentoo are because I've overlooked something simple, but specifying the mount point on the mount line was not, unfortunately, the problem (or at least not the primary one). I changed my init to specify the mount point of /mnt/root, recompiled the initramfs, and rebooted, but got the same kernel panic.

[s0be]

your error code (-40) looks like it could be:

#define ELOOP 40 /* Too many symbolic links encountered */

So I bet you have some broken symbolic links once you consider that the top level of your init tree becomes / in initramfs.

The output of:

Code: Select all

for f in $(find -type l); do ls -l $f; done;

Will show what your symlinks are. They should all be 'relative' so that they work from both initramfs and when you're editing it.

Code: Select all

s0be@anvil /usr/src/initramfs $ for f in $(find -type l); do ls -l $f; done;
lrwxrwxrwx 1 s0be s0be 3 Oct 11  2012 ./sbin -> bin
lrwxrwxrwx 1 s0be s0be 8 Oct 11  2012 ./usr/sbin -> ../sbin/
lrwxrwxrwx 1 s0be s0be 6 Oct 11  2012 ./usr/bin -> ../bin

[juggling_ben]

So thinking about this a bit, the problem isn't related to your init script. Your boot process isn't getting that far.

Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.
[code="su -p -c "find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-custom-3.18.2-gentoo""]
.
./bin
./bin/busybox
./lib64
./dev
./dev/null
./dev/tty
./dev/console
./dev/md126p5
./root
./proc
./mnt
./mnt/root
./sbin
./sbin/mdadm
./lib
./init
./etc
./etc/mdadm.conf
./sys
7 blocks
[/code]

7 blocks? That does not sound right to me. If I only include statically-linked busybox in my own, it is > 5000 blocks. What is bin/busybox in your initramfs? (ie what does "file bin/busybox" say?) If you copied a symlink or something instead, it would fit with my earlier confusion about the 'error -40'

[s0be]

So thinking about this a bit, the problem isn't related to your init script. Your boot process isn't getting that far.

Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.
[code="su -p -c "find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-custom-3.18.2-gentoo""]
.
./bin
./bin/busybox
./lib64
./dev
./dev/null
./dev/tty
./dev/console
./dev/md126p5
./root
./proc
./mnt
./mnt/root
./sbin
./sbin/mdadm
./lib
./init
./etc
./etc/mdadm.conf
./sys
7 blocks
[/code]

7 blocks? That does not sound right to me. If I only include statically-linked busybox in my own, it is > 5000 blocks. What is bin/busybox in your initramfs? (ie what does "file bin/busybox" say?) If you copied a symlink or something instead, it would fit with my earlier confusion about the 'error -40'

That makes a lot of sense, I bet $INITRDPATH/bin/busybox is a symlink to /bin/busybox. That'd lead the the error -40 (ELOOP) as it'd end up being a symlink to itself, and it would give him the tiny initrd rather than one of appreciable size.

[triquetra]

Yes, that was it. Thanks for the help debugging this issue. I thought I would be clever and symlink the files needed for the initramfs so that I was sure to always have the current version. Obviously, that did not work. Would hard links accomplish that goal, or is it better to just use a mkinitramfs script to build the initramfs files and directory structure each time I rebuild the initramfs?

[NeddySeagoon]

triquetra,

Provided that there are no kernel modules in the initrd, its just like fimware.
Why would you ever rebuild it?

Its a closed system that does some stuff and gets abandoned once it done.

[triquetra]

@NeddySeagoon

Good question. I had just gotten into the habit (when using genkernel) of building a new initramfs every time I upgraded my kernel.

[s0be]

Yes, that was it. Thanks for the help debugging this issue. I thought I would be clever and symlink the files needed for the initramfs so that I was sure to always have the current version. Obviously, that did not work. Would hard links accomplish that goal, or is it better to just use a mkinitramfs script to build the initramfs files and directory structure each time I rebuild the initramfs?

When it comes to initramfs, etc, making changes as rarely as possible is better, IMHO. You don't always want the 'latest and greatest' versions in it, because they change behaviors and whatnot, and pre-init is a very delicate time. I haven't upgraded the busybox in my initramfs since May 14, 2012. If it works, don't replace it .

[Hu]

Other posters make a good argument for why not to rebuild your initramfs. If you decide you want to rebuild it regularly, I recommend letting the kernel build system create it and embed it into the kernel. This has the disadvantage of keeping one copy of the initramfs for each kernel you build, but the advantage that each kernel is wholly self-contained and has an initramfs loaded even if you use a bootloader that does not support loading an initramfs.

[s0be]

I didn't want to suggest embedding it directly..... but....

It's exactly what I do. There's a problem with rebuilding kernels with embedded initramfs, here's the patch I've been using for years:

Code: Select all

diff --git a/usr/Makefile b/usr/Makefile
index e767f01..041333b 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -60,15 +60,7 @@ targets := initramfs_data.cpio.gz initramfs_data.cpio.bz2 \
        initramfs_data.cpio.lzma initramfs_data.cpio.xz \
        initramfs_data.cpio.lzo initramfs_data.cpio.lz4 \
        initramfs_data.cpio
-# do not try to update files included in initramfs
-$(deps_initramfs): ;
-
-$(deps_initramfs): klibcdirs
-# We rebuild initramfs_data.cpio if:
-# 1) Any included file is newer then initramfs_data.cpio
-# 2) There are changes in which files are included (added or deleted)
-# 3) If gen_init_cpio are newer than initramfs_data.cpio
-# 4) arguments to gen_initramfs.sh changes
-$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio $(deps_initramfs) klibcdirs
+
+$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio
        $(Q)$(initramfs) -l $(ramfs-input) > $(obj)/.initramfs_data.cpio.d
        $(call if_changed,initfs)

to work around the issue.

Here's the error you'll see:

Code: Select all

usr/Makefile:64: *** target pattern contains no '%'.  Stop.
Makefile:964: recipe for target 'usr' failed
make: *** [usr] Error 2

This patch might already be included in gentoo-sources, I built linux-next kernels semi-regularly as I do a lot of dev against it.