Trying to debug my custom initramfs: ENOENT [SOLVED]

[Zucca]

Firstly, this is a amd64/musl/llvm system.

I've managed to set up almost everything in order, and I decided (yet again) to build a completely custom initramfs.
For some reason Linux, the kernel, cannot execute /init on my initramfs, so I extracted the image and tried to chroot into it.
Whatever I pass to chroot as the command I get "no such file or directory" -error.
So I started to dig deeper:

Code: Select all

# strace chroot /home/zucca/initramfstest /bin/bb
execve("/usr/bin/chroot", ["chroot", "/home/zucca/initramfstest", "/bin/bb"], 0x7ffe7011d220 /* 49 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7eff32cb7358) = 0
set_tid_address(0x7eff32cb4d88)         = 5383
brk(NULL)                               = 0x55d0fa79d000
brk(0x55d0fa79f000)                     = 0x55d0fa79f000
mmap(0x55d0fa79d000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x55d0fa79d000
mprotect(0x7eff32cb3000, 4096, PROT_READ) = 0
mprotect(0x55d0dbf1e000, 4096, PROT_READ) = 0
readlink("/home", 0x7ffee236af40, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/zucca", 0x7ffee236af40, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/zucca/initramfstest", 0x7ffee236af40, 1023) = -1 EINVAL (Invalid argument)
chroot("/home/zucca/initramfstest")     = 0
chdir("/")                              = 0
execve("/bin/bb", ["/bin/bb"], 0x7ffee236b4e8 /* 49 vars */) = -1 ENOENT (No such file or directory)
fcntl(1, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
writev(2, [{iov_base="chroot: ", iov_len=8}, {iov_base=NULL, iov_len=0}], 2chroot: ) = 8
writev(2, [{iov_base="", iov_len=0}, {iov_base="failed to run command \342\200\230/bin/bb"..., iov_len=35}], 2failed to run command ‘/bin/bb’) = 35
writev(2, [{iov_base=": No such file or directory", iov_len=27}, {iov_base=NULL, iov_len=0}], 2: No such file or directory) = 27
writev(2, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
close(1)                                = 0
close(2)                                = 0
exit_group(127)                         = ?
+++ exited with 127 +++

... those readlink calls getting failed doesn't seem like a problem. I guess chroot checks the path for symlinks. The tests failing indicate there's no symlink... That's how I read it.
Anyway. For the tests I've included every file musl package owns (qfile musl >> filelist). So I should have at least all the musl related dependencies there.

Any pointers?
Yes. My busybox and bash, etc aren't static. This same method has been working on my amd64/glibc/gcc boxes, so suspect llvm and/or musl bring something other to the table which I haven't noticed.

[pingtoo]

If /bin/bb actually exist in the chroot dir. Then may be add "-ff" to strace to see if the child process will show why it complaint missing something.

[Zucca]

Adding that doesn't produce any different results. I also tried with /bin/bash (which I added to the image because of this), but nope. It says no such file.

[Hu]

In my experience, the most common reason for ENOENT when trying to run a program that does exist is that the program specifies a program interpreter that does not exist. On an amd64 glibc-based system, the normal program interpreter for an ELF would be /lib64/ld-linux-x86-64.so.2. What does scanelf -i files show for the chroot's /bin/bash and /bin/bb? Does the shown interpreter exist in the chroot at the shown path?

[NeddySeagoon]

Zucca,

You need all the bits

Code: Select all

$ lddtree /bin/bash
/bin/bash (interpreter => /lib64/ld-linux-x86-64.so.2)
    libreadline.so.8 => /usr/lib64/libreadline.so.8
        libtinfow.so.6 => /lib64/libtinfow.so.6
    libtinfo.so.6 => /lib64/libtinfo.so.6
    libc.so.6 => /lib64/libc.so.6

or

Code: Select all

$ lddtree /bin/busybox 
/bin/busybox (interpreter => None)

busybox is static as it works when nothing else does.
Here bb is a symbolic link ... heh, so is busybox.
They both point to /ginit

[zen_desu]

Did you make sure to include /etc/ld-musl-{arch}.path?

[Zucca]

Not a single error indicated I had something missing, but...

Code: Select all

qlist musl | grep '^/lib'

... returned nothing (strange...). So I manually added few essential files from /lib.
Also thanks zen_desu, I thought I had that file, but at some point I cleared my file list and forgot to re-add that.

Neddy, it seems my initramfs script needs some improvements. :) Indeed it failed to gather few dependencies. It uses ldd internally.
Since I now got busybox working, but bash fails to start. libreadline is missing some dependencies. Although I don't need bash there. busybox is enough.

I guess I need to add all the dependencies manually just to be sure.

[Zucca]

I think I've progressed a bit. I now have a somewhat working initramfs image.
But I'm bothered about one thing:

Code: Select all

# qlist musl | grep ^/lib
# equery f musl | grep ^/lib
# qfile /lib/libc.so
sys-libs/musl: /usr/lib/libc.so
# equery b /lib/libc.so
 * Searching for /lib/libc.so ... 
sys-libs/musl-1.2.5-r3 (/usr/lib/libc.so)
# file /lib/libc.so
/lib/libc.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

Now... there's something I don't understand how portage associates files to a package.
qfile and equery f only scan for database for the package. The files in /lib are not recorded to this database? Right? But are later hardlinked/reflinked/copied to /lib too for compability?

While my script doesn't use these portage tools (q and equery), this still might lead to why it didn't gather the required dependencies.

EDIT: Obviously /lib is a symlink to /usr/lib :facepalm:
Ok. Continuing... Nothing to see here. ':D

[Zucca]

So to conclude this:

• I commonly use my own script to gather components to my initramfs image.
  It automatically pulls in libraries which ldd lists. However musl version of ldd does not display a header when processing more than one file. This header only consisted of the filename being processed.
  The solution to this was a simple

  Code: Select all

  xargs -n 1 ldd

  ... Not optimal but works.
  
  Also busybox awk segfaulted in the chroot for some reason. But not outside. My quick fix for it was to recompile busybox with very basic CFLAGS. This seemed strange...
  
  Lastly there was a bug in my gawk code which adds all the path components to the list in right order.

So now it "works" again. It still cannot magically pull in all the dependencies (like on case of, say shell scripts) as it solely relies on ldd, but that's just enough for me.

[zen_desu]

So to conclude this:

• I commonly use my own script to gather components to my initramfs image.
  It automatically pulls in libraries which ldd lists. However musl version of ldd does not display a header when processing more than one file. This header only consisted of the filename being processed.
  The solution to this was a simple

  Code: Select all

  xargs -n 1 ldd

  ... Not optimal but works.
  
  Also busybox awk segfaulted in the chroot for some reason. But not outside. My quick fix for it was to recompile busybox with very basic CFLAGS. This seemed strange...
  
  Lastly there was a bug in my gawk code which adds all the path components to the list in right order.

So now it "works" again. It still cannot magically pull in all the dependencies (like on case of, say shell scripts) as it solely relies on ldd, but that's just enough for me.

Does lddtree work better?

[Zucca]

Does lddtree work better?

The very first "pre-alpha" versions I developed tried to use lddtree. I cannot recall what was the problem with lddtree.