arm chroot on x86_64 host [solved]

[dinominant]

I'm trying to build an arm stage4 template on my x86_64 box. I have been following Crossdev qemu-static-user-chroot and Compiling with qemu user chroot to get going, but I am running into a problem when I finally chroot into the stage3 environment.

My static qemu-arm binary appears to be installed and working on the x86_64 host as I can run it just fine on it's own. I can also use it to directly run the statically linked busybox inside the stage3 tarball:

Code: Select all

x86_64-host ~ # file armv6j_hardfp-20140723/bin/busybox
armv6j_hardfp-20140723/bin/busybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, for GNU/Linux 2.6.16, stripped
x86_64-host ~ # qemu-arm armv6j_hardfp-20140723/bin/busybox cat --help
BusyBox v1.21.0 (2014-06-05 15:17:01) multi-call binary.

Usage: cat [FILE]...

Concatenate FILEs and print them to stdout

I also have the /etc/init.d/qemu-binfmt service running and the qemu-arm interpreter is working because I can run that same arm busybox binary directly:

Code: Select all

x86_64-host ~ # armv6j_hardfp-20140723/bin/busybox cat --help
BusyBox v1.21.0 (2014-06-05 15:17:01) multi-call binary.

Usage: cat [FILE]...

Concatenate FILEs and print them to stdout

However, when I mount /proc, /dev, /sys, and attempt to chroot into the arm environment I get this error:

Code: Select all

x86_64-host ~ # chroot armv6j_hardfp-20140723 /bin/bash
/bin/bash: /bin/bash: cannot execute binary file

I've extracted the statically linked qemu-arm binary from the qemu-user-static_2.0.0+dfsg-6_arm64.deb package and that is what is in /usr/bin/qemu-arm on both the host and chroot environment. Do you have any ideas on what I might be missing and why I can't chroot into the stage3 environment?

[Telemin]

Hi there, I've been messing about with this too, and ran into the same problem. I'm still working on it, but I *think* I may have found a fix. I need to get on the embedded mailing list and ask some questions to be sure I understand what is going on though.

So, to what I know:

• The message

  Code: Select all

   /bin/bash: /bin/bash: cannot execute binary file

  means that the system can't find an appropriate interpreter for the file, or the interpreter is failing, I can't be sure which.
• I found a "raspbian" VMDK image (read ubuntu image seemingly used for raspbian development), in which an arm chroot does work: http://archive.raspbian.org/qemu/rpi-qe ... ev.tar.bz2, launch the vmdk in qemu etc. and take a look in the /chroots dir
• The difference between the debian setup and gentoo comes in the binfmt_misc handlers:

Gentoo:

Code: Select all

# cat /proc/sys/fs/binfmt_misc/arm 
enabled
interpreter /usr/bin/qemu-arm
flags: P
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00ffffffffffff00fffeffffff

Raspbian-ubuntu-dev:

Code: Select all

# cat /proc/sys/fs/binfmt_misc/qemu-arm 
enabled
interpreter /usr/bin/qemu-arm
flags: OC
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

Notice in particular that the mode flag for debian is OC, and for gentoo is P. Now I will admit that this is the part that I don't full understand, however taking a look at the kernel.org documentation for binfmt_misc, it appears that the raspbian image is using the new mode, and gentoo is using the old legacy mode. Changing this mode on gentoo fixes the issue, but I am unsure at what cost, hence I say I need to check this out with people who know more than me, probably the embedded mailing list.

There is also a difference in one byte of the mask, which I matched in my setup also. Since I don't really have enough knowledge of the ELF format, I am not 100% sure what effect this has, apart from to say the debian version is somehow stricter in what it matches (since setting the magic to 00 and the mask to 00 effectively means "ignore this byte"), although looking at wikipedia it seems the debian form checks the last byte of the padding block is set to 0x00, the gentoo form ignores it.

In the meantime however, by patching the provided /etc/init.d/qemu-binfmt script to match the raspbian settings, I have been able to get my chroot minimally working. My next step is to provide a native binary arm compiler and a native python interpreter to the chroot, but it remains to be seen how easy that will be or when I get the time to sit down and sort it out. My intention is eventually to get a working lxc template written for this and posted to github+wiki, but at the moment I am in a very preliminary stage.

Hope this is of some help

-Telemin-

P.S watch out for the step

Code: Select all

chroot . /bin/busybox mdev -s

it can (probably will) trash your host /dev requiring a reboot. I have made do with bind mounting dev and not running mdev

[dinominant]

Thanks for the help Telemin. That fixed it for me! Modifying the /etc/init.d/qemu-binfmt script to use OC instead of P fixed it for me.

In /etc/init.d/qemu-binfmt change this

Code: Select all

   if [ $cpu != "arm" -a -x "/usr/bin/qemu-arm" ] ; then
      echo   ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\x00\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:P' > /proc/sys/fs/binfmt_misc/register
   fi

To this

Code: Select all

   if [ $cpu != "arm" -a -x "/usr/bin/qemu-arm" ] ; then
      echo   ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\x00\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:OC' > /proc/sys/fs/binfmt_misc/register
   fi

I also updated https://bugs.gentoo.org/show_bug.cgi?id=512780 so it can be changed to confirmed and hopefully fixed in the actual ebuild.