Raspberry Pi 3 - 64-bit Gentoo

Message

vanjo9800 · Post by **vanjo9800** » Thu Apr 28, 2016 5:14 pm

Hello, I am trying to put a 64-bit Gentoo on a Raspberry Pi 3.

As I know for other Pi's the process is the following:
1) compile the kernel with crossdev
2) put Stage3
3) put Portage tree
4) add some firmware things from Raspberry Pi Foundation (bootcode.bin, or something like this)
5) boot and configure distcc
6) enjoy Gentoo

However, for the Pi 3, some things are a little more complicated for me:
1) for the kernel - Is it ok to compile it with crossdev aarch64-armv8a-linux-gnu
2) where can I download a stage3, or I should compile it myself; If I should compile it myself, how? and where to start?
3) the Portage tree is the same as for other Pi's I suppose because it does not contain anything specific for the architecture
4) are the firmware things the same, or different ones ( I know that I have to say in cmdline.txt that the processor should enter 64-bit mode
5) Should I boot it on the raspberry, or I should first do it with a virtual machine?

Thank you very much in advance, and sorry for asking for so many things -> that is my first time I try doing something that big!!!

Post by **NeddySeagoon** » Thu Apr 28, 2016 5:42 pm

vanjo9800,

Its aarch64-unknown-linux-gnu not aarch64-armv8a-linux-gnu.

Start out with crossdev. Make it build you a aarch64-unknown-linux-gnu toolchain.
This will give you a directory /usr/aarch64-unknown-linux-gnu which looks very much like a Gentoo root filesystem but with very little in it.

You use your cross toolchain to build all the things that are needed to boot and get to a shell prompt.
The mainline kernel won't work. You need to get one from git.
Likewise you need a git version of Das U-Boot, which is a boot loader.

This thread is a good resource ... all of it and the thread on the Raspberry Pi forums linked from this thread.

If you want to practice some of the skills you will need, try the following
Install a 32 bit Gentoo onto a Raspberry Pi.
Set up cross distcc for the 32 bit install. Build a few packages on the Pi with distcc doing all the hard work.
Take advantage of the 32 bit target root filesystem and cross compile something. Install the cross compiled binaries on the pi and see if they work.

There is a lot to get right all at once to bring up a 64 bit Pi. You will need the skills you develop with the more straight forward 32 bit.

vanjo9800 · Post by **vanjo9800** » Thu Apr 28, 2016 7:41 pm

Thank you for the fast response!!!

I already managed to set up a 32-bit Gentoo on Raspberry Pi B+ with distcc and it works pretty good now. That is why I got enthusiastic about running a 64-bit release on the Raspberry Pi 3. I will use both boards for a cluster, so I think Gentoo is the best choice. For the Raspberry Pi 3, Gentoo is better because I can use the full capacity of the board with a 64-bit operating system compiled directly for the Pi 3, not for ARMv7.

So now, I will cross-compile the kernel and U-Boot and try achieving something like a shell.

For the threads you mentioned, maybe you forgot to paste the hyperlink because I cannot access them.

Thank you again for the support!!!

Post by **NeddySeagoon** » Thu Apr 28, 2016 9:01 pm

vanjo9800,

You need to read all of this thread (the one you have posted in). When you do that, you will find several references, with links to the Raspberry Pi forum.
Read all of that thread too. There is this Wiki page too. HDMI video works now so that a serial console is no longer essential, unless you get nothing on the HDMI video at all.

That page dates from a time when there was only the serial console. The background is still relevant.

vanjo9800 · Post by **vanjo9800** » Thu Apr 28, 2016 9:08 pm

I found a version of U-Boot that already has a configuration for 64-bit Raspberry Pi 3. It is located here: https://github.com/zeldin/u-boot-rpi3

So I downloaded the source and compiled it with:
CROSS_COMPILE=aarch64-unknown-linux-gnu- make rpi_3_defconfig
CROSS_COMPILE=aarch64-unknown-linux-gnu- make -j5

And then as written in the manual:
http://elinux.org/RPi_U-Boot#Copy_U-Boo ... ur_SD_card

copied U-Boot to the SD card.
I have no other files than the U-Boot binary on the SD card. I have the two partitions one 100MB, FAT32, the rest is EXT4. The U-Boot binary is on the FAT32 partition.

However, the Raspberry does not load the U-Boot through HDMI. Just the red light turns on when I connect the power supply. The green light does not change its state.

Can you tell me what is the problem here?

Post by **NeddySeagoon** » Thu Apr 28, 2016 9:16 pm

vanjo9800,

The Pi probably does not boot the way you think it does. The GPU does the booting with the ARM CPU held in reset until everything is loaded.
It sounds like the files that the GPU needs are missing.

Keep in mind that unlike a PC, a Pi has no BIOS in FLASH memory on the motherboard. If its not on the SDCard, its not there. That's why you can't brick a Pi.
You really need to read those references. Its all explained there.

vanjo9800 · Post by **vanjo9800** » Fri Apr 29, 2016 3:01 pm

Hello NeddySeagoon,

Thank you very much for both threads. I carefully read them and now I have a working U-Boot and a working kernel. I also have a working root partition. However, I experience an error I have never seen in the threads.

U-Boot starts and I type boot to boot with bootargs: dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait"

Then the kernel starts booting. OpenRC starts and suddenly everything vanishes away, with the TV saying that there is no signal from the HDMI port and the green light stops flashing again.

What can be a possible problem here?

Thank you in advance!!!

[EDIT]
After capturing the booting process and reading at each entry i figured out the following
Entries that might cause problems:
Clock skew detected with /etc/init.d/functions.sh
Adjusting mtime of /run/openrc/deptree
...
Failed to set the system clock
...
WARNING: clock skew detected!
ERROR: netmount needs service(s) net

And the whole process finishes with
Starting local ...
and immediately the screen goes away.

I think this would be a lot helpful than the initial post.

Post by **NeddySeagoon** » Fri Apr 29, 2016 4:10 pm

vanjo9800,

The clock skew doesn't matter at boot time. There is no hardware clock. The fix is to remove hwclock from its runlevel and replace it with swclock.
swclock saves the current time at shutdown and restores that time to the system time at boot. That makes sure that time is monototonic, which is what matters.
For correct time you need ntp-client and ntpd in the default runlevel.

Two things, both in your /etc/inittab.

First, you need --noclear

Code: Select all

# TERMINALS
#x1:12345:respawn:/sbin/agetty 38400 console linux
c1:12345:respawn:/sbin/agetty 38400  --noclear tty1 linux

That stops the boot sequence clearing the boot messages before it outputs the login prompt.

You still may not see the prompt.
Check that the following block is commented out.

Code: Select all

# SERIAL CONSOLES
#s0:12345:respawn:/sbin/agetty -L 115200 ttyAMA0 vt340
#s1:12345:respawn:/sbin/agetty -L 9600 ttyS1 vt100

and the very last line ...

Code: Select all

# Architecture specific features
#f0:12345:respawn:/sbin/agetty 9600 ttyAMA0 vt100

That puts the console onto /dev/ttyAMA0 on the Pi. If the login prompt goes there, you will never see it on the HDMI output.
For me, it switched the baud rate from 115200 to 9600 so I didn't get a login prompt for a different reason.

It sounds like you are making good progress.

Notice the ordering of the commands. The system will clear the HDMI console, then switch console on you.

vanjo9800 · Post by **vanjo9800** » Fri Apr 29, 2016 4:14 pm

Thank you very much again!!!

I just figured out that and changed it moments before you posted!!!

Now I think I have a working 64-bit Gentoo let's say!!!

vanjo9800 · Post by **vanjo9800** » Fri Apr 29, 2016 7:17 pm

Okay, up to now I have access over TTY through HDMI output, but I face the following problems.

1) Portage does not seem to work!
- eselect profile list gives my current profile
- however, other Portage commands seem to do nothing; the RPI 3 freezes after the command and does nothing
- I have a downloaded portage tree under /usr

2) HTOP gives information about only one core working
- is this the right response, or I have made something completely wrong with the whole installation

3) Bad screen resolution:
- with U-Boot the resolution is perfect, but when the kernel starts booting it changes and now the screen looks ugly
- how can I fix that?
But the last is a minor problem, I think I can fix it on myself?

What is your position for the other problems, mostly for the Portage one, because Gentoo without Portage is let's say nothing...

Thank you in advance!!!

Post by **NeddySeagoon** » Fri Apr 29, 2016 8:01 pm

vanjo9800,

Portage works

Code: Select all

Pi3_64 # eselect profile list  
Available profile symlink targets:
  [1]   default/linux/arm64/13.0
  [2]   default/linux/arm64/13.0/desktop *
  [3]   default/linux/arm64/13.0/desktop/systemd
  [4]   default/linux/arm64/13.0/developer
  [5]   default/linux/arm64/13.0/systemd
  [6]   hardened/linux/musl/arm64

All four cores work.

Code: Select all

top - 20:59:52 up  4:38,  2 users,  load average: 0.23, 0.37, 0.22
Tasks:  95 total,   1 running,  94 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.9 sy,  0.0 ni, 99.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   874064 total,   580976 free,    15436 used,   277652 buff/cache
KiB Swap:  1952764 total,  1952764 free,        0 used.   811076 avail Mem

The simple framebuffer works, for a framebuffer console and Xorg.

Code: Select all

[    99.532] Build Operating System: Linux 4.5.0+ aarch64 Gentoo
[    99.534] Current Operating System: Linux Pi3 64bit 4.5.0+ #3 SMP PREEMPT Sun Apr 17 17:59:43 BST 2016 aarch64
[    99.534] Kernel command line: dwc_otg.lpm_enable=0 console=tty root=/dev/sda1 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
[    99.538] Build Date: 07 April 2016  10:24:35PM
...
[    99.800] (==) FBDEV(0): Default visual is TrueColor
[    99.800] (==) FBDEV(0): Using gamma correction (1.0, 1.0, 1.0)
[    99.801] (II) FBDEV(0): hardware: simple (video memory: 3505kB)
[    99.801] (II) FBDEV(0): checking modes against framebuffer device...
[    99.801] (II) FBDEV(0): checking modes against monitor...
[    99.801] (--) FBDEV(0): Virtual size is 1824x984 (pitch 1824)

That strange Xorg size can probably be fixed in /boot/config.txt.

vanjo9800 · Post by **vanjo9800** » Fri Apr 29, 2016 8:09 pm

Yeah,

I know that Portage should normally work. I asked what can be the cause that it does not work in my case.

For the resolution and htop, I think they can easily be fixed when I get Portage working.

[EDIT]

After looking at the kernel log, I saw that actually I was using only one core:

Code: Select all

CPU1: failed to come online
CPU2: failed to come online
CPU3: failed to come online
Brought up 1 CPUs
SMP: Total of 1 processors activated

Also when tested the SD card speed it turned out to be something like 10MB/s.
Can this cause a problem with Portage?

[EDIT-2]
I think the problem is a bigger one because the kernel reads only 40MB of the available RAM.

What can be the cause?

I recompiled the kernel, but still nothing...

Post by **NeddySeagoon** » Sat Apr 30, 2016 8:36 am

vanjo9800,

Its all in that Raspberry Pi thread. The one core problem starts even before U-Boot is loaded.
If it helps, I've not managed to make U-Boot work on 4 cores either. I'm using a prebuilt U-Boot and stub.

You need four things.
A stub that runs before U-Boot than puts three cores into endless loops (called a spin) that they will exit when the kernel tells them to.
A U-Boot that has a start address at the end of the stub.
A .dtb file that tells the kernel the addresses to use to start the other cores.
A SMP aware kernel.

The slow SD card will make everything slow, it will not prevent portage operating at all.
I've not tested my SDcard speed becase it only houses /boot. Originally, there was no DMA in 64 bit mode but that's fixed now.

Only using 40Mb RAM would probably drive portage to swap. That would give the appearance of hanging.

vanjo9800 · Post by **vanjo9800** » Sat Apr 30, 2016 10:04 am

Ok thank you!

After reading the post in the Raspberry Pi forum about Electron752 setting up his own Gentoo, it became a little bit more clear. However, some questions appeared in my head. How to actually make that stub, like how to add something to the u-boot binary.

I tried to use the u-boot-stubbed.bin of Electron752, but it did not work. Constantly dots appear on the screen and I cannot type my boot commands.

Also what do you mean by an SMP aware kernel - to be justified from the configs because I already did that.

And last, how can I fix the problem with the RAM. It seems that the Raspberry reads only 40MB of the whole RAM of the board. And maybe you are right that this causes Portage to not work.

Thank you in advance!!!

[EDIT]

After using the boot configurations of Electron752, now I have 4 cores and 854 MBs of RAM.

And now even Portage seems to work fine.

YESSS!!! I have an approximately working Gentoo

Post by **NeddySeagoon** » Sat Apr 30, 2016 11:38 am

vanjo9800,

For the stubded u-boot, you build the stub and u-boot separately, then use cat to assemble the two parts.
I can build the parts and put then together but I only ever get the pretty coloured screen.
The trick here is that uboot needs to be built to load at address 0x08000, not its normal 0x0. I guess (I've not checked) that I'm getting that wrong.

The stubbed uboot works for me.

SMP aware kernel means that the kernel has SMP enabled an Max CPUs set to 4 or more. The defaults work.

Maybe the RAM issus will go away on its own once the rest is fixed. There is a boot parameter to set the maximum RAM the kernel will use.
I've not needed it for a very long time. On the Pi you can also set nte GPU/CPU RAM split. The default is for the GPU to have 16Mb.

vanjo9800 · Post by **vanjo9800** » Sat Apr 30, 2016 11:50 am

I preferred to not go that deep to make my own stubbed U Boot.

Electron752's version works pretty fine for me. It is strange that in order to work, I need to have "enable_uart=1" in my config.txt.

However, now I need to set up a gcc and start compiling. As I learned from the guide, GCC cannot me compiled the usual way with:

Code: Select all

aarch64-unknown-linux-gnu-emerge -a gcc

because it gives a linker error.

So it is again something complicated.

Is this thread a good resource for how to compile it successfully: https://forums.gentoo.org/viewtopic-t-1 ... art-0.html .

And is there a way to download a binary GCC in order to just compile GCC successfully on the Pi 3, and then use the compiled GCC for all other packages.

Thank you in advance!!!

Post by **NeddySeagoon** » Sat Apr 30, 2016 12:54 pm

vanjo9800,

gcc is reported to cross compile if you do not give a -march. I've not tested it.

There is my BINHOST if you want a binary or two.

vanjo9800 · Post by **vanjo9800** » Sat Apr 30, 2016 1:21 pm

I do not have a -march in /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf. However, gmp fails to build because of wrong linker. So for me it does not work.

I found a script which will compile it from NoName1337 in the thread I mentioned in the last post and it seem to work fine for now.
Up to now with this configuration, I think that GMP compiled successfully.

vanjo9800 · Post by **vanjo9800** » Sat Apr 30, 2016 7:07 pm

Sadly, GCC with NoName1337's configurations does not compile successfully.
The error I get is the following:

Code: Select all

ib/gcc/aarch64-unknown-linux-gnu/5.3.0/crtendS.o /usr/aarch64-unknown-linux-gnu/usr/lib/../lib64/crtn.o    -Wl,-soname -Wl,libasan.so.2 -o .libs/libasan.so.2.0.0
/usr/libexec/gcc/aarch64-unknown-linux-gnu/ld: skipping incompatible /lib64/libpthread.so.0 when searching for /lib64/libpthread.so.0
/usr/libexec/gcc/aarch64-unknown-linux-gnu/ld: cannot find /lib64/libpthread.so.0
/usr/libexec/gcc/aarch64-unknown-linux-gnu/ld: skipping incompatible /usr/lib64/libpthread_nonshared.a when searching for /usr/lib64/libpthread_nonshared.a
/usr/libexec/gcc/aarch64-unknown-linux-gnu/ld: cannot find /usr/lib64/libpthread_nonshared.a
collect2: error: ld returned 1 exit status
libtool: install: error: relink `libasan.la' with the above command before installing it
Makefile:384: recipe for target 'install-toolexeclibLTLIBRARIES' failed
make[4]: *** [install-toolexeclibLTLIBRARIES] Error 1
make[4]: Leaving directory '/mnt/raspberryroot/root/build-gcc/aarch64-unknown-linux-gnu/libsanitizer/asan'
Makefile:557: recipe for target 'install-am' failed
make[3]: *** [install-am] Error 2
make[3]: Leaving directory '/mnt/raspberryroot/root/build-gcc/aarch64-unknown-linux-gnu/libsanitizer/asan'
Makefile:437: recipe for target 'install-recursive' failed
make[2]: *** [install-recursive] Error 1
make[2]: Leaving directory '/mnt/raspberryroot/root/build-gcc/aarch64-unknown-linux-gnu/libsanitizer'
Makefile:11991: recipe for target 'install-target-libsanitizer' failed
make[1]: *** [install-target-libsanitizer] Error 2
make[1]: Leaving directory '/mnt/raspberryroot/root/build-gcc'
Makefile:2286: recipe for target 'install' failed
make: *** [install] Error 2
FAILED COMMAND: make install

What could be the problem here?

Post by **NeddySeagoon** » Sat Apr 30, 2016 7:19 pm

vanjo9800,

If the first error is really

Code: Select all

/usr/libexec/gcc/aarch64-unknown-linux-gnu/ld: skipping incompatible /lib64/libpthread.so.0 when searching for /lib64/libpthread.so.0
/usr/libexec/gcc/aarch64-unknown-linux-gnu/ld: cannot find /lib64/libpthread.so.0

it says that /lib64/libpthread.so.0 exists but it has a problem. That lib is provided by glibc.
You either have a problem with glbc or with the gcc you are using.

What does file have to say?

Code: Select all

roy@Pi3 64bit ~ $ file  /lib64/libpthread-2.23.so
/lib64/libpthread-2.23.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, not stripped

vanjo9800 · Post by **vanjo9800** » Sat Apr 30, 2016 7:28 pm

I understood its meaning, but I am cross-compiling GCC for the Raspberry Pi 3 on my desktop using aarch64-unknown-linux-gnu. That is why I am not sure which libpthread causes the error. The one on my local GCC, or the one which I cross-compiled.

krinn · Post by **krinn** » Sat Apr 30, 2016 7:46 pm

well, sorry you are not cross-compiling there, and your /lib64/libpthread.so.0 is certainly the one from your arch, and not the one it should be (some kind of /usr/aarch64-unknown-linux-gnu/lib64/libpthread.so.0)
if you had put its content like NeddySeagoon has done, you would have certainly already seen it doesn't say ARM aarch64 but more probably whatever is your cross-compiler host arch is using.
Show NeddySeagoon's given command (the file...) and also show the same command but with /usr/aarch64-unknown-linux-gnu/lib64/libpthread-??-??.so
(and not the libpthread.so.0, we don't really care to know that libpthread.so.0 is a symlink, but we care about where is pointing at)

vanjo9800 · Post by **vanjo9800** » Sat Apr 30, 2016 7:49 pm

Yeah, after I issued file

Code: Select all

/lib64/libpthread-2.23.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

this was the response.

However, that whole process with the error happened while cross-compiling GCC for the Raspberry Pi 3. So the fact that it uses my native files for the desktop means that the build is not working.

So from where can I find a way to cross-compile GCC for my Raspberry Pi 3?

krinn · Post by **krinn** » Sat Apr 30, 2016 7:55 pm

From my knowledge, gcc build xgcc and use xgcc to build the final gcc.
What this mean, is that it just cannot be cross-build, because your x86-64 host will be able to build the xgcc (arm64 host) but will fail to run it.

But i'm unsure if there's a way to bypass that, NeddySeagoon will certainly have the real answer.
I would consider NeddySeagoon binhost and steal his toolchain (yeah i know it was a kindly offered, but i prefer to pretend i steal it, like an old punk).

Post by **NeddySeagoon** » Sat Apr 30, 2016 8:10 pm

vanjo9800,

I didn't cross compile gcc. There is a link a few pages back to a non gentoo native gcc for aarch64. It installs into /opt.
I used that to build the gentoo gcc on the Pi.
There are also some gcc patches which already seemed to be applied for me.

krinn,
gcc used to cross compile. In fact, the tool chain was the least cross compile hostile set of packages out there.