Raspberry Pi 3 - 64-bit Gentoo

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Hello, I am trying to put a 64-bit Gentoo on a Raspberry Pi 3.

As I know for other Pi's the process is the following:
1) compile the kernel with crossdev
2) put Stage3
3) put Portage tree
4) add some firmware things from Raspberry Pi Foundation (bootcode.bin, or something like this)
5) boot and configure distcc
6) enjoy Gentoo

However, for the Pi 3, some things are a little more complicated for me:
1) for the kernel - Is it ok to compile it with crossdev aarch64-armv8a-linux-gnu
2) where can I download a stage3, or I should compile it myself; If I should compile it myself, how? and where to start?
3) the Portage tree is the same as for other Pi's I suppose because it does not contain anything specific for the architecture
4) are the firmware things the same, or different ones ( I know that I have to say in cmdline.txt that the processor should enter 64-bit mode
5) Should I boot it on the raspberry, or I should first do it with a virtual machine?

Thank you very much in advance, and sorry for asking for so many things -> that is my first time I try doing something that big!!!

NeddySeagoon · Posted: Thu Apr 28, 2016 5:42 pm Post subject:

vanjo9800,

Its aarch64-unknown-linux-gnu not aarch64-armv8a-linux-gnu.

Start out with crossdev. Make it build you a aarch64-unknown-linux-gnu toolchain.
This will give you a directory /usr/aarch64-unknown-linux-gnu which looks very much like a Gentoo root filesystem but with very little in it.

You use your cross toolchain to build all the things that are needed to boot and get to a shell prompt.
The mainline kernel won't work. You need to get one from git.
Likewise you need a git version of Das U-Boot, which is a boot loader.

This thread is a good resource ... all of it and the thread on the Raspberry Pi forums linked from this thread.

If you want to practice some of the skills you will need, try the following
Install a 32 bit Gentoo onto a Raspberry Pi.
Set up cross distcc for the 32 bit install. Build a few packages on the Pi with distcc doing all the hard work.
Take advantage of the 32 bit target root filesystem and cross compile something. Install the cross compiled binaries on the pi and see if they work.

There is a lot to get right all at once to bring up a 64 bit Pi. You will need the skills you develop with the more straight forward 32 bit.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Thank you for the fast response!!!

I already managed to set up a 32-bit Gentoo on Raspberry Pi B+ with distcc and it works pretty good now. That is why I got enthusiastic about running a 64-bit release on the Raspberry Pi 3. I will use both boards for a cluster, so I think Gentoo is the best choice. For the Raspberry Pi 3, Gentoo is better because I can use the full capacity of the board with a 64-bit operating system compiled directly for the Pi 3, not for ARMv7.

So now, I will cross-compile the kernel and U-Boot and try achieving something like a shell.

For the threads you mentioned, maybe you forgot to paste the hyperlink because I cannot access them.

Thank you again for the support!!!

NeddySeagoon · Posted: Thu Apr 28, 2016 9:01 pm Post subject:

vanjo9800,

You need to read all of this thread (the one you have posted in). When you do that, you will find several references, with links to the Raspberry Pi forum.
Read all of that thread too. There is this Wiki page too. HDMI video works now so that a serial console is no longer essential, unless you get nothing on the HDMI video at all.

That page dates from a time when there was only the serial console. The background is still relevant.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

I found a version of U-Boot that already has a configuration for 64-bit Raspberry Pi 3. It is located here: https://github.com/zeldin/u-boot-rpi3

So I downloaded the source and compiled it with:
CROSS_COMPILE=aarch64-unknown-linux-gnu- make rpi_3_defconfig
CROSS_COMPILE=aarch64-unknown-linux-gnu- make -j5

And then as written in the manual:
http://elinux.org/RPi_U-Boot#Copy_U-Boot_to_your_SD_card

copied U-Boot to the SD card.
I have no other files than the U-Boot binary on the SD card. I have the two partitions one 100MB, FAT32, the rest is EXT4. The U-Boot binary is on the FAT32 partition.

However, the Raspberry does not load the U-Boot through HDMI. Just the red light turns on when I connect the power supply. The green light does not change its state.

Can you tell me what is the problem here?

NeddySeagoon · Posted: Thu Apr 28, 2016 9:16 pm Post subject:

vanjo9800,

The Pi probably does not boot the way you think it does. The GPU does the booting with the ARM CPU held in reset until everything is loaded.
It sounds like the files that the GPU needs are missing.

Keep in mind that unlike a PC, a Pi has no BIOS in FLASH memory on the motherboard. If its not on the SDCard, its not there. That's why you can't brick a Pi.
You really need to read those references. Its all explained there.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Hello NeddySeagoon,

Thank you very much for both threads. I carefully read them and now I have a working U-Boot and a working kernel. I also have a working root partition. However, I experience an error I have never seen in the threads.

U-Boot starts and I type boot to boot with bootargs: dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait"

Then the kernel starts booting. OpenRC starts and suddenly everything vanishes away, with the TV saying that there is no signal from the HDMI port and the green light stops flashing again.

What can be a possible problem here?

Thank you in advance!!!

[EDIT]
After capturing the booting process and reading at each entry i figured out the following
Entries that might cause problems:
Clock skew detected with /etc/init.d/functions.sh
Adjusting mtime of /run/openrc/deptree
...
Failed to set the system clock
...
WARNING: clock skew detected!
ERROR: netmount needs service(s) net

And the whole process finishes with
Starting local ...
and immediately the screen goes away.

I think this would be a lot helpful than the initial post.

NeddySeagoon · Posted: Fri Apr 29, 2016 4:10 pm Post subject:

vanjo9800,

The clock skew doesn't matter at boot time. There is no hardware clock. The fix is to remove hwclock from its runlevel and replace it with swclock.
swclock saves the current time at shutdown and restores that time to the system time at boot. That makes sure that time is monototonic, which is what matters.
For correct time you need ntp-client and ntpd in the default runlevel.

Two things, both in your /etc/inittab.

First, you need --noclear

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Thank you very much again!!!

I just figured out that and changed it moments before you posted!!!

Now I think I have a working 64-bit Gentoo let's say!!!

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Okay, up to now I have access over TTY through HDMI output, but I face the following problems.

1) Portage does not seem to work!
- eselect profile list gives my current profile
- however, other Portage commands seem to do nothing; the RPI 3 freezes after the command and does nothing
- I have a downloaded portage tree under /usr

2) HTOP gives information about only one core working
- is this the right response, or I have made something completely wrong with the whole installation

3) Bad screen resolution:
- with U-Boot the resolution is perfect, but when the kernel starts booting it changes and now the screen looks ugly
- how can I fix that?
But the last is a minor problem, I think I can fix it on myself?

What is your position for the other problems, mostly for the Portage one, because Gentoo without Portage is let's say nothing...

Thank you in advance!!!

NeddySeagoon · Posted: Fri Apr 29, 2016 8:01 pm Post subject:

vanjo9800,

Portage works

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Yeah,

I know that Portage should normally work. I asked what can be the cause that it does not work in my case.

For the resolution and htop, I think they can easily be fixed when I get Portage working.

[EDIT]

After looking at the kernel log, I saw that actually I was using only one core:

NeddySeagoon · Posted: Sat Apr 30, 2016 8:36 am Post subject:

vanjo9800,

Its all in that Raspberry Pi thread. The one core problem starts even before U-Boot is loaded.
If it helps, I've not managed to make U-Boot work on 4 cores either. I'm using a prebuilt U-Boot and stub.

You need four things.
A stub that runs before U-Boot than puts three cores into endless loops (called a spin) that they will exit when the kernel tells them to.
A U-Boot that has a start address at the end of the stub.
A .dtb file that tells the kernel the addresses to use to start the other cores.
A SMP aware kernel.

The slow SD card will make everything slow, it will not prevent portage operating at all.
I've not tested my SDcard speed becase it only houses /boot. Originally, there was no DMA in 64 bit mode but that's fixed now.

Only using 40Mb RAM would probably drive portage to swap. That would give the appearance of hanging.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Ok thank you!

After reading the post in the Raspberry Pi forum about Electron752 setting up his own Gentoo, it became a little bit more clear. However, some questions appeared in my head. How to actually make that stub, like how to add something to the u-boot binary.

I tried to use the u-boot-stubbed.bin of Electron752, but it did not work. Constantly dots appear on the screen and I cannot type my boot commands.

Also what do you mean by an SMP aware kernel - to be justified from the configs because I already did that.

And last, how can I fix the problem with the RAM. It seems that the Raspberry reads only 40MB of the whole RAM of the board. And maybe you are right that this causes Portage to not work.

Thank you in advance!!!

[EDIT]

After using the boot configurations of Electron752, now I have 4 cores and 854 MBs of RAM.

And now even Portage seems to work fine.

YESSS!!! I have an approximately working Gentoo

NeddySeagoon · Posted: Sat Apr 30, 2016 11:38 am Post subject:

vanjo9800,

For the stubded u-boot, you build the stub and u-boot separately, then use cat to assemble the two parts.
I can build the parts and put then together but I only ever get the pretty coloured screen.
The trick here is that uboot needs to be built to load at address 0x08000, not its normal 0x0. I guess (I've not checked) that I'm getting that wrong.

The stubbed uboot works for me.

SMP aware kernel means that the kernel has SMP enabled an Max CPUs set to 4 or more. The defaults work.

Maybe the RAM issus will go away on its own once the rest is fixed. There is a boot parameter to set the maximum RAM the kernel will use.
I've not needed it for a very long time. On the Pi you can also set nte GPU/CPU RAM split. The default is for the GPU to have 16Mb.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

I preferred to not go that deep to make my own stubbed U Boot.

Electron752's version works pretty fine for me. It is strange that in order to work, I need to have "enable_uart=1" in my config.txt.

However, now I need to set up a gcc and start compiling. As I learned from the guide, GCC cannot me compiled the usual way with:

NeddySeagoon · Posted: Sat Apr 30, 2016 12:54 pm Post subject:

vanjo9800,

gcc is reported to cross compile if you do not give a -march. I've not tested it.

There is my BINHOST if you want a binary or two.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

I do not have a -march in /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf. However, gmp fails to build because of wrong linker. So for me it does not work.

I found a script which will compile it from NoName1337 in the thread I mentioned in the last post and it seem to work fine for now.
Up to now with this configuration, I think that GMP compiled successfully.

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Sadly, GCC with NoName1337's configurations does not compile successfully.
The error I get is the following:

NeddySeagoon · Posted: Sat Apr 30, 2016 7:19 pm Post subject:

vanjo9800,

If the first error is really

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

I understood its meaning, but I am cross-compiling GCC for the Raspberry Pi 3 on my desktop using aarch64-unknown-linux-gnu. That is why I am not sure which libpthread causes the error. The one on my local GCC, or the one which I cross-compiled.

krinn · Watchman Joined: 02 May 2003 Posts: 7470

well, sorry you are not cross-compiling there, and your /lib64/libpthread.so.0 is certainly the one from your arch, and not the one it should be (some kind of /usr/aarch64-unknown-linux-gnu/lib64/libpthread.so.0)
if you had put its content like NeddySeagoon has done, you would have certainly already seen it doesn't say ARM aarch64 but more probably whatever is your cross-compiler host arch is using.
Show NeddySeagoon's given command (the file...) and also show the same command but with /usr/aarch64-unknown-linux-gnu/lib64/libpthread-??-??.so
(and not the libpthread.so.0, we don't really care to know that libpthread.so.0 is a symlink, but we care about where is pointing at)

vanjo9800 · Tux's lil' helper Joined: 16 Mar 2015 Posts: 76

Yeah, after I issued file

krinn · Watchman Joined: 02 May 2003 Posts: 7470

From my knowledge, gcc build xgcc and use xgcc to build the final gcc.
What this mean, is that it just cannot be cross-build, because your x86-64 host will be able to build the xgcc (arm64 host) but will fail to run it.

But i'm unsure if there's a way to bypass that, NeddySeagoon will certainly have the real answer.
I would consider NeddySeagoon binhost and steal his toolchain (yeah i know it was a kindly offered, but i prefer to pretend i steal it, like an old punk).

NeddySeagoon · Posted: Sat Apr 30, 2016 8:10 pm Post subject:

vanjo9800,

I didn't cross compile gcc. There is a link a few pages back to a non gentoo native gcc for aarch64. It installs into /opt.
I used that to build the gentoo gcc on the Pi.
There are also some gcc patches which already seemed to be applied for me.

krinn,
gcc used to cross compile. In fact, the tool chain was the least cross compile hostile set of packages out there.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.