Slowly compilation on RPi3 32bit mode

[madurani]

Could you please help me configure correctly make.conf and solve speed issue of compilation? I installed gentoo according to https://wiki.gentoo.org/wiki/Raspberry_Pi . After booting, system works fine. But when I am trying to compile/emerge some packages it works extremely slowly. CPU is running on 900MHz, but performance during compiling is too low. Example:

I am trying to compile sys-devel/distcc (compilation is running more than 2 hours now and still hasn't finished)

uptime:
02:14:46 up 2:14, 4 users, load average: 1.10, 1.14, 1.11
There are 4 CPU and load everage is onli 1.1 !!!!!. In make.conf I have value
MAKEOPTS="-j5". It looks like it is not active

MY make.conf:

CHOST="armv7a-hardfloat-linux-gnueabi"
PORTDIR="/usr/portage"
DISTDIR="/usr/portage/distfiles"
PKGDIR="/usr/portage/packages"
CCEPT_KEYWORDS="arm ~arm"


# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C


# Raspberry Pi 2, or Raspberry Pi 3 running in 32 bit mode (recommended):
CFLAGS="-O2 -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j5"
#EMERGE_DEFAULT_OPTS="--jobs=5 --load-average=4"

# Additional USE flags in addition to those specified by the current profile.
USE="bindist -mudflap -sanitize"
USE="${USE} bluetooth egl gles1 gles2 lock thunar qt4 ffmpeg"
USE="${USE} -gnome -kde"


VIDEO_CARDS="fbdev vc4"
INPUT_DEVICES="evdev synaptics"

[NeddySeagoon]

madurani,

Welcome to Gentoo.

Where did you get your kernel from?
You may be using the powersave CPU governer, that locks the CPU to 600 MHz.
Follow the steps https://forums.gentoo.org/viewtopic-p-8035350.html#8035350]here to check.
That thread is about 64 bit on Raspberry Pi3 but the CPU governor control is the same for everyone.

The Pi3 does not support 900MHz operation, only 600MHz and 1.2GHz.

The microSD card is slow. Speeds vary from card to card. I've sees 5MB/sec to 25Mb/sec.
As the SD card is clocked at 50MHz, 25Mb/sec is as good as it gets. Once the SD card has to do erase before write, the write speeds will fall off.
Don't put swap or the portage build space on the SD card.
An external USB HDD is not a lot faster, I get 30MB/sec but there is no erase time. I have everything but /boot there.

Some packages to not build correctly with MAKEOPTS any more than -j1. Thats usually set in the ebuild if its needed.
Some stages of the build process are single threaded too. e.g. configure and install. I'm not suer about linking.

distcc depends on other packages. To install distcc, you will be installing its dependences too. Notice that distcc has a gtk USE flag.
That provides the GUI distcc-mon application. If you don't have a GUI installed yet, that will pull in most of Xorg. You won't want that.
In fact, as there is only 1G RAM, you won't be running a GUI when you are compiling, so you way as well set USE=-gtk for distcc, on a per package basis.

Please install wgetpaste and use it to put your dmesg onto a pastebin. It won't fit into a post.

Code: Select all

emerge wgetpaste
dmesg | wgetpaste

and tell the URL.
This will show lots of interesting things about the startup.

-- edit --

The Raspberry Pi Foundation maintains a branch of the Linux kernel that will run on the Raspberry Pi, including a compiled version which we use here:

Ahh. That kernel does default to the powersave CPU governor.

[madurani]

Hello NeddySeagoon

Here is requested output:
http://codepad.org/r2zI9yNh

localhost portage # uname -a
Linux localhost 4.4.50-raspberrypi-v7+ #2 SMP Tue Mar 7 21:05:56 CET 2017 armv7l BCM2709 GNU/Linux

There variables were use(below), For emerging is onlly one package.

USE="-gtk -ipv6" emerge --ask sys-devel/distcc

alculating dependencies... done!
[ebuild N ] sys-devel/distcc-3.2_rc1-r4 USE="-crossdev -gnome -gssapi -gtk (-hardened) -ipv6 (-selinux) -xinetd -zeroconf" PYTHON_TARGETS="python2_7"

Would you like to merge these packages? [Yes/No]

Kernel was compiled on other system via cross compilation according https://wiki.gentoo.org/wiki/Raspberry_Pi .


over clocking settings from /boot/config.txt
##Medium
arm_freq=900
core_freq=333
sdram_freq=450
over_voltage=2
force_turbo=1

[NeddySeagoon]

madurani,

Code: Select all

[    0.937852] mmc0: Problem switching card into high-speed mode!

Try another microSD card.

Code: Select all

[    0.947718] mmcblk0: mmc0:0001 SD16G 14.6 GiB (quirks 0x80000000)

That's a 16G card. Maybe it doesn't support high speed mode.
Check the speed class marked on your card.
The Pi3 supports C10 or U1. That's C with the 10 written inside or U with the 1 written inside.
U3 (faster still) should work but I'm not sure if you see the speed benefit.
Lower C numbers are slower.

Code: Select all

##Medium
arm_freq=900
core_freq=333
sdram_freq=450
over_voltage=2
force_turbo=1

This will confuse a Pi3. The numbers look like they are for the original Pi. That has a 700MHz CPU.

force_turbo=1 is OK but remove the rest or find some Pi3 specific overclock information.
force_turbo=1 is the same as using the kernel performance cpu governor.
On demand lets the CPU run at 600MHz when it's not busy and 1.2GHz when it is.
The Pi 3 CPU will need some cooling when its working hard at 1.2GHz, never mind overclocked and over volted.

[madurani]

SDAcard should be >>> Kingston microSDHC 16GB UHS-I U3 Action Card 90R/45W

I nstalled sys-power/cpupower and in /boot/config.txt set arm_freq=1200. now is frequency changed between 600-1200MHz. But for compilation it wasn't useful.

localhost ~ # cpupower frequency-info
analyzing CPU 0:
driver: BCM2835 CPUFreq
CPUs which run at the same hardware frequency: 0 1 2 3
CPUs which need to have their frequency coordinated by software: 0 1 2 3
maximum transition latency: 355 us
hardware limits: 600 MHz - 1.20 GHz
available frequency steps: 600 MHz, 1.20 GHz
available cpufreq governors: conservative ondemand userspace powersave performance
current policy: frequency should be within 600 MHz and 1.20 GHz.
The governor "ondemand" may decide which speed to use
within this range.
current CPU frequency: 1.20 GHz (asserted by call to hardware)

[NeddySeagoon]

madurani,

That SD card should be good.

Install

Code: Select all

sys-apps/hdparm

When nothing else is using the SD card,
run

Code: Select all

hdparm -tT /dev/mmcblk0

It will show the buffered and unbuffered sequential read speeds.
The buffered speed indicates the RAM read speed, the unbuffered speed is the sdcard.

For compiling, -j5 may be past the limit where the speedup by having more parallel compiles is more that outweighed by the lack of RAM, so the system is driven to use swap.
Swap is incredibly slow compared to RAM, so more parallel makes actually lead to a significant slowdown.
The use of the swap partition is only the tip of the swap iceberg. RAM contents that have a permanent home on disk (or SD card) can be dropped to free RAM.
They are reloaded as required, which is a lot of extra reads.

Some packages push gcc to use more than 512MB RAM per gcc instance. At -j5 thats over 2.5G RAM required.
MAKEOPTS can and should be set per package for things like that.
Keep an eye on your swap use. If its 100s MB, during building, you have run out of RAM and are using swap.
A few things need -j1 and still swap.

[madurani]

Now is running emerge -ave system approximately one hour. Here is output related with ram usage(I can see 128m free ram / swap isn't in use):

Code: Select all

localhost ~ # free -m
              total        used        free      shared  buff/cache   available
Mem:            957          86         128           0         742         856
Swap:           236           0         235

Code: Select all

localhost ~ # uptime
 02:32:36 up  2:32,  5 users,  load average: 1.06, 1.23, 1.31

I dont understand why is load average so low 1.06 when is there 4 cpu and make options is -j5.

Here i output from hdparm (p3 is root fs)

Code: Select all

localhost ~ # hdparm -tT /dev/mmcblk0                 

/dev/mmcblk0:
 Timing cached reads:   1086 MB in  2.00 seconds = 542.58 MB/sec
 Timing buffered disk reads:  32 MB in  3.07 seconds =  10.43 MB/sec

localhost ~ # hdparm -tT /dev/mmcblk0p3

/dev/mmcblk0p3:
 Timing cached reads:   1036 MB in  2.00 seconds = 517.81 MB/sec
 Timing buffered disk reads:  32 MB in  3.01 seconds =  10.62 MB/sec

[R0b0t1]

Is your power supply capable of providing ~2.5A? Your Raspberry Pi should be either browning out or overheating, unless you've put a heatsink on it.

Your low system load average is most likely due to SD card IO speed being the limiting factor in compilation. There isn't enough space in RAM to store compilation results, so the disk needs to be used for intermediate steps. I encountered this problem even with a very high end SD card. SD cards are not made for random access.

The last thing is 32bit mode seems to be 30-60% slower for unknown reasons based on benchmarks in the thread about Sakaki's image, but you might still expect compilation results to be faster. If you have the inclination you should set up crossdev and attempt to cross compile most of your packages, and consider reporting those packages which do not compile. At the current rate of progression I would expect everything to "just work" in crossdev if you came back in six months, but it seems like more breaking changes are being added and not fixed.

I ordered an odroid-c2 for the better GPU support and the faster IO but it seems like it will still be insufficient for reasonable compilation times. The odroid-xu4 has USB3 ports and far faster access through that than through eMMC, but would still be cross-compiling to support aarch64.

[NeddySeagoon]

madurani,

If nothing else was using the SD card,

Code: Select all

Timing buffered disk reads:  32 MB in  3.01 seconds =  10.62 MB/sec

is not very good.
If other things were accessing the card, this number will be reduced as timings test will share IO bandwidth with the other things.

If the 10.62MB/sec is for real, try another card. dmesg is right, high speed mode is not being used.

With only 86MB RAM in use, gcc is not running even one thread.
That's what MAKEOPS sets.

Do

Code: Select all

ps -Alf | wgetpaste

so that we can see the processes that are running.

The command

Code: Select all

emerge -ave system

will compute the dependency graph, then show you the results.
Computing the dependency graph is single threaded.
It won't get to multi threaded operation until you have told it to build the packages.

Its takes about 30 min to do dependency graph calculation here, with a disk read speed of 30MB/sec.

[madurani]

Here is output from wgetpaste:

Code: Select all

localhost ~ # ps -Alf | wgetpaste -s codepad
Your paste can be seen here: http://codepad.org/BDECbWv6

Here is MAKEOPTS

Code: Select all

localhost ~ # grep MAKEOPTS /etc/portage/make.conf 
MAKEOPTS="-j5 -l4"

[/code]


Graphic part wasn't installed till now, but I concern, that with speed issue of compilation it wasn't possible.

emerge system is runing:

Code: Select all

localhost ~ # free -m
              total        used        free      shared  buff/cache   available
Mem:            957          80         457           0         419         861
Swap:           236           0         236
localhost ~ # uptime
 16:28:09 up 16 min,  3 users,  load average: 1.18, 1.08, 0.68

[NeddySeagoon]

madurani,

Code: Select all

make -j5 -l4 -C /var/tmp/portage/sys-libs/glibc-2.23-r3/work/build-arm-armv7a-hardfloat-linux-gnueabi-nptl

That's glibc being built with your MAKEOPTS.

Code: Select all

/bin/busybox udhcpc --interface=wlan0 --now --script=/lib/netifrc/sh/udhcpc-hook.sh --pidfile=/var/run/udhcpc-wlan0.pid

Shows that the wifi is in use. The Wifi is attached to a 50MHz SDIO port, like the SDCard. That limits it to (theoretical max) 50Mbit/sec half duplex.
Its less that that for user data as their are the wifi management and TCP overheads.
Downloading over the wifi will be slow but it should keep up with your SD card.
It looks like you are also running ssh over the wifi, that will need some bandwidth too.

Has it started building glibc or is it still downloading patching and configuring?
They are single threaded activities.

[madurani]

Could be for this issue responsible bad system date? From me unknown reason was system date year 1970(on files too). I found out, that some processes have with this problem. I tried change modification time with touch and immediately load average jump to value 4. now it look very good. I compiled more packages with success.

Code: Select all

localhost ~ # free -m
              total        used        free      shared  buff/cache   available
Mem:            957         139          37           0         779         802
Swap:           236           0         235

localhost ~ # uptime
 17:12:45 up 57 min,  5 users,  load average: 3.98, 3.82, 2.84

[NeddySeagoon]

madurani,

Ah. The Pi has no real time clock, that can confuse make.

You need to use swclock in place of hwclock. That ensures that time is monotonic across shutdowns.
You also need ntpd and ntp-client in the default runlevel.
ntp-client gets time from the internet and sets the system time. It will be a forward step from that provided by swclock on startup.
ntpd keeps the system time right.

[madurani]

During installation I used stage3 package and there missing packages as iwconfig(wifi configuration) and ntp. Problem was, that after first booting I don't know nothing installed/emerged, because there was performance issue. I don't know why this packages missing in gentoo stage3.

But in /dev I still have some file with old date, can be it some potential issuue?

Code: Select all

localhost dev # ls -la tt*
crw-rw-rw- 1 root tty    5,  0 Jan  1  1970 tty
crw--w---- 1 root tty    4,  0 Jan  1  1970 tty0
crw------- 1 root tty    4,  1 Mar  8 19:25 tty1
crw--w---- 1 root tty    4, 10 Mar  8 18:06 tty10
crw--w---- 1 root tty    4, 11 Mar  8 18:06 tty11
crw------- 1 root root   4, 12 Mar  8 19:21 tty12
crw--w---- 1 root tty    4, 13 Jan  1  1970 tty13
crw--w---- 1 root tty    4, 14 Jan  1  1970 tty14
crw--w---- 1 root tty    4, 15 Jan  1  1970 tty15
crw--w---- 1 root tty    4, 16 Jan  1  1970 tty16
crw--w---- 1 root tty    4, 17 Jan  1  1970 tty17
crw--w---- 1 root tty    4, 18 Jan  1  1970 tty18
crw--w---- 1 root tty    4, 19 Jan  1  1970 tty19

in log files is error message related with tty0

Code: Select all

localhost dev # tail -5 /var/log/messages 
Mar  8 19:27:28 localhost agetty[20239]: /dev/ttyS0: No such file or directory
Mar  8 19:27:38 localhost agetty[21863]: /dev/ttyS0: No such file or directory
Mar  8 19:27:48 localhost agetty[22858]: /dev/ttyS0: No such file or directory
Mar  8 19:27:58 localhost agetty[23740]: /dev/ttyS0: No such file or directory
Mar  8 19:28:09 localhost agetty[24009]: /dev/ttyS0: No such file or directory

[NeddySeagoon]

madurani,

There are several packages that provide the same functionality.
Neither are key to system operation, so you choose which ones you will use, if any.
iwconfig is depreciated. You should use net-wireless/iw

Files in /dev will always be dated 1970. They are created by the kernel devtmpfs, mostly before root is mounted and before swclock starts, so time starts from zero, which is midnight at the start of 1 Jan 1970. This is normal and not a problem. If you attach a device and a new /dev node is created, it will have the correct timestamp.

The error message indicates that something is trying to use ttyS0. That's a serial port. agetty looks for logins.
Do you have something in /etc/inittab telling agetty to listen for logins on ttyS0?
If so, comment it out.

[madurani]

Hello All,

Compilation of system was successful. I installed xorg and lightdm. X work fine, but I have a problem. After implementation vga driver in file confix.txt (dtoverlay=vc4-kms-v3d) I am facing a TV resolution problem. Resolution is bigger than resolution of TV(1920x1080p). Some unreachable pixels are out of screen area. All variables which serve for "overscan" are ignored.

Please advise me, how I can correct setting of TV(hdmi) resolution.

[NeddySeagoon]

madurani,

Put the following into /boot/config.txt

Code: Select all

# have a properly sized image
disable_overscan=1

Then reboot so that config.txt is reread.

Don't forget to mount /boot first.

[madurani]

I found directly on TV function for autocorrection of screen and it solved a problem with resolution of TV (it took me few days trying different variables mentioned above).
I want to ask on vga drivers. In path /boot/overlays/ i found two: vc4-fkms-v3d.dtbo and vc4-kms-v3d.dtbo.
Is there some difference between them?

[madurani]

After compilation new kernel I lost my gentoo32bit (don't want booting), so I switch to gentoo-on-rpi3-64bit (sakaki). There I have some issue with vga driver, after some working in X (for example start Firefox, look some avi in vlc or only start screen saver) it crashed and isn't possible back to X screen. After typing password X crashed and back login screen(stiill again).

In config.txt I have:

Code: Select all

dtoverlay=vc4-fkms-v3d,cma-256
gpu_mem=128

Here is some error outputs from Xorg.0.log and /var/log/messages

Code: Select all

[  1885.208] (WW) glamor: Failed to allocate 1920x1080 FBO due to GL_OUT_OF_MEMORY.
[  1885.327] (WW) glamor: Expect reduced performance.

Mar 14 20:50:50 pi64 kernel: vc4-drm soc:gpu: failed to allocate buffer with size 8355840
Mar 14 20:50:50 pi64 kernel: vc4-drm soc:gpu: failed to allocate buffer with size 8355840
Mar 14 20:50:50 pi64 kernel: [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA:
Mar 14 20:50:50 pi64 kernel: [drm] num bos allocated: 146
Mar 14 20:50:50 pi64 kernel: [drm] size bos allocated: 64024kb
Mar 14 20:50:50 pi64 kernel: [drm] num bos used: 145
Mar 14 20:50:50 pi64 kernel: [drm] size bos used: 55864kb
Mar 14 20:50:50 pi64 kernel: [drm] num bos cached: 1
Mar 14 20:50:50 pi64 kernel: [drm] size bos cached: 8160kb

[NeddySeagoon]

madurani,

The

Code: Select all

gpu_mem=128

reserves 128Mb of RAM for the old graphics driver that is not being used.
Sakakis image uses the open source vc4 driver that allocates RAM from the CMA.

Code: Select all

Mar 14 20:50:50 pi64 kernel: vc4-drm soc:gpu: failed to allocate buffer with size 8355840
Mar 14 20:50:50 pi64 kernel: [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA: 

The 128Mb allocated in

Code: Select all

gpu_mem=128

is wasted.
Set that entry to

Code: Select all

gpu_mem=16

which is the smallest acceptable value.
That will take the pressure off the CMA, which is not only useh by the video driver.

[madurani]

Can be some from this variables deleted from config.txt? Are in use?

dtoverlay=vc4-fkms-v3d,cma-256
gpu_mem=128

[NeddySeagoon]

madurani,

You need

Code: Select all

dtoverlay=vc4-fkms-v3d,cma-256 

for the open source graphics driver.
It loads a piece of the device tree and sets up the CMA.

If you delete

Code: Select all

gpu_mem=128

you will get the default value, which varies depending on the RAM fitted to the Pi.
Since you only need give the GPU the minimum amount of RAM it needs for its own purposes, set it to gpu_mem=16.

Do not compare the GPU on a Pi to the GPU on a PC. It does much more.
A better comparison would be the VC4 GPU on the Pi is the GPU and chipset on a PC.

[madurani]

Hello All,

Could you please advise with cloning sd card. I want migrate my gentoo64 (sakaki) from old sd card to new(is bigger and faster). I found on net lot of "how to" but not work fo me. On system were done lot compilation a I don want begin with new img.

Attempt 1:
I did it via dd:

Code: Select all

dd bs=4M if=/dev/sdc of=/volume01/gentoo.img    # From old card to PC
dd bs=4M if=/volume01/gentoo.img of=/dev/sdc    # From PC to new sd card

Problem is that, system begin booting, but stay/hang on rainbow screen.

Attempt 2:

Via dd I copy sakaki img to new card and then via rsync move data from old sd card(on running raspberry) to new(on PC). System booting, but more tool aren't work correctly. for example NetwokManager don't know working with wifi. Device is unavailable.


Marek

[NeddySeagoon]

madurani,

There are no files on the Pi outside of the filesystem space.

Partition your new SD card with fdisk.
Mark the boot partition as bootable with a FAT32 partition type. (The Pi will check)
Make a FAT32 filesystem on /boot
Make an ext4 filesystem on root.

If you can mount both SD cards at the same time, you can use cp -a to copy all the files between them.
If not, cp -a to your HDD then from the HDD to the new card.

dd is not a good idea. It will copy the partition table and all the unallocated space from the old SD card.
As a result of copying the partition table, all your extra space will vanish.

[madurani]

Hello All,

I do it, but no work. I copied(cp -arv source destination) data from old sd-cardt to pc and then from pc to new card, but after turn on raspberry system stay on rainbow screen. Is there some possibility change rainbow screen with some screen, which are showing some error messages? Below are some configuration things. I haven't idea why not booting it. Can some sd-cards have different device name as mmcblk0pX?

Fdisk output( sd-card is insert in laptop):

Code: Select all

Device     Boot   Start       End   Sectors  Size Id Type
/dev/sdc1  *       2048    206847    204800  100M  c W95 FAT32 (LBA)
/dev/sdc2        206848   4401151   4194304    2G 82 Linux swap / Solaris
/dev/sdc3       4401152 122142719 117741568 56,1G 83 Linux

Code: Select all

argo raspberry # cat /mnt/raspberrypiroot/boot/cmdline.txt  
dwc_otg.lpm_enable=0 root=/dev/mmcblk0p3 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Output from fstab

Code: Select all

# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
/dev/mmcblk0p1          /boot           vfat    defaults          0       2
/dev/mmcblk0p3          /               ext4    defaults,noatime  0       1
/dev/mmcblk0p2          none            swap    sw                0       0

Marek