Gentoo on limited ressources

[mkrsn]

Hi all,

I've got 4 tiny Servers (1 vCPU; 512 MiB RAM; 9.7GiB Storage) running Gentoo. Until the last GCC update all was fine (GCC acceptable) so far. Since Gentoo (GCC & Kernel too) is getting bigger and bigger every year I have now problems in compiling GCC on such small Servers. It's currently not possible because of the limited resources.

Some years ago I already put the whole portage tree into a squashfs because of the limited Size. But still, Gentoo is 5GB big (just installed some tools, FRR and telegraf - nothing big, 100MiB logs) - which isn't enough for a swap file & storage for GCC compiling.

- The most obvious thing was to
-- check if there is a gcc-bin package -> unfortunately not (does anyone know why?)
-- building a portage env for GCC and limiting compiling to one process
- I was already thinking about bin-packaging it by myself: that would mean build for each server a own package (they don't have the same CFLAGS)

Unfortunately i don't have an furthers ideas on howto solve this Issue. I even already thought about switching to Artix, but i don't like that idea.

Does anyone here has a Idea of howto solve this problem?


Greetings
Matthias

[pingtoo]

Can you share what is limiting the build process? 512MB and 9.7G storage should allow you build gcc. however it might take very long time.

Do you use swap?

Are all 4 servers on same network?

[mkrsn]

Available Storage is ~4.2GiB - without Swapfile. Thats why the problem depends on the try.

I just need swap when compiling GCC. Otherwise it's removed from the system.

- 2GiB Swap: gcc was killed by OOM
- 2.5GiB Swap: gcc was killed by OOM
- 3GiB Swap: gcc complained about missing storage

The vServers are spreaded around the world (DE, ES, UK, US) but are in a VPN.


While writing this i totally forgot the "root reserved space" from EXT4 - I never freed them, which are ~500 MiB of free storage. Maybe this is now the missing necessary Storage ^^

[pingtoo]

Since servers spread over long distance so using network storage is not a good option.

is the 4.2G after gcc source code dump? or source code will take further more space?

do you have other linux device(s) in LAN that may have resource(memory or storage) to share?

I heard gcc 13 have something changes will reduce generated sourced to smaller piece so each compile unit will use less resource.

the main constrain is available resource in your use case. so unless there are other nodes that can share resources with the building node, there is not much can be done. I think gcc build have multiple stages so there is chance you can break build processes into multiple steps and do some cleanup in between steps but that require some research on what can be clean up between stages.

if you have nodes on LAN that can share resources memory or storage you can consider use NFS for quick setup, or NBD (network block device) with compress for a little bit more complicated procedures.

Let me know if you need more detail if you think you can use network resources but need more information.

[NeddySeagoon]

mkrsn,

Do you have any other Gentoo systems?

Build and running Gentoo can be separated. They usually are for weaker embedded systems.
The idea is to build on a powerful system then install the binaries on the target system, like this. The target system need not build anything.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[mkrsn]

@pingtoo,

I already thought about NFS, NBD and co. I think i would swap the Storage Problems with connection Problems because of the long distances. But even if this were not the case, it would still be effort that is not really worth it.

The 4.2GiB is without distfiles, swap and extracted source code.There is also a script which dleans distfiles on a daily base.

In the meantime I've freed the root reserved space and tested it again with 2.5GiB & 3GiB Swap. Both tests were aborted because of missing Storage.

@NeddySeagoon

I also thought about building packages. There are plenty of resources on other Servers. But as I mentioned: It would need to maintain 4 different build chains because of the 4 different CFLAG variations. It's still not worth the effort, then it would be easier to switch all vServers to a different Distro like Artix. I Also thought about building a more generic GCC Version (as like in the stage 3) but i couldn't find an information on that.


@both
I've removed gentoo-sources which brings additional 1.8GiB of storage - which would be a temporary solution (hopefully - compiling still runs) until gcc 13 jumps in.

[pingtoo]

[NeddySeagoon]

mkrsn,

Is the ::gentoo repo in squashfs?
The entire repo is 50MB. You throw it away every update. Not rsync.
That compares to my 619M for /var/db/repos/gentoo/ using a filesystem with a 4k block size.

Do you really need 4 different CFLAG variations, or there a lowest common denominator that will run everywhere and still provide the required performance?
What are the 4 settings?

e.g. build gcc once and run it everywhere. The CPUFLAGS it is built with affect the instruction set it will require to run, not the instruction set it can output code for.

The compressed kernel sources can be piped through tar to mksquashfs to get a read only usable to build file that is mounted somewhere to access it.
The kernel cannot be built in the squashfs, as its read only but the kernel build system has an option to put the build products somewhere else. The Makefile says

[mkrsn]

Good Morning,

after removing all kernel sources i had ~3.5GiB of free Storage (with 3GiB Swap file). Even that is not enough resources. The compiling were killed because of missing storage (3.4G /var/tmp/portage/sys-devel/)
The last chance is now to reduce the swapfile a little (to 2.5GiB - praying for not seeing OOM) to have ~4GiB of free storage.

God, I really really hate GCC right now... XD


@pingtoo,

thank you for pointing out catalyst - I'll give it a try.


@NeddySeagoon,

the Portagetree is already in a compressed Squashfs (57MiB).

For GCC i don't need 4 different CFLAG variations. One generic GCC package would be pretty fine - even when compiling is a little slower. But i couldn't find any information on how a generic CFLAG would look like. I yesterday eyeballed a bit into dev-util/catalyst and it looked like I had found here a generic one. I need to test it.

The compressed kernel sources looks really interesting, I'll keep that in mind for other projects - thank you. Unfortunately it won't work here because even without kernel-sources it's not possible to compile GCC 12.2 :/

[NeddySeagoon]

mkrsn,

Generic for AMD64 is

[Goverp]

Do you really need 3GB swap? my laptop has CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold" and CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo" backed by 1GB disk. The laptop has 4 GB ram, so zswap default allocation of 20% is 800 MB, tripled to 2.4 GB, so effective memory space is 3.2 + 2.4 = 5.6 GB.

I also have CONFIG_ZRAM=m, CONFIG_ZRAM_DEF_COMP="lzo-rle", with /var/tmp/portage set to zram - you could go further with all /var/tmp set there.

[mkrsn]

I could solve this issue by removing "-pipe" from CFLAGS, setting swapfile to 2GiB (which not worked before removing pipe in any case) and gentoo-sources.

Thanks for all your comments, thoughts and help


@NeddySeagoon

thx - i will keep that CFLAG in mind. I pray that gcc 13 will use resources a bit more efficiently than v12 does.


@Goverp

On a system with multiple dedicated CPUs ZRAM & co may work. But on a shared single vCore I'm not sure if this is the way to go. Therefore i try to avoid ZRAM & co here.

[NeddySeagoon]

mkrsn,

zram swaps CPU time for space.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[sam_]