Portage crashes terminal, unable to compile @world set

[logrusx]

Code: Select all

MiB Mem :  27908.0 total,    238.2 free,  27721.8 used,    381.7 buff/cache     
MiB Swap:  65536.0 total,  32308.3 free,  33227.7 used.    186.2 avail Mem 

Code: Select all

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                
2254082 portage   20   0   38.3g  19.1g   1108 S  31.2  70.1  12:12.51 TensileCreateLi

Is there any reason you would want to build this monster? Can't you find a binary version from somewhere? What do you need it for? If it's absolutely necessary go buy a 128GB kit and hope for the best.

p.s. had I noticed how much memory it was taking I would have terminated it earlier, but now I'm willing to let it go and give you the binary. Hopefully my 5800H has the same flags as your 5600, because I have -march=native in make.conf.

Best Regards,
Georgi

[logrusx]

I'm sorry but that's going for far too long and I have no way of knowing how much longer so I have to terminate it. Maybe try to disable gfx942 and if that doesn't work, then gfx941, gfx940 AMDGPU_TARGETS.

I see this ebuild is marked N and you're running a world update, are you sure you really need it?

EDIT: just as I wrote that, before terminating it, fans sped up through the roof and I was curious if it'll finish soon.

Here's what I saw:

Code: Select all

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                        
2291592 portage   20   0  166900 143216   4012 R  86.8   0.5   3:53.40 python3.12                                                                                                     
2291613 portage   20   0  175556 150832   4216 R  86.4   0.5   3:54.25 python3.12                                                                                                     
2291615 portage   20   0  171904 148044   3992 R  86.4   0.5   3:54.58 python3.12                                                                                                     
2291589 portage   20   0  166684 143228   3544 S  86.1   0.5   3:54.14 python3.12                                                                                                     
2291595 portage   20   0  168292 145124   3888 S  84.4   0.5   3:54.09 python3.12                                                                                                     
2291611 portage   20   0  168248 143708   4132 R  84.4   0.5   3:55.68 python3.12                                                                                                     
2291617 portage   20   0  166968 143552   4372 R  84.4   0.5   3:53.53 python3.12                                                                                                     
2291600 portage   20   0  178168 155252   3820 R  84.1   0.5   3:52.95 python3.12                                                                                                     
2291608 portage   20   0  173864 150240   3780 R  84.1   0.5   3:55.11 python3.12                                                                                                     
2291606 portage   20   0  172232 148204   3664 R  82.8   0.5   3:55.49 python3.12                                                                                                     
2291602 portage   20   0  166388 143204   3820 R  82.1   0.5   3:54.63 python3.12                                                                                                     
2291604 portage   20   0  167864 143696   3876 S  81.8   0.5   3:53.62 python3.12                                                                                                     
2291612 portage   20   0  167284 143088   3608 R  81.1   0.5   3:55.84 python3.12                                                                                                     
2291616 portage   20   0  167896 144428   3732 R  80.5   0.5   3:53.97 python3.12                                                                                                     
2291598 portage   20   0  171476 148368   3592 R  79.8   0.5   3:53.89 python3.12                                                                                                     
2291614 portage   20   0  174220 151148   3712 S  78.8   0.5   3:55.08 python3.12                                                                                                     
2215213 joro      20   0 3268360 105648  24108 S   4.3   0.4  15:22.19 Hyprland                                                                                                       
2254082 portage   20   0   45.2g  16.9g   3200 S   3.6  62.0  35:30.34 TensileCreateLi                                                                                                
2359157 portage   20   0  232344  58956  50764 R   3.3   0.2   0:00.10 clang++                                                                                                        
2359159 portage   20   0  231396  57988  50308 R   2.3   0.2   0:00.07 clang++                                                                                                        
2359160 portage   20   0  231976  57940  50260 R   2.0   0.2   0:00.06 clang++                                                                                                        
    121 root      20   0       0      0      0 S   1.7   0.0  24:16.06 kswapd0                                                                                                        

In reality it runs more threads than make is instructed. Maybe make jobs are only 16 but some of them invoke other jobs too, so this package's build system doesn't seem to respect those. And it's likely not easy to make it do so. I'll wait just a little bit more as I hope this is the final stage of the compilation of the final target - gfx942.

EDIT2: I'm sorry, I gave it more than 40 minutes to finish, it didn't. After more than 2 hours compile time total and hundreds of gigabytes, if not terabytes wasted write cycles for this package only, I'm terminating it.

If you want to compile it badly, setup 64 GB swap and let it do its job. Have that in mind too:

Code: Select all

du -sh /var/tmp/portage/sci-libs/hipBLASLt-6.1.1-r1
33G	/var/tmp/portage/sci-libs/hipBLASLt-6.1.1-r1

Best Regards,
Georgi

[Zucca]

If it's absolutely necessary go buy a 128GB kit and hope for the best.

As it happens I just bought a workstation and stuffed it with 256GB of RAM. Although I don't have a working Gentoo install on it just yet.
But... I just cannot believe the memory requirements. Some build processes requiring 16GB seem realistic with big projects, and maybe 32GB. Something requiring something between 64GB or even 128GB is just not real.
I'd say with quite high confidence that the build system is broken... or the ebuild has some nasty bug.

I could be wrong. I haven't looked too deep into this.

[logrusx]

If it's absolutely necessary go buy a 128GB kit and hope for the best.

As it happens I just bought a workstation and stuffed it with 256GB of RAM.

From what I saw last night, 64GB might suffice even without swap, but swap better be there to prevent OOMs that might arise for a few hundred megabytes shortage.

But... I just cannot believe the memory requirements. Some build processes requiring 16GB seem realistic with big projects, and maybe 32GB. Something requiring something between 64GB or even 128GB is just not real.
I'd say with quite high confidence that the build system is broken... or the ebuild has some nasty bug.

I could be wrong. I haven't looked too deep into this.

This is not a normal build process. This is not a compiler. This is more like building a model, so the perspective you're looking from is not the right one. Just like when I tried to run Llama. When it didn't fit the GPU memory it simply run on the CPU. This is not a bug but a different type of software wrt it doesn't build executables but a model and it doesn't compile source but generates that model. And it doesn't use a compiler.

Best Regards,
Georgi

[picarica]

I'm building it now. It indeed runs Tensile:

Code: Select all

################################################################################
# Tensile Create Library
# Detected local GPU with ISA: gfx1030
# Detected local GPU with ISA: gfx1100
# Detected local GPU with ISA: gfx906
# Detected local GPU with ISA: gfx908
# Detected local GPU with ISA: gfx90a
# Detected local GPU with ISA: gfx942

<<a huge matrix with features for the above targets here>>

Tensile::WARNING: Global parameter WriteMasterSolutionIndex = False unrecognized.
# CodeObjectVersion from TensileCreateLibrary: default
# CxxCompiler       from TensileCreateLibrary: hipcc
# Architecture      from TensileCreateLibrary: gfx1030_gfx1100_gfx906_gfx908_gfx90a_gfx942
# LibraryFormat     from TensileCreateLibrary: msgpack
# LibraryLogicFiles:

<<a huge list of logic files here >>

Reading logic files: Launching 16 threads for 665 tasks...
Reading logic files: Done.
[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||] 100% (97.3 secs elapsed)

<< output very similar to the above here  followed by intensive fan noise here >>

but this is rocBLAS package, 2 more to go and get to hipBLASLt.

So it indeed runs the number of jobs defined in MAKEOPTS, but those are not compiler jobs.

Sixteen threads with 32GB RAM here, from which 27 available due to the iGPU consuming 4. Swap usage approached 2GB but dropped down below 1.

We'll see where that goes.

I don't know why the build broke before and why portage was giving me trouble with conflicts and blocks but now this part went fine.

EDIT: I can't tolerate that load now. I'll let it go tonight.

Best Regards,
Georgi

how did it go?

[picarica]

oh sorry havent noticed there is 2nd page, but well this cannot be right, how does it use so many GB? i nedded it for development, i had to install windows to use rocm there, isnt this a bug? since when i need 256gb of ram. i was usng rocm versiopnm 5.6 which didnt had this issue

[logrusx]

how does it use so many GB?

I don't know, ask AMD.

since when i need 256gb of ram

Since you want to use that software and that version. And I said 64 with some swap might suffice. Depending on what type of disk you have. You might be able to build it with 32 if you have an SSD and are willing to sacrifice so many write cycles.

isnt this a bug?

Mind you, this comes a little bit rude after all the time and resources I wasted on your issue. Did you read what I reported? If you think this is a bug, report it to AMD, don't complain here.

Best Regards,
Georgi

[Zucca]

Code: Select all

d-mill ~ # cat /etc/portage/package.accept_keywords/ROCm
=dev-util/Tensile-6.3.2

=dev-util/rocminfo-6.3.2
=dev-util/hipcc-6.3.2
=dev-build/rocm-cmake-6.3.2
=dev-libs/rocr-runtime-6.3.2
=dev-libs/rocm-comgr-6.3.2-r1
=dev-util/rocm-smi-6.3.2
=dev-libs/roct-thunk-interface-6.3.2
=dev-libs/rocm-device-libs-6.3.2
=dev-util/hip-6.3.2

If the memory usage increases linearly with compiling threads, my machine will choke too.

I might have a compelling reason to compile this too. I may have some use or ROCm etc.

Let's try this...

[logrusx]

Code: Select all

d-mill ~ # cat /etc/portage/package.accept_keywords/ROCm
=dev-util/Tensile-6.3.2

=dev-util/rocminfo-6.3.2
=dev-util/hipcc-6.3.2
=dev-build/rocm-cmake-6.3.2
=dev-libs/rocr-runtime-6.3.2
=dev-libs/rocm-comgr-6.3.2-r1
=dev-util/rocm-smi-6.3.2
=dev-libs/roct-thunk-interface-6.3.2
=dev-libs/rocm-device-libs-6.3.2
=dev-util/hip-6.3.2

If the memory usage increases linearly with compiling threads, my machine will choke too.

I might have a compelling reason to compile this too. I may have some use or ROCm etc.

Let's try this...

I've already noted it's one process that consumes all threads and all memory and it's not a compiler, so you can't control memory consumption through adjusting make jobs. Moreover it likely requires that much memory to build the model or library or whatever it's building for that particular target.

I already explained that's not a compiler and it does not produce executable code, it calculates something that it then puts in some kind of a library, much like the model of an LLM AI.

The way forward is to either add 32 GB of memory or wear the SSD by swapping. It won't work on an HDD as it'll go into swap thrashing.

Another way is to simply disable the gfx942 target. This is what was consuming that much memory and time. You see, this is some kind of AI or similar stuff computation-wise. Those models/libraries need to be of certain size, otherwise it doesn't work.

As to why it requires so much memory, well get used to it. Things are going only one direction until they hit the limitations of what size can be reasonably organized, accessed and used.

Best Regards,
Georgi

[Zucca]

Code: Select all

d-mill ~ # qlop -vH
2025-02-17T23:53:04 >>> dev-python/cloudpickle-3.1.1: 1 minute, 25 seconds
2025-02-17T23:53:04 >>> dev-build/rocm-cmake-6.3.2: 1 minute, 29 seconds
2025-02-17T23:53:04 >>> dev-libs/roct-thunk-interface-6.3.2: 1 minute, 34 seconds
2025-02-17T23:53:04 >>> dev-python/psutil-6.1.1: 1 minute, 38 seconds
2025-02-17T23:53:04 >>> dev-python/msgpack-1.1.0: 1 minute, 42 seconds
2025-02-17T23:53:04 >>> dev-util/rocm-smi-6.3.2: 1 minute, 47 seconds
2025-02-17T23:53:04 >>> dev-util/hipcc-6.3.2: 1 minute, 51 seconds
2025-02-17T23:53:04 >>> app-editors/vim-core-9.1.0794: 1 minute, 58 seconds
2025-02-17T23:53:04 >>> dev-build/b2-5.2.1: 2 minutes, 2 seconds
2025-02-17T23:55:07 >>> dev-python/loky-3.4.1: 3 minutes, 28 seconds
2025-02-17T23:55:07 >>> dev-libs/rocm-device-libs-6.3.2: 3 minutes, 32 seconds
2025-02-17T23:55:06 >>> dev-libs/boost-1.85.0-r1: 3 minutes, 44 seconds
2025-02-17T23:58:51 >>> dev-cpp/msgpack-cxx-6.1.0: 43 seconds
2025-02-17T23:58:50 >>> dev-python/joblib-1.4.2: 48 seconds
2025-02-17T23:58:51 >>> dev-libs/rocr-runtime-6.3.2: 52 seconds
2025-02-17T23:58:51 >>> dev-libs/rocm-comgr-6.3.2-r1: 56 seconds
2025-02-17T23:59:47 >>> dev-util/rocminfo-6.3.2: 12 seconds
2025-02-17T23:59:59 >>> dev-util/hip-6.3.2: 40 seconds
2025-02-18T00:00:39 >>> dev-util/Tensile-6.3.2: 24 seconds

And also:

Code: Select all

d-mill ~ # cat /etc/portage/package.use/ROCm 
*/* AMDGPU_TARGETS: -* gfx900

...

I barely saw memory usage go up.

Exactly what package made the memory usage go out of control with you?

EDIT01: I missed this

Another way is to simply disable the gfx942 target. This is what was consuming that much memory and time.

... Oh well... let's try again.

EDIT02: After enabling gfx942 and recompiling with --newuse, nothing significant happened. Emerging took about 30 seconds.

Do we have the same versions we're compiling?

[logrusx]

EDIT02: After enabling gfx942 and recompiling with --newuse, nothing significant happened. Emerging took about 30 seconds.

Do we have the same versions we're compiling?

Are we talking about the same package? I'm talking about hipBLASLt. The process that takes all the memory and threads is Tensile<something starting with capital letter here> while building some kind of library related to gfx942. If you have a different GPU higher target may be enabled and I guess it'll take even more memory to build the libraries.

Also since then I think more versions have entered the tree. At that time it was hipBLASLt-6.1.1-r1

EDIT: more targets are enabled for 6.3.2

For 6.1.1-r1 they are AMDGPU_TARGETS="gfx90a gfx940 gfx941 gfx942"
and for 6.3.2: AMDGPU_TARGETS="gfx90a gfx908 gfx940 gfx941 gfx942 gfx1100 gfx1101"

Also the llvm slot has changed to 19. I might give it another try just to verify if my findings are still relevant.

But 30 seconds, we're talking about something else. I remember the was a package that took like a minute on the background of all other packages taking hours.

EDIT2: interesting. Last time it took hours before getting to hipBLASLt and now it took only minutes. I'm not sure what's going on and I definitely don't have the logs. I also think I depcleaned after that experiment. Version 6.3.2 indeed enables more targets:

Code: Select all

-- AMDGPU_TARGETS: gfx1100;gfx1101;gfx908;gfx90a;gfx940;gfx941;gfx942;

I'll see it I can stop it before taking over my swap partition and enable only the targets that compiled successfully without too much RAM usage.

EDIT3: nah, it immediately starts gradually increasing swap usage. I guess I just didn't notice it last time. It definitely requires more RAM and will be faster with more RAM as swap, albeit on an SSD, is still much slower than RAM.

On [1/35] the swap usage jumped to 3 GB and on [2/35] to 32GB. I can only imagine where it can go.

EDIT4: I'm terminating this too. Further experiments are meaningless. It still consumes insane amounts of memory to build that libraries. And I can only guess it'll get even higher for the higher targets. I'm not attempting that anymore.

There are packages built for other distros, I think a -bin ebuild sounds reasonable, but I have no experience with that. If OP wants, they may file a bug and if they are lucky, somebody may write such an ebuild for them.

Best Regards,
Georgi

[Zucca]

I'm talking about hipBLASLt

I compiled that too, but since it doesn't have support for my gpu (gfx900), it only built a dummy target.

I'll try with gfx942 target later. Hopefully within next 24 hours.