Rust binary rust-bin, binary-free, binary chains experience

[trickygnome]

I have been doing “emerge --update” recently and noticed installation
of dev-lang/rust-bin package.

Bootstrapping compiler is a multi-stage compilation of compiler that
is written in the same programming language (self-hosting).

GCC is the default C, C++ compiler in Gentoo that installed with Stage 1 tarball.

The initial step version of compiler:
- Rust = OCaml, but this approach was abandoned long ago.
- Go = C
- LLVM/Clang = C++
- GCC = C
- Python = C

As you can see, all compilers, except of Rust, may be build from sources.

Today, Rust bootstrap with an existing Rust compiler, typically a beta
or stable version of “rustc”. Alternatives:
- OCaml :: this method is no longer practical or maintained.
- mrustc and dtolnay/bootstrap :: not an official part of the Rust
project. C++-based Rust compiler that can compile an early version
of the Rust compiler.
- gccrs and Cranelift :: experimental and not yet fully developed.

But the main problem I faced is not Rust, but Gentoo emerge that
download binary rust-bin and run it *without my approvement*. I see
that Gentoo is getting old and lazy, that is not right. I don't know
for shure, but manually installing “dev-lang/rust” package will use
old version to update.

One solution to block binaries is to use file “/etc/portage/package.mask/all” with:
: */*-bin
: */*-bootstrap

“-bin” suffix for binary package naming is not strict and just recommended:
https://devmanual.gentoo.org/ebuild-wri ... index.html
Who knows which packages have binaries, also? Ex. google-chrome.

For updating GCC, Gentoo don't download a binary
version of the compiler. Instead, it update GCC by compiling the new
version from source code.

From this perspective *Go compiler* looks better, because may be build with help of “gcc-go” or
“gollvm” parts of GCC and LLVM. But Gentoo download binary Go 1.20 dev-lang/go-bootstrap from
https://dev.gentoo.org/~williamh/dist

[b]I suggest: to make “-bin” strict and always ask before installing such things.[/b]

[eschwartz]

But the main problem I faced is not Rust, but Gentoo emerge that
download binary rust-bin and run it *without my approvement*. I see
that Gentoo is getting old and lazy, that is not right. I don't know
for shure, but manually installing “dev-lang/rust” package will use
old version to update.

This is a grave misunderstanding. Gentoo doesn't download a binary rust-bin without your approval.

It actually used to, and we just fixed that bug. Previously, there was a USE flag: "system-bootstrap". With that USE flag:


- enabled, rust would depend on an existing /usr/bin/rustc to compile the new version of rust. This might be dev-lang/rust, or it might be dev-lang/rust-bin.
- disabled, rust would unconditionally download a SRC_URI tarball of a prebuilt rust-bin, unpack it into a temporary directory, and compile the new copy of rust. It would then delete the temporary prebuilt rust-bin copy in the temporary directory.


The recent change to how rust is packaged means the USE=system-bootstrap flag has been ***removed***, and is now mandatory. The dev-lang/rust package no longer permits downloading a SRC_URI tarball containing a prebuilt rust-bin. You MUST bootstrap using a system-bootstrap compiler for rust.

If you have dev-lang/rust installed, it will use that. If you do not have it, you must bootstrap it somehow -- most commonly, this requires using rust-bin, but it is fully under your control as it no longer happens hidden inside of SRC_URI, but happens as an emerge dependency.

Your conspiracy theory about "old and lazy" is misplaced. Sorry.

For updating GCC, Gentoo don't download a binary
version of the compiler. Instead, it update GCC by compiling the new
version from source code.

GCC and rust are exactly the same here. Including the fact that before you can update GCC, you must first install a copy of GCC. If you don't have a GCC yet, the only way to get GCC on Gentoo is to download a prebuilt gcc bin, which is usually provided inside the stage3 tarball. (A stage3 tarball is literally a tarball containing lots of prebuilt bins that together make up a base install.)

There is another small difference -- GCC typically can be compiled with older versions of GCC going back several versions. Currently, GCC 13 and 14 support being compiled with a preexisting copy of GCC 4.8. In contrast, Rust requires the previous most recent version of rust in order to compile a new version -- so rust 1.82.0 only supports being compiled with rust 1.81 (or 1.82).

[sam_]

Eli is, of course, completely right. If this situation causes you distress, please donate or otherwise contribute to the gccrs effort.

In future, please start with asking a question in good faith, rather than presuming "laziness".

I'll also note the following:

• I think there's an inconsistency between it being supposedly lazy to have the status quo (which it decidedly is not, not least because of the progress made here actually facilitates a proper bootstrap chain) and also noting the difficulty, as you did, in your post wrt the messy and poor situation even outside of Gentoo wrt bootstrapping Rust.
• The truly "lazy" thing for us to do would've been to: a) not slot Rust; b) not make this system-bootstrap change so that dev-lang/rust continued to download a binary and nobody would complain.
• We've not always packaged every single version of Rust given Rust requires work to maintain. Occasionally because of people being busy, a version wasn't packaged, so this will have happened (several times) before, just nobody noticed because of the behaviour of dev-lang/rust in the past.
• Prior to slotting, there was no real way to have a proper chain because you couldn't have multiple Rusts installed, nor could you control properly which ones are available for use to bootstrap.
• I have a proposal I need to finish wrt making it easier to identify binaries, see bug 885827.
• GCC is not the same as Rust, given that (for one) to include Rust in stages would inevitably make various people upset, and would also imply we need something in stage3s that hard-require Rust. That has obvious downsides I'm sure I don't need to repeat.
• The situation was described in detail in the commit message that made this change. See also bug 943144. I recommend checking git log if you're interested in process and rationale.
• bug 943706 was filed earlier to handle adding the missing slots for the current range of Rust versions in Gentoo and also to discuss going back to mrustc and having a path from there. Perhaps you're interested in volunteering?
• Within that same bug, I explain the issue with Go, and I note that I'd like to get that fixed too. I also filed bug 941769 to that end.
• With regard to Go, I've no idea how viable gollvm for this, although gccgo is indeed viable for bootstrapping once slotting is sorted.

[bunder]

I'm more curious about the changes to rust, as all of my systems have removed rust with a depclean in favour of rust-bin... Was that intentional? I mean, it's one less rust-related compile to wind up failing randomly, but I don't normally install binary packages unless there isn't a non-bin one for a particular application/dependancy... Thanks

[Ionen]

I'm more curious about the changes to rust, as all of my systems have removed rust with a depclean in favour of rust-bin... Was that intentional? I mean, it's one less rust-related compile to wind up failing randomly, but I don't normally install binary packages unless there isn't one for a particular application/dependancy... Thanks

Current defaults prefer rust-bin just as a quality-of-life thing (been like that for a while, even before the recent changes), but the new slotting had a side-effect for non-bin users (or at least it's what I roughly observed, maybe I'm wrong given I didn't test portage's behaviour much, nor what having it in world file would do).

Previously if you `emerge -1 dev-lang/rust` (or already had it from the old default, likely not in world file) it'd keep using dev-lang/rust and update it without considering rust-bin again (with system-bootstrap it'd also build itself with that built rust assuming the versions are not too far apart, otherwise it may have tried rust-bin as a fallback to build itself).

With slots however (similar to being separate packages), it considers rust-bin again with each new slot (e.g. if you have dev-lang/rust:1.82.0 installed and got an update pending for 1.83.0, when you emerge ... @world it'll install dev-lang/rust-bin:1.83.0 by default, or at least if not in world file but haven't tried that yet). So, your systems likely got updated to -bin that way then depcleaned the old.

Makes me wonder how many non-bin users we have left given they have to go out of their way to keep it (I do use it myself given rust-bin is broken with -native-symlinks while non-bin is not and I never looked into why, not that this affects normal users).

[logrusx]

Makes me wonder how many non-bin users we have left given they have to go out of their way to keep it (I do use it myself given rust-bin is broken with -native-symlinks while non-bin is not and I never looked into why, not that this affects normal users).

Currently portage offers me the following update:

Code: Select all

[ebuild     U  ] dev-lang/rust-1.82.0-r101:1.82.0::gentoo [1.82.0-r100:1.82.0::gentoo] 

I moved off of bin since regular rust was on the binhost, is that still the case? If not, it makes sense to move back to bin.

Best Regards,
Georgi

[Ionen]

Currently portage offers me the following update:

Code: Select all

[ebuild     U  ] dev-lang/rust-1.82.0-r101:1.82.0::gentoo [1.82.0-r100:1.82.0::gentoo] 

For same slot that's expected, I believe it'll offer you rust-bin when updating to a different slot (1.83.0), or at least it's what it did for me. May differ if you have it in world file or masked rust-bin.

[logrusx]

For same slot that's expected, I believe it'll offer you rust-bin when updating to a different slot (1.83.0), or at least it's what it did for me. May differ if you have it in world file or masked rust-bin.

I don't think I have either of them. I'll report back when it hits stable and I update. Currently I'm waiting for the binary to become available on the binhost and I won't let that update through until then.

Best Regards,
Georgi

[Zentoo]

Interesting thread about rust bootstrapping that I found after have noticed the "rust" USE flag for sys-devel/gcc.

So now I wonder:
- Is it possible actually to use sys-devel/gcc[rust] to bootstrap rust ?
- Is it possible actually to replace dev-lang/rust or dev-lang/rust-bin by sys-devel/gcc[rust] as main rust ?
- if the answers to preceding questions are no, is it a purpose to use sys-devel/gcc[rust] as main rust on the long run ?

[sam_]

Interesting thread about rust bootstrapping that I found after have noticed the "rust" USE flag for sys-devel/gcc.

So now I wonder:
- Is it possible actually to use sys-devel/gcc[rust] to bootstrap rust ?
- Is it possible actually to replace dev-lang/rust or dev-lang/rust-bin by sys-devel/gcc[rust] as main rust ?
- if the answers to preceding questions are no, is it a purpose to use sys-devel/gcc[rust] as main rust on the long run ?

Not yet, not yet, yes.

[sam_]

For those of you interested in bootstrapping, Kangie has been working on getting mrustc wired up. See https://bugs.gentoo.org/943706#c13 where he calls for testing from interested users. I especially hope those lobbing accusations of "laziness" will be giving constructive feedback.

[Zentoo]

Interesting thread about rust bootstrapping that I found after have noticed the "rust" USE flag for sys-devel/gcc.

So now I wonder:
- Is it possible actually to use sys-devel/gcc[rust] to bootstrap rust ?
- Is it possible actually to replace dev-lang/rust or dev-lang/rust-bin by sys-devel/gcc[rust] as main rust ?
- if the answers to preceding questions are no, is it a purpose to use sys-devel/gcc[rust] as main rust on the long run ?

Not yet, not yet, yes.

Thanks. That sounds great

[bstaletic]

I'm curious to try mrustc ebuild and bootstraping rust 1.74.
What would be the correct way to "merge" the pull request for a rsync portage user?

I tried just backing up my /var/db/repos/gentoo and cloning Kangie's fork/branch in the same place, but eix and emerge started complaining about some digests.

I guess grabbing the patch off github and manually invoking [color]patch -Np1 < kangies-pull-req.patch[/color] should work?

[miket]

I had a surprise when updating a system after a few months and so dev-lang/rust was multiple versions behind: emerge took a good bit of time in doing dependency resolution in order to stage updates to each of the slotted versions in turn. Yikes!!

It was then I saw to my great relief that chromium no longer needs Rust to be built with the profiler USE flag--which was not available in rust-bin. (Yes, I had been hoping to switch to rust-bin but chromium thwarted me.) So I canceled that first emerge run, removed dev-lang/rust (and masked it for good measure), and restarted my world update. Dependency resolution now took about a third of the time it did before.

Egad! Slotted Rust?! It's very telling that the Rust specification is so unstable that packages need specific versions of it--and the compiler itself must be built with exactly the version before it. I keep hoping that a language like Zig will displace Rust and its whole lunatic ecosystem.

[logrusx]

For same slot that's expected, I believe it'll offer you rust-bin when updating to a different slot (1.83.0), or at least it's what it did for me. May differ if you have it in world file or masked rust-bin.

I don't think I have either of them. I'll report back when it hits stable and I update. Currently I'm waiting for the binary to become available on the binhost and I won't let that update through until then.

Best Regards,
Georgi

I had mostly forgotten about this thread, but it came up again. Out of curiosity I run the following and here's what the results are:

Code: Select all

~ # ACCEPT_KEYWORDS="~amd64" emerge -av rust

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 10.76 s (backtrack: 1/20).

[ebuild  NS    ] dev-lang/rust-1.83.0-r1:1.83.0::gentoo [1.81.0-r100:1.81.0::gentoo, 1.82.0-r101:1.82.0::gentoo] USE="(-big-endian) -clippy -debug -dist -doc (-llvm-libunwind) -lto (-miri) -nightly (-parallel-compiler) -rust-analyzer -rust-src -rustfmt -system-llvm -test -verify-sig -wasm" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" LLVM_SLOT="(19)" LLVM_TARGETS="(X86) -AArch64 -AMDGPU -ARC -ARM -AVR -BPF -CSKY -DirectX -Hexagon -Lanai -LoongArch -M68k -MSP430 -Mips -NVPTX -PowerPC -RISCV -SPIRV -Sparc -SystemZ -VE -WebAssembly -XCore -Xtensa" 345,517 KiB
[ebuild  NS    ] dev-lang/rust-1.84.0:1.84.0::gentoo [1.81.0-r100:1.81.0::gentoo, 1.82.0-r101:1.82.0::gentoo] USE="(-big-endian) -clippy -debug -dist -doc (-llvm-libunwind) -lto (-miri) -nightly (-parallel-compiler) -rust-analyzer -rust-src -rustfmt -system-llvm -test -verify-sig -wasm" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" LLVM_SLOT="19" LLVM_TARGETS="(X86) -AArch64 -AMDGPU -ARC -ARM -AVR -BPF -CSKY -DirectX -Hexagon -Lanai -LoongArch -M68k -MSP430 -Mips -NVPTX -PowerPC -RISCV -SPIRV -Sparc -SystemZ -VE -WebAssembly -XCore -Xtensa" 346,674 KiB

So no, it doesn't offer rust-bin, but to build 1.83. I'm not sure how the situation will change when it hits stable and there are binary packages on the binhost. I guess it'll offer me the binary instead.

Best Regards,
Georgi

[sam_]

See bug 947587. You can workaround this with emerge -uvDU @world dev-lang/rust-bin -1 or similar.

[logrusx]

See bug 947587. You can workaround this with emerge -uvDU @world dev-lang/rust-bin -1 or similar.

Thanks for letting me know. I read your comment.

* Evaluate both paths and cost them by # of new packages

This option is what I most agree with. Only remark is there might be multiple paths, not only two. It should offer the shortest one. Users should use package.mask and/or emerge precise package slots and/or versions if they want to alter it.

However I'm not sure to what extent it would be customizable.

To always discard the existing slot with the presumption it would be older also deserves thought. I'm not sure I fully comprehend the consequences of that though.

I don't understand what change it so that this rule is not applied and already installed slot is preferred.

Best Regards,
Georgi

[sam_]

I'm curious to try mrustc ebuild and bootstraping rust 1.74.
What would be the correct way to "merge" the pull request for a rsync portage user?

I tried just backing up my /var/db/repos/gentoo and cloning Kangie's fork/branch in the same place, but eix and emerge started complaining about some digests.

I guess grabbing the patch off github and manually invoking [color]patch -Np1 < kangies-pull-req.patch[/color] should work?

It's far harder if not syncing your repository with git, because the Manifests aren't thin then. You'd have to regenerate all touched files with `ebuild .. digest`.

[logrusx]

So, I again totally forgot about this thread and updated rust. When using the binhost it went smooth, directly to 1.83, which was to be expected. I then run depclean and noticed a rust-1.82 was due cleaning.

I went and manually unmerged 1.83 and run full world update this time without --getbinkg and it offered me to build rust 1.83, which again I think was to be expected since I have rust 1.82.

Then I went and manually unmerged 1.83 and run the update again and it offered me to build both 1.82 and 1.83. No preference for rust-bin.

Finally I run full wold update with --getbinpkg which offered me to merge 1.83 directly, which again I think was to be expected.

Honestly, I don't see any value in building rust from source. But as we see, people who prefer it, have nothing to complain about. No binary will be downloaded by default.

Best Regards,
Georgi