Profile 23.0 upgrade

[Princess Nell]

I read the news item and it's not fully clear what my correct upgrade path is.

Current profile:

[5] default/linux/amd64/17.1/desktop (exp) *

Upgrade profile as per https://wiki.gentoo.org/wiki/Project:To ... date_table: default/linux/alpha/23.0/split-usr/desktop.

However, this system does not use split-usr:

localhost # df /usr
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/root 81987992 22614216 55163088 30% /
localhost #

I assume it's ok to just use

[23] default/linux/amd64/23.0/desktop (stable)

in this situation?

[grknight]

No. All 17.1 profiles except ones marked merged-usr should use the split-usr profiles initially.
If you wish to use the non-split-usr profiles, then you will need the sys-apps/merge-usr migration tool done after.

merged-usr has nothing to do with partitions (besides making separate /usr without initramfs impossible). It moves all of /lib, /lib64 and /bin into their respective /usr directories. It also moves /sbin and /usr/sbin into /usr/bin

Edit: fixed typo

[Hu]

Code: Select all

  [5]   default/linux/amd64/17.1/desktop (exp) *

Why is this marked as experimental?

Upgrade profile as per https://wiki.gentoo.org/wiki/Project:To ... date_table: default/linux/alpha/23.0/split-usr/desktop.

First you showed amd64, now you are showing alpha. Switching architectures like that is wrong. You need an amd64 profile after if you have one before.

However, this system does not use split-usr:

Code: Select all

localhost # df /usr
Filesystem            1K-blocks     Used Available Use% Mounted on
/dev/mapper/root  81987992 22614216  55163088  30% /
localhost # 

This is not the right way to test for split-usr. The naming and multiple migrations of usr is generally confusing. Years ago, we had the "separate /usr is broken" meme from the freedesktop people, deprecating the system configuration of a filesystem mounted at /usr that was distinct from the filesystem mounted at /. Your output shows you do not have separate /usr, which is good. Support for that has been decaying for a long time, and Separate /usr now requires an initramfs was the most recent step in that.

split-usr, which is the reverse of merged-usr, is about a different meme, which seeks to move all files from /bin into /usr/bin (among other moves). Your output does not tell us whether you are on a merged-usr system or not. Follow grknight's guidance on that point.

[grknight]

Code: Select all

  [5]   default/linux/amd64/17.1/desktop (exp) *

Why is this marked as experimental?

Seems a change from about 5 hours ago has swapped them

[Princess Nell]

Sorry, I slipped into the wrong line. default/linux/amd64/23.0/split-usr/desktop is the new profile suggested.

I read up on split-usr in the meantime.

[Hu]

I see. That seems very strange, as there is nothing experimental about the old profiles. They have served many people well for years. They may not be preferred anymore, although as I read the news item, they are not even deprecated yet.

[ipic]

For a open-rc user who doesn't give a f**k about systemd - what does "split user" mean?

[Lumpy Gravy]

I'm having the same question.
Going throught the proccess right now, from 17.1/desktop/plasma to 23.0/split-usr/desktop/plasma.
What would be the benefit of moving to a merge-usr profile?

I've just finished my new Gentoo install yesterday, when 23.0 was still experimental.

[sdauth]

About step 5:

Code: Select all

5. Edit /etc/portage/make.conf; if there is a line defining the CHOST variable,
   remove it. Also delete all lines defining CHOST_... variables.

After upgrade is done, do I need to put back :

CHOST="x86_64-pc-linux-gnu"

in make.conf or not ?

edit : It is defined in arch/amd64/make.defaults so I assume I don't need to add a duplicate entry in /etc/portage/make.conf right ?
Just want to make sure because I've always had that variable in my /etc/portage/make.conf

[Banana]

About step 5:

Code: Select all

5. Edit /etc/portage/make.conf; if there is a line defining the CHOST variable,
   remove it. Also delete all lines defining CHOST_... variables.

After upgrade is done, do I need to put back :

CHOST="x86_64-pc-linux-gnu"

in make.conf or not ?

edit : It is defined in arch/amd64/make.defaults so I assume I don't need to add a duplicate entry in /etc/portage/make.conf right ?
Just want to make sure because I've always had that variable in my /etc/portage/make.conf

no, don't put it back.

[Hu]

For a open-rc user - what does "split user" mean?

It means whatever the freedesktop people decide it to mean today, no more and no less.

From observation, "split /usr" refers to the historical layout where each of /bin, /sbin, /usr/bin, and /usr/sbin were distinct directories, each with a specific and non-overlapping set of programs in them, and /lib and /usr/lib were distinct directories, with non-overlapping[1] sets of libraries in them, and for 64-bit users, /lib64 and /usr/lib64 were distinct directories. Under split-/usr, if /bin/program exists, then none of /sbin/program, /usr/bin/program, or /usr/sbin/program would exist (unless the distribution maintainer was trying to confuse people), so anyone wanting to run program either needs to know the canonical directory for it, or needs to search all four directories. Under merged-/usr, most of this changes. /bin becomes a symlink to /usr/bin. /sbin points (ultimately) to /usr/bin. /usr/sbin points to /usr/bin. /lib points to /usr/lib. Since these are now symlinks, all of /bin/program, /sbin/program, /usr/bin/program, and /usr/sbin/program are aliases to the same program, and its canonical path is now /usr/bin/program.

As a practical matter, under this scheme, if /usr is a separate filesystem and is not mounted by an initramfs, then /bin and /sbin point to a non-existent directory, so you have none of the standard tools available (including /sbin/init, which is now really /usr/sbin/init), and the system is unusable. This is why a separate /usr that is not pre-mounted by the initramfs renders merged-/usr systems completely unusable, and therefore unsupported.

[1]: There is some weirdness with linker scripts in /usr/lib that masquerade as shared libraries and refer the linker to /lib. Ignore those for this discussion.

[ipic]

If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?

Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult.

[sam_]

If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?

Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult.

There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO.

[ipic]

If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?

Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult.

There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO.

Still doesn't tell me why an entire system rebuild is needed. Even the CHOST change wiki page says it's not "strictly" needed - mainly tool chains if I interpret it correctly.

For example, what does -fstack-clash-protection do exactly? Change the way things are compiled? So, a currently compiled program without that would still run - right?

[sam_]

If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?

Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult.

There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO.

Still doesn't tell me why an entire system rebuild is needed. Even the CHOST change wiki page says it's not "strictly" needed - mainly tool chains if I interpret it correctly.

For example, what does -fstack-clash-protection do exactly? Change the way things are compiled? So, a currently compiled program without that would still run - right?

In general, we can't really win. If we make news items extremely verbose, people don't read them, or skip steps. If we make them terse, we apparently miss people's questions

Yes, they all affect compiled binaries, and their performance/size/security properties. DT_RELR uses a different representation for relocations which is more efficient, which reduces binary sizes and startup time. Stack clash protection introduces checks when making stack allocations as a security measure. RELRO is again another security measure where mappings are marked as read-only immediately (it also has a nice property in that it tells you immediately if a binary is broken, not later on at runtime when something enters a codepath). None of them break ABI though so programs will still run.

[ipic]

I have tested a different approach on one of my VMs and this works:

Take a note of all CHOST related settings as per the wiki page on changing CHOST
Follow the instructions up to step 15.
Check the CHOST settings as above, and confirm that NOTHING HAS CHANGED.
If that is the case, instead of an --emptytree rebuild, do this:

Code: Select all

emerge -uNDpv --with-bdeps y @world
emerge -uND --with-bdeps y --jobs --keep-going --quiet-fail y @world

reboot

NOTE: I do not use ANY binary packages. Clearly if you use binary packages testing whether you need an --emptytree rebuild is down to you.

From now on, normal package refresh will take care of moving things onto whatever settings the devs felt were so important.
Given I've been running a system for over 20 years without them - I beg to differ.

[sam_]

You're free to do what you want. It'd be nice to have a bit less derision though. Things are not done for the sake of it.

The 20 years part doesn't really mean anything though. You're not running the same software and likely hardware you were 20 years ago. By all means, feel free to do as you wish, but that part isn't really rationale for it.

[ipic]

You're free to do what you want. It'd be nice to have a bit less derision though. Things are not done for the sake of it.

The 20 years part doesn't really mean anything though. You're not running the same software and likely hardware you were 20 years ago. By all means, feel free to do as you wish, but that part isn't really rationale for it.

The 20 years refers to the period of time that I have been dealing with Gentoo developer instructions. Unlike the hardware - they have not tended to improve.

Giving instructions that will result in a 20+ hour rebuild, that then turns out to not be necessary unless the CHOST settings are changed, sort of deserves a bit of derision, in my humble opinion.

[sam_]

I just explained that they are necessary, though. Then we're just quibbling about "how necessary is necessary". CET needs to be consistent between the toolchain packages. If debugging a problem, we generally want RELRO/BIND_NOW to be consistent across everything (an example would be if something doesn't work, and we'd expect BIND_NOW to normally mean it errors out immediately, the fact it then doesn't implies it's probably something else and leads one down the wrong path). There's also the general question of expected behaviour when everything works, and then when something goes wrong. In theory, none of these things break ABI, and everything works fine. In practice, deploying something to thousands and thousands of users with their own custom configuration requires a strict procedure so that diagnosis of any problems is easier.

It wasn't for the sake of it and it isn't pointless. And a rebuild in general is not some sort of super rare event. I've started writing up why at https://wiki.gentoo.org/wiki/User:Sam/D ... mptytree[b][/b].

You're free to use something else if you hold us in contempt. A bit of politeness doesn't cost anything. You've asked what the changes are, I've tried to explain them briefly. If you want more detail, free free to ask.

[NeddySeagoon]

ipic,

Gentoo has changed a lot over the last 20 years.
Try Historical Gentoo if you want to see how much,

Your two steps

Code: Select all

emerge -uNDpv --with-bdeps y @world
emerge -uND --with-bdeps y --jobs --keep-going --quiet-fail y @world 

can be reduced to

Code: Select all

emerge -uNDav --with-bdeps y --jobs --keep-going --quiet-fail y @world 

so that you only calculate the dependency tree once.

--with-bdeps y is redundant. Its the default anyway.

Take care using mixed gcc. The kernel and all of its modules must be built with the same gcc.
It detects a /17.0/ gcc built kernel and /23.0/ gcc built modules. e.g. virtualbox-modules and refuses to load them.

[ipic]

Thanks Neddy,
The two step emerge I use is just to give me an unstressed look at what will change before committing to it. Not a fan of the --ask setting.

I've walked with Gentoo through those 20 years - I think today is way better than back when I started, no mistake.

On the kernel/modules point - I am assuming that a 17.0 Kernel and modules will boot into a 23.0 system (since kernel + modules all built together).
Personally, I no longer have any out of tree modules.

However, the tests so far on my VMs do not test module loading, since their kernels are all compiled inline. So it's the bare metal machine that will be the real test.

[Hu]

There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO.

Thank you for recapping this. I have not started the migration yet, so I only glanced the news item upgrade instructions, and had not read the associated Wiki page. (I plan to read both of them properly and carefully before doing a migration.) In the meantime, I had been seeing questions about why --emptytree was needed here, and had wondered the same. I assumed, incorrectly, that it was for the more questionable purpose of getting all packages fixed up with the new paths that merged-usr causes. That struck me as an overkill solution, but I sympathized with your concern about giving people so many paths that every user opens a unique problem report, and from there I assumed that the overkill solution was chosen because it minimized the ways people can break things. Thus, I am pleased to see that there is a good reason behind the --emptytree, and that this was documented ahead of time.

Personally, I think it would be really nice if we had a succinct way to tell Portage to rebuild all, and only, packages that are sensitive to the toolchain. Reinstalling documentation packages, or packages containing only scripts, does not benefit from the toolchain improvements. However, that would be a general Portage improvement that is outside the scope of this upgrade, and really determined users could probably find a way to emulate it with the tools we have today. They would, of course, "get to keep the pieces" if their emulated mechanism manages to skip something important.

[GDH-gentoo]

Yes, they all affect compiled binaries, and their performance/size/security properties. DT_RELR uses a different representation for relocations which is more efficient, which reduces binary sizes and startup time. Stack clash protection introduces checks when making stack allocations as a security measure. RELRO is again another security measure where mappings are marked as read-only immediately (it also has a nice property in that it tells you immediately if a binary is broken, not later on at runtime when something enters a codepath). None of them break ABI though so programs will still run.

So, since emerge --emptytree can be costly on many systems, to clarify: the rationale for step #16 of the migration process is simply making sure that all toolchain setting changes are applied to all packages? There isn't a priori knowledge that not doing this (or implicitly doing it incrementally when packages upgrade during regular world updates) will result in breakage, right?

And CHOST removal from make.conf (step #5) was included just to ensure consistency by having profiles specifiy it, since it can (but not necessarily will, e. g. differences in the "vendor" part of the GNU triplet) affect compiled programs and library ABI, right?

[sam_]

Yes, they all affect compiled binaries, and their performance/size/security properties. DT_RELR uses a different representation for relocations which is more efficient, which reduces binary sizes and startup time. Stack clash protection introduces checks when making stack allocations as a security measure. RELRO is again another security measure where mappings are marked as read-only immediately (it also has a nice property in that it tells you immediately if a binary is broken, not later on at runtime when something enters a codepath). None of them break ABI though so programs will still run.

So, since emerge --emptytree can be costly on many systems, to clarify: the rationale for step #16 of the migration process is simply making sure that all toolchain setting changes are applied to all packages? There isn't a priori knowledge that not doing this (or implicitly doing it incrementally when packages upgrade during regular world updates) will result in breakage, right?

And CHOST removal from make.conf (step #5) was included just to ensure consistency by having profiles specifiy it, since it can (but not necessarily will, e. g. differences in the "vendor" part of the GNU triplet) affect compiled programs and library ABI, right?

Yes, with the exception of...
a) profiles where CHOST actually changed (not just moving where it is defined), in which case I'd say it's worth doing the whole lot because I can see misc. other packages getting confused, or
b) ppc64 where the ABI did change for IEEE long double.

I can also imagine that if we get to the point where we can make CET enforced by default, it might be required, but that's a long way off.

Off the top of my head. Sorry, I'm juggling a lot of balls today.

[sam_]

There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO.

Thank you for recapping this. I have not started the migration yet, so I only glanced the news item upgrade instructions, and had not read the associated Wiki page. (I plan to read both of them properly and carefully before doing a migration.) In the meantime, I had been seeing questions about why --emptytree was needed here, and had wondered the same. I assumed, incorrectly, that it was for the more questionable purpose of getting all packages fixed up with the new paths that merged-usr causes. That struck me as an overkill solution, but I sympathized with your concern about giving people so many paths that every user opens a unique problem report, and from there I assumed that the overkill solution was chosen because it minimized the ways people can break things. Thus, I am pleased to see that there is a good reason behind the --emptytree, and that this was documented ahead of time.

Personally, I think it would be really nice if we had a succinct way to tell Portage to rebuild all, and only, packages that are sensitive to the toolchain. Reinstalling documentation packages, or packages containing only scripts, does not benefit from the toolchain improvements. However, that would be a general Portage improvement that is outside the scope of this upgrade, and really determined users could probably find a way to emulate it with the tools we have today. They would, of course, "get to keep the pieces" if their emulated mechanism manages to skip something important.

Swamped so will keep it brief, but: thank you!

I think it'd be worth us figuring out some way to only re-emerge things with ELF in the recorded files. It can be done right now via grepping the VDB but that feels gross.