| View previous topic :: View next topic |
| Author |
Message |
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 916
|
 Posted: Fri Mar 22, 2024 9:27 pm Post subject: Profile 23.0 upgrade |
 |
|
I read the news item and it's not fully clear what my correct upgrade path is.
Current profile:
| Quote: |
[5] default/linux/amd64/17.1/desktop (exp) *
|
Upgrade profile as per https://wiki.gentoo.org/wiki/Project:Toolchain/23.0_update_table: default/linux/alpha/23.0/split-usr/desktop.
However, this system does not use split-usr:
| Quote: |
localhost # df /usr
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/root 81987992 22614216 55163088 30% /
localhost #
|
I assume it's ok to just use
| Quote: |
[23] default/linux/amd64/23.0/desktop (stable)
|
in this situation? |
|
| Back to top |
|
 |
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1660
|
| Posted: Fri Mar 22, 2024 10:00 pm Post subject: |
|
|
No. All 17.1 profiles except ones marked merged-usr should use the split-usr profiles initially.
If you wish to use the non-split-usr profiles, then you will need the sys-apps/merge-usr migration tool done after.
merged-usr has nothing to do with partitions (besides making separate /usr without initramfs impossible). It moves all of /lib, /lib64 and /bin into their respective /usr directories. It also moves /sbin and /usr/sbin into /usr/bin
Edit: fixed typo
Last edited by grknight on Sat Mar 23, 2024 12:22 am; edited 2 times in total |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Fri Mar 22, 2024 10:57 pm Post subject: Re: Profile 23.0 upgrade |
|
|
| Princess Nell wrote: | | Code: | | [5] default/linux/amd64/17.1/desktop (exp) * |
|
Why is this marked as experimental?
First you showed amd64, now you are showing alpha. Switching architectures like that is wrong. You need an amd64 profile after if you have one before.
| Princess Nell wrote: | However, this system does not use split-usr: | Code: | localhost # df /usr
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/root 81987992 22614216 55163088 30% /
localhost # |
|
This is not the right way to test for split-usr. The naming and multiple migrations of usr is generally confusing. Years ago, we had the "separate /usr is broken" meme from the freedesktop people, deprecating the system configuration of a filesystem mounted at /usr that was distinct from the filesystem mounted at /. Your output shows you do not have separate /usr, which is good. Support for that has been decaying for a long time, and Separate /usr now requires an initramfs was the most recent step in that.
split-usr, which is the reverse of merged-usr, is about a different meme, which seeks to move all files from /bin into /usr/bin (among other moves). Your output does not tell us whether you are on a merged-usr system or not. Follow grknight's guidance on that point. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1660
|
| Posted: Fri Mar 22, 2024 11:06 pm Post subject: |
|
|
| Hu wrote: | | Princess Nell wrote: | | Code: | | [5] default/linux/amd64/17.1/desktop (exp) * |
|
Why is this marked as experimental? |
Seems a change from about 5 hours ago has swapped them |
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 916
|
| Posted: Sat Mar 23, 2024 12:10 am Post subject: |
|
|
Sorry, I slipped into the wrong line. default/linux/amd64/23.0/split-usr/desktop is the new profile suggested.
I read up on split-usr in the meantime. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Sat Mar 23, 2024 1:19 am Post subject: |
|
|
| I see. That seems very strange, as there is nothing experimental about the old profiles. They have served many people well for years. They may not be preferred anymore, although as I read the news item, they are not even deprecated yet. |
|
| Back to top |
|
|
ipic
Guru
Joined: 29 Dec 2003
Posts: 377
Location: UK
|
| Posted: Sat Mar 23, 2024 9:05 am Post subject: |
|
|
| For a open-rc user who doesn't give a f**k about systemd - what does "split user" mean? |
|
| Back to top |
|
|
Lumpy Gravy
n00b
Joined: 28 Dec 2016
Posts: 41
Location: France
|
| Posted: Sat Mar 23, 2024 9:56 am Post subject: |
|
|
I'm having the same question.
Going throught the proccess right now, from 17.1/desktop/plasma to 23.0/split-usr/desktop/plasma.
What would be the benefit of moving to a merge-usr profile?
I've just finished my new Gentoo install yesterday, when 23.0 was still experimental.  |
|
| Back to top |
|
|
sdauth
Guru
Joined: 19 Sep 2018
Posts: 569
Location: Ásgarðr
|
| Posted: Sat Mar 23, 2024 10:31 am Post subject: |
|
|
About step 5:
| Code: | 5. Edit /etc/portage/make.conf; if there is a line defining the CHOST variable,
remove it. Also delete all lines defining CHOST_... variables. |
After upgrade is done, do I need to put back :
CHOST="x86_64-pc-linux-gnu"
in make.conf or not ?
edit : It is defined in arch/amd64/make.defaults so I assume I don't need to add a duplicate entry in /etc/portage/make.conf right ?
Just want to make sure because I've always had that variable in my /etc/portage/make.conf  |
|
| Back to top |
|
|
Banana
Veteran
Joined: 21 May 2004
Posts: 1392
Location: Germany
|
| Posted: Sat Mar 23, 2024 11:31 am Post subject: |
|
|
| sdauth wrote: | About step 5:
| Code: | 5. Edit /etc/portage/make.conf; if there is a line defining the CHOST variable,
remove it. Also delete all lines defining CHOST_... variables. |
After upgrade is done, do I need to put back :
CHOST="x86_64-pc-linux-gnu"
in make.conf or not ?
edit : It is defined in arch/amd64/make.defaults so I assume I don't need to add a duplicate entry in /etc/portage/make.conf right ?
Just want to make sure because I've always had that variable in my /etc/portage/make.conf |
no, don't put it back.
_________________
My personal space
My delta-labs.org snippets do expire
PFL - Portage file list - find which package a file or command belongs to. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Sat Mar 23, 2024 3:07 pm Post subject: |
|
|
| ipic wrote: | | For a open-rc user - what does "split user" mean? |
It means whatever the freedesktop people decide it to mean today, no more and no less.
From observation, "split /usr" refers to the historical layout where each of /bin, /sbin, /usr/bin, and /usr/sbin were distinct directories, each with a specific and non-overlapping set of programs in them, and /lib and /usr/lib were distinct directories, with non-overlapping[1] sets of libraries in them, and for 64-bit users, /lib64 and /usr/lib64 were distinct directories. Under split-/usr, if /bin/program exists, then none of /sbin/program, /usr/bin/program, or /usr/sbin/program would exist (unless the distribution maintainer was trying to confuse people), so anyone wanting to run program either needs to know the canonical directory for it, or needs to search all four directories. Under merged-/usr, most of this changes. /bin becomes a symlink to /usr/bin. /sbin points (ultimately) to /usr/bin. /usr/sbin points to /usr/bin. /lib points to /usr/lib. Since these are now symlinks, all of /bin/program, /sbin/program, /usr/bin/program, and /usr/sbin/program are aliases to the same program, and its canonical path is now /usr/bin/program.
As a practical matter, under this scheme, if /usr is a separate filesystem and is not mounted by an initramfs, then /bin and /sbin point to a non-existent directory, so you have none of the standard tools available (including /sbin/init, which is now really /usr/sbin/init), and the system is unusable. This is why a separate /usr that is not pre-mounted by the initramfs renders merged-/usr systems completely unusable, and therefore unsupported.
[1]: There is some weirdness with linker scripts in /usr/lib that masquerade as shared libraries and refer the linker to /lib. Ignore those for this discussion. |
|
| Back to top |
|
|
ipic
Guru
Joined: 29 Dec 2003
Posts: 377
Location: UK
|
| Posted: Sun Mar 24, 2024 12:12 pm Post subject: |
|
|
If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?
Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult. |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Mar 24, 2024 12:19 pm Post subject: |
|
|
| ipic wrote: | If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?
Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult. |
There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO. |
|
| Back to top |
|
|
ipic
Guru
Joined: 29 Dec 2003
Posts: 377
Location: UK
|
| Posted: Sun Mar 24, 2024 12:36 pm Post subject: |
|
|
| sam_ wrote: | | ipic wrote: | If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?
Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult. |
There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO. |
Still doesn't tell me why an entire system rebuild is needed. Even the CHOST change wiki page says it's not "strictly" needed - mainly tool chains if I interpret it correctly.
For example, what does -fstack-clash-protection do exactly? Change the way things are compiled? So, a currently compiled program without that would still run - right? |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Mar 24, 2024 12:42 pm Post subject: |
|
|
| ipic wrote: | | sam_ wrote: | | ipic wrote: | If one is changing from split-usr (17.1 implicit) to split-usr (23.0 explicit), and CHOST does not change, why is it necessary to rebuild the whole system?
Given one of my systems has close to 2000 packages, and a rebuild will take in the order of 20 hours, I feel like I need a good explanation of why it is necessary before wasting that much time, and incurring the risk of a power failure making things very difficult. |
There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO. |
Still doesn't tell me why an entire system rebuild is needed. Even the CHOST change wiki page says it's not "strictly" needed - mainly tool chains if I interpret it correctly.
For example, what does -fstack-clash-protection do exactly? Change the way things are compiled? So, a currently compiled program without that would still run - right? |
In general, we can't really win. If we make news items extremely verbose, people don't read them, or skip steps. If we make them terse, we apparently miss people's questions
Yes, they all affect compiled binaries, and their performance/size/security properties. DT_RELR uses a different representation for relocations which is more efficient, which reduces binary sizes and startup time. Stack clash protection introduces checks when making stack allocations as a security measure. RELRO is again another security measure where mappings are marked as read-only immediately (it also has a nice property in that it tells you immediately if a binary is broken, not later on at runtime when something enters a codepath). None of them break ABI though so programs will still run. |
|
| Back to top |
|
|
ipic
Guru
Joined: 29 Dec 2003
Posts: 377
Location: UK
|
| Posted: Sun Mar 24, 2024 1:42 pm Post subject: |
|
|
I have tested a different approach on one of my VMs and this works:
Take a note of all CHOST related settings as per the wiki page on changing CHOST
Follow the instructions up to step 15.
Check the CHOST settings as above, and confirm that NOTHING HAS CHANGED.
If that is the case, instead of an --emptytree rebuild, do this:
| Code: |
emerge -uNDpv --with-bdeps y @world
emerge -uND --with-bdeps y --jobs --keep-going --quiet-fail y @world
|
reboot
NOTE: I do not use ANY binary packages. Clearly if you use binary packages testing whether you need an --emptytree rebuild is down to you.
From now on, normal package refresh will take care of moving things onto whatever settings the devs felt were so important.
Given I've been running a system for over 20 years without them - I beg to differ. |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Mar 24, 2024 1:44 pm Post subject: |
|
|
You're free to do what you want. It'd be nice to have a bit less derision though. Things are not done for the sake of it.
The 20 years part doesn't really mean anything though. You're not running the same software and likely hardware you were 20 years ago. By all means, feel free to do as you wish, but that part isn't really rationale for it. |
|
| Back to top |
|
|
ipic
Guru
Joined: 29 Dec 2003
Posts: 377
Location: UK
|
| Posted: Sun Mar 24, 2024 1:57 pm Post subject: |
|
|
| sam_ wrote: | You're free to do what you want. It'd be nice to have a bit less derision though. Things are not done for the sake of it.
The 20 years part doesn't really mean anything though. You're not running the same software and likely hardware you were 20 years ago. By all means, feel free to do as you wish, but that part isn't really rationale for it. |
The 20 years refers to the period of time that I have been dealing with Gentoo developer instructions. Unlike the hardware - they have not tended to improve.
Giving instructions that will result in a 20+ hour rebuild, that then turns out to not be necessary unless the CHOST settings are changed, sort of deserves a bit of derision, in my humble opinion. |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Mar 24, 2024 2:00 pm Post subject: |
|
|
I just explained that they are necessary, though. Then we're just quibbling about "how necessary is necessary". CET needs to be consistent between the toolchain packages. If debugging a problem, we generally want RELRO/BIND_NOW to be consistent across everything (an example would be if something doesn't work, and we'd expect BIND_NOW to normally mean it errors out immediately, the fact it then doesn't implies it's probably something else and leads one down the wrong path). There's also the general question of expected behaviour when everything works, and then when something goes wrong. In theory, none of these things break ABI, and everything works fine. In practice, deploying something to thousands and thousands of users with their own custom configuration requires a strict procedure so that diagnosis of any problems is easier.
It wasn't for the sake of it and it isn't pointless. And a rebuild in general is not some sort of super rare event. I've started writing up why at https://wiki.gentoo.org/wiki/User:Sam/Drafts/On_occasional_emptytree.
You're free to use something else if you hold us in contempt. A bit of politeness doesn't cost anything. You've asked what the changes are, I've tried to explain them briefly. If you want more detail, free free to ask. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54250
Location: 56N 3W
|
| Posted: Sun Mar 24, 2024 2:10 pm Post subject: |
|
|
ipic,
Gentoo has changed a lot over the last 20 years.
Try Historical Gentoo if you want to see how much,
Your two steps | Code: | emerge -uNDpv --with-bdeps y @world
emerge -uND --with-bdeps y --jobs --keep-going --quiet-fail y @world |
can be reduced to | Code: | | emerge -uNDav --with-bdeps y --jobs --keep-going --quiet-fail y @world |
so that you only calculate the dependency tree once.
--with-bdeps y is redundant. Its the default anyway.
Take care using mixed gcc. The kernel and all of its modules must be built with the same gcc.
It detects a /17.0/ gcc built kernel and /23.0/ gcc built modules. e.g. virtualbox-modules and refuses to load them.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
ipic
Guru
Joined: 29 Dec 2003
Posts: 377
Location: UK
|
| Posted: Sun Mar 24, 2024 2:27 pm Post subject: |
|
|
Thanks Neddy,
The two step emerge I use is just to give me an unstressed look at what will change before committing to it. Not a fan of the --ask setting.
I've walked with Gentoo through those 20 years - I think today is way better than back when I started, no mistake.
On the kernel/modules point - I am assuming that a 17.0 Kernel and modules will boot into a 23.0 system (since kernel + modules all built together).
Personally, I no longer have any out of tree modules.
However, the tests so far on my VMs do not test module loading, since their kernels are all compiled inline. So it's the bare metal machine that will be the real test. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Sun Mar 24, 2024 3:34 pm Post subject: |
|
|
| sam_ wrote: | | There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO. |
Thank you for recapping this. I have not started the migration yet, so I only glanced the news item upgrade instructions, and had not read the associated Wiki page. (I plan to read both of them properly and carefully before doing a migration.) In the meantime, I had been seeing questions about why --emptytree was needed here, and had wondered the same. I assumed, incorrectly, that it was for the more questionable purpose of getting all packages fixed up with the new paths that merged-usr causes. That struck me as an overkill solution, but I sympathized with your concern about giving people so many paths that every user opens a unique problem report, and from there I assumed that the overkill solution was chosen because it minimized the ways people can break things. Thus, I am pleased to see that there is a good reason behind the --emptytree, and that this was documented ahead of time.
Personally, I think it would be really nice if we had a succinct way to tell Portage to rebuild all, and only, packages that are sensitive to the toolchain. Reinstalling documentation packages, or packages containing only scripts, does not benefit from the toolchain improvements. However, that would be a general Portage improvement that is outside the scope of this upgrade, and really determined users could probably find a way to emulate it with the tools we have today. They would, of course, "get to keep the pieces" if their emulated mechanism manages to skip something important.  |
|
| Back to top |
|
|
GDH-gentoo
Veteran
Joined: 20 Jul 2019
Posts: 1532
Location: South America
|
| Posted: Sun Mar 24, 2024 3:57 pm Post subject: |
|
|
| sam_ wrote: | | Yes, they all affect compiled binaries, and their performance/size/security properties. DT_RELR uses a different representation for relocations which is more efficient, which reduces binary sizes and startup time. Stack clash protection introduces checks when making stack allocations as a security measure. RELRO is again another security measure where mappings are marked as read-only immediately (it also has a nice property in that it tells you immediately if a binary is broken, not later on at runtime when something enters a codepath). None of them break ABI though so programs will still run. |
So, since emerge --emptytree can be costly on many systems, to clarify: the rationale for step #16 of the migration process is simply making sure that all toolchain setting changes are applied to all packages? There isn't a priori knowledge that not doing this (or implicitly doing it incrementally when packages upgrade during regular world updates) will result in breakage, right?
And CHOST removal from make.conf (step #5) was included just to ensure consistency by having profiles specifiy it, since it can (but not necessarily will, e. g. differences in the "vendor" part of the GNU triplet) affect compiled programs and library ABI, right?
_________________
| NeddySeagoon wrote: | | I'm not a witch, I'm a retired electronics engineer |
| Ionen wrote: | | As a packager I just don't want things to get messier with weird build systems and multiple toolchains requirements though |
|
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Mar 24, 2024 4:16 pm Post subject: |
|
|
| GDH-gentoo wrote: | | sam_ wrote: | | Yes, they all affect compiled binaries, and their performance/size/security properties. DT_RELR uses a different representation for relocations which is more efficient, which reduces binary sizes and startup time. Stack clash protection introduces checks when making stack allocations as a security measure. RELRO is again another security measure where mappings are marked as read-only immediately (it also has a nice property in that it tells you immediately if a binary is broken, not later on at runtime when something enters a codepath). None of them break ABI though so programs will still run. |
So, since emerge --emptytree can be costly on many systems, to clarify: the rationale for step #16 of the migration process is simply making sure that all toolchain setting changes are applied to all packages? There isn't a priori knowledge that not doing this (or implicitly doing it incrementally when packages upgrade during regular world updates) will result in breakage, right?
And CHOST removal from make.conf (step #5) was included just to ensure consistency by having profiles specifiy it, since it can (but not necessarily will, e. g. differences in the "vendor" part of the GNU triplet) affect compiled programs and library ABI, right? |
Yes, with the exception of...
a) profiles where CHOST actually changed (not just moving where it is defined), in which case I'd say it's worth doing the whole lot because I can see misc. other packages getting confused, or
b) ppc64 where the ABI did change for IEEE long double.
I can also imagine that if we get to the point where we can make CET enforced by default, it might be required, but that's a long way off.
Off the top of my head. Sorry, I'm juggling a lot of balls today. |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Mar 24, 2024 4:20 pm Post subject: |
|
|
| Hu wrote: | | sam_ wrote: | | There are other changes mentioned, like DT_RELR, -fstack-clash-protection, and RELRO. |
Thank you for recapping this. I have not started the migration yet, so I only glanced the news item upgrade instructions, and had not read the associated Wiki page. (I plan to read both of them properly and carefully before doing a migration.) In the meantime, I had been seeing questions about why --emptytree was needed here, and had wondered the same. I assumed, incorrectly, that it was for the more questionable purpose of getting all packages fixed up with the new paths that merged-usr causes. That struck me as an overkill solution, but I sympathized with your concern about giving people so many paths that every user opens a unique problem report, and from there I assumed that the overkill solution was chosen because it minimized the ways people can break things. Thus, I am pleased to see that there is a good reason behind the --emptytree, and that this was documented ahead of time.
Personally, I think it would be really nice if we had a succinct way to tell Portage to rebuild all, and only, packages that are sensitive to the toolchain. Reinstalling documentation packages, or packages containing only scripts, does not benefit from the toolchain improvements. However, that would be a general Portage improvement that is outside the scope of this upgrade, and really determined users could probably find a way to emulate it with the tools we have today. They would, of course, "get to keep the pieces" if their emulated mechanism manages to skip something important. |
Swamped so will keep it brief, but: thank you!
I think it'd be worth us figuring out some way to only re-emerge things with ELF in the recorded files. It can be done right now via grepping the VDB but that feels gross. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
