Profile 23.0 upgrade

[ipic]

There used to be a group - @system - that was intended for this.

Seriously - forcing a rebuild on a system where it would take the best part of a 24 hour day should never be an option.
"Enforcing" things has never been the Gentoo way either.

[Hu]

No, @system is the set of packages that all ebuilds can assume are installed. It may contain non-program packages, such as virtuals, which do not need to be rebuilt here. It usually will omit at least some packages that deserve a rebuild on a toolchain improvement, because those packages are not system critical. Thus, it fails in both directions. sam_'s comment about searching the VDB is what I had in mind, and yes, that is a somewhat ugly approach, but I cannot think of an alternative that will (1) find all packages which can benefit, (2) exclude all packages that clearly cannot benefit (documentation, scripts), (3) avoid trying to rebuild things which are not packages (/usr/local/bin/ programs). It's still not perfect, since it will flag for inclusion things which distribute ELF files prebuilt upstream (such as firefox-bin), but it excludes quite a bit that --emptytree would include. It might be possible to abuse some of the ebuild's QA_ variables to identify prebuilt files and exclude them from the package's effective contents before deciding whether it ships anything worth rebuilding.

As for forcing - users are of course free to ignore the recommended path. This path is recommended because it is the simplest way to ensure that all packages which can benefit are rebuilt so that they do benefit. Identifying specific packages out of the VDB would be nice, but I expect that most packages excluded that way are cheap to build. If you have a long total build time, it is likely because you have many packages which are compiled by your toolchain, so excluding the unnecessary ones will not help you much. It would mainly be a savings on write cycles for limited life storage devices.

[NeddySeagoon]

ipic,

Nothing is enforced, only recommended. The further you stray from the beaten path, the less sympathy and help you are likely to get. That's not a reason to not do it, as long as you understand you can keep the pieces if it breaks.

I still use a static /dev, separate /usr and root in LVM, all without an initrd. I'm old any cynical, I know that that when it breaks, I won't even get a shoulder to cry on. :)
I had to add an initrd to mount /usr the other day but I was warned beforehand, so when it broke, it was just another entry on the syslinux menu.

"Because I can" is an excellent reason to do something with Gentoo :)

[ipic]

No, @system is the set of packages that all ebuilds can assume are installed.

So how about coming up with a different one aimed at this problem - like @toolchain for example.

.....
I can also imagine that if we get to the point where we can make CET enforced by default, it might be required, but that's a long way off.
....

Looks a lot to me like enforcing is on the table.

[sam_]

Enforcing has a specific meaning wrt security measures. In this case, it means "[allowing users to] ask glibc to enforce it" (and testing that a lot, and fixing any breakage, and documenting it), not enforcing it upon all users (i.e. making users use CET whether they like it or not).

That is, enforcing of the measure onto applications, not onto users. Why would I want to "enforce" anything on our users?

I have no idea if we could get to a point where that was done on all CPUs without opting-in at any point, and even then, it's not like it'd be mandatory.

[ipic]

ipic,

Nothing is enforced, only recommended. The further you stray from the beaten path, the less sympathy and help you are likely to get. That's not a reason to not do it, as long as you understand you can keep the pieces if it breaks.

I still use a static /dev, separate /usr and root in LVM, all without an initrd. I'm old any cynical, I know that that when it breaks, I won't even get a shoulder to cry on.
I had to add an initrd to mount /usr the other day but I was warned beforehand, so when it broke, it was just another entry on the syslinux menu.

"Because I can" is an excellent reason to do something with Gentoo

Exactly so.

What I'm finding confusing about this is that people on this thread really seem to think that one can get support from Gentoo when something breaks.

This has never been the case. If an e-build fails to build, one can submit a bug that may, eventually, be looked at by someone, who is 90% likely to tell you it's your problem. In the mean time, you find patches and sort it yourself.

If a runtime fails to work - looking for Gentoo support is by definition pointless - upstream is where the help or otherwise will come from.

And, packages such as Freecad, Kicad, Libreoffice, Chromium, anything to do with Qt, etc, etc take many, many hours to *each* to build. Not sure where the "you wouldn't save much time over --emptytree" comes from either.

[Hu]

So how about coming up with a different one aimed at this problem - like @toolchain for example.

That's exactly what I was wishing for above. However, it doesn't exist today.

What I'm finding confusing about this is that people on this thread really seem to think that one can get support from Gentoo when something breaks.

This has never been the case. If an e-build fails to build, one can submit a bug that may, eventually, be looked at by someone, who is 90% likely to tell you it's your problem. In the mean time, you find patches and sort it yourself.

Support means different things to different people. Nobody expects that they can get a helpful response by a guaranteed deadline after submission, as might be available with a commercial support contract issued by a major vendor. Reports about build failures may or may not get aid here or on the bug tracker. As a notable counterexample, I refer you to Forced to update the old system, where a concerted effort from multiple forum posters managed to update a system that had been neglected for years, and the system owner was not able to solve the update problems unaided.

If a runtime fails to work - looking for Gentoo support is by definition pointless - upstream is where the help or otherwise will come from.

That depends on why it fails to work. If Gentoo broke it with a bad customization, Gentoo may be the place to see it fixed. In some cases, the Gentoo developer will have a better understanding of the package and its upstream than you do, and can facilitate seeking upstream help.

And, packages such as Freecad, Kicad, Libreoffice, Chromium, anything to do with Qt, etc, etc take many, many hours to *each* to build. Not sure where the "you wouldn't save much time over --emptytree" comes from either.

All of those are things you build with the toolchain, so a hypothetical @toolchain would rebuild them, just as --emptytree @world does rebuild them. The @toolchain target would skip rebuilding Python documentation, reinstalling pure-Perl packages, etc. That saves a bit of time, but not much, because most of the time is spent emerging the packages that @toolchain will choose to rebuild.

[ipic]

Take a note of all CHOST related settings as per the wiki page on changing CHOST
Follow the instructions up to step 15.
Check the CHOST settings as above, and confirm that NOTHING HAS CHANGED.
If that is the case, instead of an --emptytree rebuild, do this:

Code: Select all

emerge -uNDpv --with-bdeps y @world
emerge -uND --with-bdeps y --jobs --keep-going --quiet-fail y @world

reboot

Just to report that I have used this approach on my bare metal machine - the one with the 2000 odd packages.
The approach resulted in around 40 packages being emerged. GCC was the one that took the longest, and avoiding doing it twice saved a lot.

The points that I could not test elsewhere were the kernel + modules boot (it worked) and all the packages with MULTILIB - of which I have a metric ton.

Pleased to report that system rebooted clean.
All QEMU VMs started clean
Things that need MULTILIB are all running as before (Wine mainly).
Apps I regularly use all running clean.

So, I'm going to sign off now. Some people here do not want to see the problem, or do anything about it - which is their right, but talking to them just pisses me off. So, bye.

[Hu]

That only built packages which were out of date or which had a pending USE flag change. Per earlier remarks in the thread, any packages for which you have the current version, but which were built with an older toolchain, have not been rebuilt and are not benefiting from the improvements built into the new profile. You will eventually rebuild those with the new toolchain as updates become available, but for some projects, that could be quite a ways in the future.

[UlvHare]

Do I need to move to merged-usr (

Code: Select all

[27]  default/linux/amd64/23.0/desktop/plasma (stable)

) from my current

Code: Select all

[8]   default/linux/amd64/17.1/desktop/plasma (exp)

if I have /usr in the same partition as /? Are there any benefits in it for "normal user"? I mean not a developer, just work (R, QGIS) and enjoy ebooks and films. But widely using ebuilds from zugaina.

[flysideways]

UlvHare,

The table found here: https://wiki.gentoo.org/wiki/Project:To ... date_table , suggests, default/linux/amd64/17.1/desktop/plasma --> default/linux/amd64/23.0/split-usr/desktop/plasma

[flysideways]

Four of the six provided links are broken here

Code: Select all

[1] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_transition
[2] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_timeline
[3] https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are-now-stable.html
[4] https://www.gentoo.org/support/news-items/2022-12-01-systemd-usrmerge.html
[5] https://wiki.g.o/wiki/Project:Toolchain/23.0_update_table
[6] https://wiki.g.o/wiki/Changing_the_CHOST_variable#Verifying_things_work

from https://wiki.gentoo.org/wiki/Project:To ... structions .

Otherwise, the instructions were adequate for me to follow to a successful resolution.

Thanks @Gentoo Devs

[houtworm]

I am almost at the end of the news message about this topic. Now I have to rebuild world. I have a lot of binary packages because this pc also builds for other pc's. Can I still use these binary packages, or do I have to rebuild everything?

[Hu]

Do I need to move to merged-usr (

Code: Select all

[27]  default/linux/amd64/23.0/desktop/plasma (stable)

) from my current

Code: Select all

[8]   default/linux/amd64/17.1/desktop/plasma (exp)

if I have /usr in the same partition as /? Are there any benefits in it for "normal user"? I mean not a developer, just work (R, QGIS) and enjoy ebooks and films. But widely using ebuilds from zugaina.

As I explained above, the filesystem on which /usr resides is the "separate /usr" discussion. The "split /usr" vs "merged /usr" discussion is about whether /bin is a symlink to /usr/bin. As to whether you should move, I suggest that you follow the upgrade table, and handle "merged /usr" when the Gentoo developers explicitly instruct you to do so, and not sooner.

[CaptainBlood]

Four of the six provided links are broken here

Code: Select all

[1] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_transition
[2] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_timeline
[3] https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are-now-stable.html
[4] https://www.gentoo.org/support/news-items/2022-12-01-systemd-usrmerge.html
[5] https://wiki.g.o/wiki/Project:Toolchain/23.0_update_table
[6] https://wiki.g.o/wiki/Changing_the_CHOST_variable#Verifying_things_work

from https://wiki.gentoo.org/wiki/Project:To ... structions .

Otherwise, the instructions were adequate for me to follow to a successful resolution.

Thanks @Gentoo Devs

Indeed
The "wiki.g.o" should be replaced by "wiki.gentoo.org".
Maybe done on purpose?
However unconfortable imo.

Thks 4 ur attention, interest & support.

[mkyral]

It looks like openrc binpkg installs binaries to wrong place (/usr/sbin) and kernel can't find /sbin/openrc to run and boot fails. I had to force portage to compile openrc package. Then correct folders are used and my system boots again.

I'm on default/linux/amd64/23.0/split-usr/desktop/plasma profile, split-usr USE flag is set, but there is no such flag on openrc.

[figueroa]

Sam_, your explanations and your patience are greatly appreciated. I haven't found the instructions difficult to follow. But, I have a question.

On an old x86 openrc system (CHOST="i686-pc-linux-gnu" with an AMD Phenom CPU) migrating from [5] to [28], and having just compiled binutils, gcc, and glibc with no CHOST or version changes, would it be sound to use selected excludes in the --emptytree rebuild in order to speed things up.

Code: Select all

emerge --ask --emptytree --exclude "sys-devel/gcc sys-devel/binutils sys-libs/glibc sys-kernel/linux-firmware sys-kernel/gentoo-sources sys-kernel/linux-headers sys-apps/baselayout dev-build/libtool" @world

On this system, with no binary packages or other complications, just rebuilding gcc takes two hours, so I can probably save quite a few electrons with those excludes.

Finally, after migration, shouldn't the kernel be rebuilt also?

ADDED: Notes at 15:04 20240326
1. I did recompile the kernel because it just seemed wrong to not do so.
2. I did use all the excludes noted above.
3. After 24 hours doing emerge --emptytree, it failed on Audacity. I did emerge --skipfirst --resume and the remainder finished in another two hours.
4. Rebooted, did emerge --sync and normal updates and everything appears to be as it should.

[e8root]

It wasn't for the sake of it and it isn't pointless. And a rebuild in general is not some sort of super rare event. I've started writing up why at https://wiki.gentoo.org/wiki/User:Sam/D ... mptytree[b][/b].

Great article.
Good to know I need to rebuild with --emptytree after changing CFLAGS. Numbers of packages reported by neofetch and when emerging world didn't add up... now with --emptytree they do

Emerging world with --emptytree seems like it will take some more hours compared to normal world because number of packages increased drastically but it also looks like these mostly are small packages that sit on configure phase most of the time and so overall CPU utilization is very low so not as bad as just chromium or libreoffice alone.

[UlvHare]

As to whether you should move, I suggest that you follow the upgrade table, and handle "merged /usr" when the Gentoo developers explicitly instruct you to do so, and not sooner.

Thank you very much!

[logrusx]

It can be done right now via grepping the VDB but that feels gross.

I already finished the migration but still I think whoever wants to get their hands dirty should know how to do it, gross or not. Also I'm curious. Could you please tell me how?

Best Regards,
Georgi

[Havin_it]

Sam_, your explanations and your patience are greatly appreciated. I haven't found the instructions difficult to follow. But, I have a question.

On an old x86 openrc system (CHOST="i686-pc-linux-gnu" with an AMD Phenom CPU) migrating from [5] to [28], and having just compiled binutils, gcc, and glibc with no CHOST or version changes, would it be sound to use selected excludes in the --emptytree rebuild in order to speed things up.

Code: Select all

emerge --ask --emptytree --exclude "sys-devel/gcc sys-devel/binutils sys-libs/glibc sys-kernel/linux-firmware sys-kernel/gentoo-sources sys-kernel/linux-headers sys-apps/baselayout dev-build/libtool" @world

On this system, with no binary packages or other complications, just rebuilding gcc takes two hours, so I can probably save quite a few electrons with those excludes.

This is what I'm planning to do. Seems no sense to re-emerge packages with no binaries to build. With that in mind I'm screening out some populous groups of non-binary ebuilds with some use of qlist from portage-utils:

Code: Select all

emerge -eav --keep-going --exclude "gcc glibc binutils libtool linux-firmware linux-headers gentoo-sources intel-microcode baselayout $(qlist -IC -- -bin) $(qlist -IC acct-) $(qlist -IC virtual/)" @world

(You may want to run those subcommands individually to ensure they don't include any compiling packages on your system.)
There are probably other categories you could target if you want to go all-out: PHP PEAR packages, if you have those; apps compiled entirely from langs other than GCC's (Java, OCaml, ...). Others I'm not sure about ... do any fonts make use of a compiler? Man pages? Probably not but I'm not certain. And past a certain point the time you spend identifying excludes outweighs the time you'd save

Finally, after migration, shouldn't the kernel be rebuilt also?

Sounds like it's not essential but like you I'll probably do it anyway. The decision taken must be applied also to any external modules of course (e.g. virtualbox-modules in my case).

3. After 24 hours doing emerge --emptytree, it failed on Audacity. I did emerge --skipfirst --resume and the remainder finished in another two hours.

I have had that build failing on me the last 2-3 (~monthly) upgrade cycles. I haven't needed the app in some time so haven't cared but maybe one of us should file a bug at some point, when there's nothing else to do

[thrashed]

yesterday i migrated from "default/linux/amd64/17.1" to "default/linux/amd64/23.0"

i read https://www.gentoo.org/support/news-ite ... files.html calmly and in detail.
and to be honest i skipped point 16. --> Rebuild world: "emerge --ask --emptytree @world".

instead of point 16, i just updated world, like i always did all the years and on this update the successfull build of sys-apps/baselayout-2.14-r2 broke my system. after this successfull build, every other emerge was not possible anymore (PORTAGE_BZIP2_COMMAND setting is invalid: 'bzip2' error messages and so on). i figured out, that /sbin and /bin were missing in my PATH variable. adjusting the PATH variabel in /etc/profile did not help, because every env-update freshly writes the PATH variable in /etc/profile. In /etc/env.d/50baselayout i found, that /sbin and /bin were missing in the PATH variable and in ROOTPATH variable. after adapting /sbin and /bin to the variables in /etc/env.d/50baselayout everyhting worked again.

actually i run the recommended "emerge --ask --emptytree @world", hope after this is everything done for the successfull profile change.

because I was very interested in it, i tried to figure out what caused my problem.
in baselayout-2.14-r2.ebuild you can read from line 198-204

Code: Select all

if ! use split-usr && ! use prefix-guest; then
		sed \
			-e 's|:/usr/sbin:|:|g' \
			-e 's|:/sbin:|:|g' \
			-e 's|:/bin:|:|g' \
			-i etc/env.d/50baselayout || die
fi

do you know, if it was a bad coincidence and an emerge of sys-apps/baselayout-2.14-r2 would have broken my system anyway or did my "emerge -avuDN --with-bdeps y world" instead of the recommended "emerge --ask --emptytree @world" after changing my profile from 17.1 to 23.0 broke my system?

thank you all!

[pietinger]

[...] do you know, if it was a bad coincidence and an emerge of sys-apps/baselayout-2.14-r2 would have broken my system anyway or did my "emerge -avuDN --with-bdeps y world" instead of the recommended "emerge --ask --emptytree @world" after changing my profile from 17.1 to 23.0 broke my system?

I think none of both ... the reason of your problem was this:

yesterday i migrated from "default/linux/amd64/17.1" to "default/linux/amd64/23.0"

Coming from an OpenRC-split-usr profile needs switching to default/linux/amd64/23.0/split-usr ... you have done the migration to 23.0 AND the migration from split-usr to merged-usr together (=not good) ...

( See more here: https://wiki.gentoo.org/wiki/Merge-usr )

[GDH-gentoo]

yesterday i migrated from "default/linux/amd64/17.1" to "default/linux/amd64/23.0"

You should have chosen default/linux/amd64/23.0/split-usr instead. Your original profile was a split /usr one. The 23.0 ones that don't have "split-usr" in the name have merged /usr by default.

That's explicitly mentioned in step #6 of the migration instructions.

[figueroa]

I suspect that you should have migrated to the split-usr profile:

Code: Select all

[45]  default/linux/amd64/23.0/split-usr (stable)

If so, I think you will continue to have trouble until such time as you either migrate to non-split-usr layout or migrate to the correct profile.

ADDED: That was either easy, or great minds (to which I make no claim) think alike.