[split] Perl upgrade, no updates for 1 year 250 days

[segmentation-fault]

Split from [topic=1063128]Blockers or slot conflicts on a Perl update?[/topic] --pjp

Let me give you a fresh (2022) look on this, from a "user perspective":

Some CAUTION is needed here: I am going to post the part of my notes that is related to Perl upgrade on a system that had not been updated for 1 year and 250 days. Someone above said "my mistake". Let me make it clear that it is a fundamental user right to have absolute control over the who, when and why of a system update! The user should dictate when and how often the updates take place, NOT the other way round! This should NEVER be negotiable - if it is, then it is a sign that something has gone wrong. But this is another story.

Regarding CAUTION: I am not going to redact my notes, in order to give you my unfiltered strong feelings about how things should work. Now, my notes do contain rants. But please don't think that this post is a rant, OK? It is not.

So, are you ready? There you go!

###########################################

Now, to make some more progress, I tried Perl:

First upgrade perl-cleaner:

Code: Select all

emerge -1av app-admin/perl-cleaner

then run it:

Code: Select all

perl-cleaner --all

This does:

Code: Select all

 * Removing perl-core packages from world file
 *    emerge --deselect  perl-core/File-Temp 
>>> No matching atoms found in "world" favorites file...
 * Updating installed Perl virtuals
 *    emerge -u1  virtual/perl-Archive-Tar virtual/perl-Attribute-Handlers virtual/perl-AutoLoader virtual/perl-CPAN-Meta virtual/perl-CPAN-Meta-Requirements virtual/perl-CPAN-Meta-YAML virtual/perl-Carp virtual/perl-Compress-Raw-Bzip2 virtual/perl-Compress-Raw-Zlib virtual/perl-DB_File virtual/perl-Data-Dumper virtual/perl-Devel-PPPort virtual/perl-Digest virtual/perl-Digest-MD5 virtual/perl-Digest-SHA virtual/perl-Encode virtual/perl-Exporter virtual/perl-ExtUtils-CBuilder virtual/perl-ExtUtils-Constant virtual/perl-ExtUtils-Install virtual/perl-ExtUtils-MakeMaker virtual/perl-ExtUtils-Manifest virtual/perl-ExtUtils-ParseXS virtual/perl-File-Path virtual/perl-File-Spec virtual/perl-File-Temp virtual/perl-Getopt-Long virtual/perl-IO virtual/perl-IO-Compress virtual/perl-IO-Socket-IP virtual/perl-IO-Zlib virtual/perl-IPC-Cmd virtual/perl-JSON-PP virtual/perl-Locale-Maketext virtual/perl-Locale-Maketext-Simple virtual/perl-MIME-Base64 virtual/perl-Math-BigInt virtual/perl-Math-BigInt-FastCalc virtual/perl-Math-BigRat virtual/perl-Math-Complex virtual/perl-Module-CoreList virtual/perl-Module-Load virtual/perl-Module-Load-Conditional virtual/perl-Module-Loaded virtual/perl-Module-Metadata virtual/perl-Params-Check virtual/perl-Parse-CPAN-Meta virtual/perl-Perl-OSType virtual/perl-Pod-Escapes virtual/perl-Pod-Parser virtual/perl-Scalar-List-Utils virtual/perl-Socket virtual/perl-Storable virtual/perl-Sys-Syslog virtual/perl-Term-ANSIColor virtual/perl-Test-Harness virtual/perl-Test-Simple virtual/perl-Text-Balanced virtual/perl-Text-ParseWords virtual/perl-Text-Tabs+Wrap virtual/perl-Time-HiRes virtual/perl-Time-Local virtual/perl-Time-Piece virtual/perl-XSLoader virtual/perl-autodie virtual/perl-bignum virtual/perl-if virtual/perl-libnet virtual/perl-parent virtual/perl-podlators virtual/perl-version 

...

emerge -v1 --backtrack=200 --selective=n app-office/gnumeric:0 net-analyzer/net-snmp:0 www-servers/nginx-unit:0 dev-db/postgresql:11 dev-db/postgresql:9.5 dev-db/postgresql:12 dev-db/postgresql:10 media-gfx/imagemagick:0 media-gfx/graphviz:0 dev-tcltk/tclperl:0 net-nds/openldap:0 www-apache/mod_perl:1 app-editors/gvim:0 app-editors/vim:0

but then perl-cleaner stopped due to:

Code: Select all

emerge: there are no ebuilds to satisfy "dev-db/postgresql:9.5".
 * perl-cleaner is stopping here:
 * Fix the problem and start perl-cleaner again.
 * 
 * 
 * Note that upgrading Perl with emerge option --ignore-built-slot-operator-deps=y is not supported.

So let me sum it up:

Because postgresql-9.5 uses Perl somewhere, I *have* to rebuild it!
But postgresql-9.5 is out of the Portage tree and even if I had the
ebuild, it would probably not work because it would use some old
EAPI or old eclasses, or both. Now what? Am I forced to either

- not upgrade Perl as long as I use postgresql-9.5 (HOW IDIOTIC IS THAT?)
- remove postgresql-9.5 just because I upgraded Perl (HOW EVEN MORE IDIOTIC IS THAT?)

How can a distant program (Perl) decide over the fate of another
distant program (Postgresql) and vice versa? HOW IDIOTIC IS THAT?

And all this, only because either postgresql had the 'perl' USE flag,
or (who knows) perl the 'postgresql' flag - give me a break!

So, to upgrade Perl, I need

- to uninstall postgreql:9.5
- to upgrade openssl, which needs to upgrade
- all packages with the ssl USE flag too
- to upgrade all virtuals in the above list of perl-cleaner
- to also upgrade all programs in the extra list of perl-cleaner
- ...anything else?

Therefore, I did

Code: Select all

emerge -av --depclean postgresql:9.5

which went well without problems. But trying the list from perl-cleaner
(notice that I took away postgresql:9.5):

Code: Select all

emerge -av1 --backtrack=200 --selective=n app-office/gnumeric:0 net-analyzer/net-snmp:0 www-servers/nginx-unit:0 dev-db/postgresql:11 dev-db/postgresql:12 dev-db/postgresql:10 media-gfx/imagemagick:0 media-gfx/graphviz:0 dev-tcltk/tclperl:0 net-nds/openldap:0 www-apache/mod_perl:1 app-editors/gvim:0 app-editors/vim:0

could not proceed due to vim asking for an upgrade of ruby,
just because vim has a 'ruby' USE flag:

Code: Select all

!!! All ebuilds that could satisfy "virtual/rubygems" have been masked.
!!! One of the following masked packages is required to complete your request:
- virtual/rubygems-16::gentoo (masked by: backtracking: slot conflict)
- virtual/rubygems-15::gentoo (masked by: backtracking: slot conflict)

(dependency required by "app-editors/gvim-8.2.3950::gentoo[ruby]" [ebuild])
(dependency required by "app-editors/gvim:0" [argument])

So to upgrade Perl, I have to upgrade Ruby first
- and, of course, all packages that have the ruby flag set.

I thus set on to upgrade ruby...

###########################################

Do you get the idea? That's upgrade hell! But it only started!

I will spare you the details of upgrading Ruby and ruby-related packages, since they would require me to post the next ~600 lines of my notes - and they are off-topic here. Let me only say that it cost me a whole day to sort it out. Here is the end of my notes regarding ruby:

###########################################

...[600 lines later:]

Code: Select all

emerge -1av dev-ruby/rack dev-ruby/rdoc dev-ruby/json dev-ruby/xmlrpc dev-ruby/test-unit dev-ruby/power_assert dev-ruby/rake dev-ruby/net-telnet dev-ruby/minitest dev-ruby/rails dev-ruby/jquery-rails dev-ruby/railties dev-ruby/actionpack:5.2 dev-ruby/actionpack:6.1 dev-ruby/rake-compiler

This offerted me 61 new, shiny packages to merge!
Of course I accepted the offer - and half(!) and hour later the battle was over!

Now, think a moment about this: a day's struggle for for an hour's battle!
Not even a whole hour's! This CANNOT be accepted as "normal"! This is NOT fun!

###########################################

Of course it's not fun - it's upgrade hell!

Let me now resume from the point where Ruby and its entourage got updated and it was Perl's turn again:

###########################################

NEXT: Perl (again!)

Code: Select all

emerge -1av $(equery list 'dev-perl/*' | sed -e 's/.*dev/dev/; s/-[0-9][0-9]*.*$//') $(equery list 'virtual/perl*' | sed -e 's/-[0-9][0-9]*.*$//')

This *almost worked* - and it would give me 500+ packages at once!
But it failed due to slot conflicts with installed packages that
required the same slot of Perl that there was at the time
*they* were built. This seemed a great opportunity to test my
understanding of the portage workings and add the
--ignore-built-slot-operator-deps=y option. ALAS, I read in various
messages regarding Perl upgrade:


* Note that upgrading Perl with the merge option
* --ignore-built-slot-operator-deps=y is not supported.


So I had to bite the bullet (again!) and make a package list
out of the long list that 'emerge' printed at the end:

Code: Select all

!!! The slot conflict(s) shown above involve package(s) which may need to
!!! be rebuilt in order to solve the conflict(s). However, the following
!!! package(s) cannot be rebuilt for the reason(s) shown:

  (net-print/cups-filters-1.27.4:0/0::gentoo, installed): ebuild is masked or unavailable
  (net-analyzer/net-snmp-5.8-r5:0/35::gentoo, installed): ebuild is masked or unavailable
  (sci-libs/gdal-3.0.4-r1:0/3.0::gentoo, installed): ebuild is masked or unavailable
  (app-office/gnumeric-1.12.46:0/0::gentoo, installed): ebuild is masked or unavailable
  (www-servers/nginx-unit-1.17.0:0/0::gentoo, installed): ebuild is masked or unavailable
  (net-nds/openldap-2.4.50:0/0::gentoo, installed): ebuild is masked or unavailable
  (net-analyzer/nagios-core-4.4.5-r6:0/0::gentoo, installed): ebuild is masked or unavailable
  (dev-perl/gnome2-perl-1.46.0:0/0::gentoo, installed): matched by --exclude argument
  (app-text/po4a-0.47-r1:0/0::gentoo, installed): ebuild is masked or unavailable
  (sys-libs/libapparmor-2.13.4:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-gfx/graphviz-2.42.3:0/0::gentoo, installed): ebuild is masked or unavailable
  (dev-vcs/subversion-1.12.2:0/0::gentoo, installed): ebuild is masked or unavailable
  (dev-python/subunit-1.2.0-r1:0/0::gentoo, installed): ebuild is masked or unavailable
  (net-fs/samba-4.11.6-r2:0/0::gentoo, installed): ebuild is masked or unavailable
  (mail-filter/spamassassin-3.4.4:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-gfx/imagemagick-7.0.10.7-r1:0/7.0.10::gentoo, installed): ebuild is masked or unavailable
  (app-backup/amanda-3.5.1-r1:0/0::gentoo, installed): ebuild is masked or unavailable
  (net-analyzer/rrdtool-1.6.0-r1:0/8.0.0::gentoo, installed): ebuild is masked or unavailable
  (media-gfx/graphicsmagick-1.3.35:0/1.3::gentoo, installed): ebuild is masked or unavailable
  (dev-vcs/git-2.26.2:0/0::gentoo, installed): ebuild is masked or unavailable
  (virtual/perl-Pod-Parser-1.630.0-r6:0/0::gentoo, installed): matched by --exclude argument
  (app-arch/rpm-4.14.1:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-libs/exiftool-11.93:0/0::gentoo, installed): ebuild is masked or unavailable
  (app-editors/gvim-8.2.0360:0/0::gentoo, installed): ebuild is masked or unavailable
  (dev-perl/PortageXS-0.02.10-r4:0/0::gentoo, installed): matched by --exclude argument
  (sys-process/parallel-20191022:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-gfx/graphite2-1.3.13:0/0::gentoo, installed): ebuild is masked or unavailable
  (dev-perl/gnome2-vfs-perl-1.83.0:0/0::gentoo, installed): matched by --exclude argument
  (app-editors/vim-8.2.0360:0/0::gentoo, installed): ebuild is masked or unavailable
  (dev-db/postgresql-12.2:12/12::gentoo, installed): ebuild is masked or unavailable

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
NOTE: How to make a list of packages from the list of packages that
"may need to be rebuilt in order to solve the conflict" printed by
'emerge' at the end in blue color:

Copy the list into vi, then do:

Code: Select all

1,$s/^  *(\(.*\)-[0-9][0-9]*.*/\1/

and join them, if you like, into one line by repeatedly pressing 'J'.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

So I tried

Code: Select all

emerge -1av --exclude 'virtual/perl-Pod-Parser dev-perl/PortageXS dev-perl/gnome2-perl dev-perl/gnome2-vfs-perl'   $(equery list 'dev-perl/*' | sed -e 's/.*dev/dev/; s/-[0-9][0-9]*.*$//') $(equery list 'virtual/perl*' | sed -e 's/.*dev/dev/; s/-[0-9][0-9]*.*$//') net-print/cups-filters net-analyzer/net-snmp sci-libs/gdal app-office/gnumeric www-servers/nginx-unit net-nds/openldap net-analyzer/nagios-core dev-perl/gnome2-perl app-text/po4a sys-libs/libapparmor media-gfx/graphviz dev-vcs/subversion dev-python/subunit net-fs/samba mail-filter/spamassassin media-gfx/imagemagick app-backup/amanda net-analyzer/rrdtool media-gfx/graphicsmagick dev-vcs/git virtual/perl-Pod-Parser app-arch/rpm media-libs/exiftool app-editors/gvim dev-perl/PortageXS sys-process/parallel media-gfx/graphite2 dev-perl/gnome2-vfs-perl app-editors/vim dev-db/postgresql

Well, almost...The following two had to go, as there were no ebuilds
AND they NEEDED to be rebuilt with the newer Perl:

Code: Select all

emerge -av --depclean dev-perl/gnome2-perl dev-perl/gnome2-vfs-perl

emerge -av --depclean virtual/perl-Pod-Parser dev-perl/PortageXS

...but the last one did NOT work - there were packages that needed them...
...but what to do if there are no ebuilds AND they must be rebuilt?

Whatever...I did:

Code: Select all

emerge -av --depclean app-portage/perl-info
emerge -av --depclean dev-perl/PortageXS

The only one that now remained was virtual/perl-Pod-Parser,
which I *had* to hard-remove, because, even though all
packages that depended on it still had ebuilds, it itself
did not - so what else can one do, other than

Code: Select all

emerge -C virtual/perl-Pod-Parser

???

O.K., before continuing, I thought I would clean up the LUA MESS
- that's because a few lua packages stood in the way for the Perl upgrade:

Code: Select all

emerge -1av dev-lang/lua:5.1::gentoo dev-lang/lua:5.4::gentoo media-video/vlc dev-lua/lua-zlib app-admin/conky dev-lua/lpeg media-video/mpv app-text/podofo x11-misc/devilspie2 app-editors/vim app-misc/gpick dev-lua/luajson media-libs/libquvi dev-lua/LuaBitOp dev-lua/luaexpat dev-lua/luasocket

...but lua pulls in the whole QT SHIT...So I *HAD* TO USE --nodeps,
otherwise this would NEVER END:

Code: Select all

emerge -1av --nodeps dev-lang/lua:5.1::gentoo dev-lang/lua:5.4::gentoo media-video/vlc dev-lua/lua-zlib app-admin/conky dev-lua/lpeg media-video/mpv app-text/podofo x11-misc/devilspie2 app-editors/vim app-misc/gpick dev-lua/luajson media-libs/libquvi dev-lua/LuaBitOp dev-lua/luaexpat dev-lua/luasocket

BACK TO PERL HELL:

Code: Select all

emerge -1av --exclude 'virtual/perl-Pod-Parser dev-perl/PortageXS dev-perl/gnome2-perl dev-perl/gnome2-vfs-perl'   $(equery list 'dev-perl/*' | sed -e 's/.*dev/dev/; s/-[0-9][0-9]*.*$//') $(equery list 'virtual/perl*' | sed -e 's/.*dev/dev/; s/-[0-9][0-9]*.*$//') net-print/cups-filters net-analyzer/net-snmp sci-libs/gdal app-office/gnumeric www-servers/nginx-unit net-nds/openldap net-analyzer/nagios-core app-text/po4a sys-libs/libapparmor media-gfx/graphviz dev-vcs/subversion dev-python/subunit net-fs/samba mail-filter/spamassassin media-gfx/imagemagick app-backup/amanda net-analyzer/rrdtool media-gfx/graphicsmagick dev-vcs/git app-arch/rpm media-libs/exiftool app-editors/gvim sys-process/parallel media-gfx/graphite2 app-editors/vim dev-db/postgresql:10 dev-db/postgresql:11 dev-db/postgresql:12 dev-db/postgresql:13 app-vim/vim-latex net-analyzer/nagios games-arcade/frozen-bubble dev-db/percona-toolkit www-apache/mod_perl

Tataaaaaa! IT WORKED RIGHT FROM THE START!!!

Total: 578 packages (507 upgrades, 55 new, 2 in new slots, 14 reinstalls), Size of downloads: 376,745 KiB
Conflict: 3 blocks (all satisfied)

Of course, I accepted this great offer...
...and went to sleep, letting it work for me all night.

###########################################

That was another day - I am currently on day 6 and still have not managed to get not even a list with blockers from portage when I try to upgrade @world.

I cannot resist, I will say it:

Gentoo developers, maintainers, users and respective wannabes: unite! We have to make Gentoo fun again!

[Hu]

I am going to post the part of my notes that is related to Perl upgrade on a system that had not been updated for 1 year and 250 days. Someone above said "my mistake". Let me make it clear that it is a fundamental user right to have absolute control over the who, when and why of a system update! The user should dictate when and how often the updates take place, NOT the other way round! This should NEVER be negotiable - if it is, then it is a sign that something has gone wrong. But this is another story.

Yes, users have the right not to update their systems. Developers have the right to assume users will upgrade according to published best practices, and not worry about whether a system that is well outside best practice will upgrade smoothly on its own. When you fall that far behind, you get to pick up the pieces. For users who are nice about it, there is often forum help from experienced users to ease the process.

but then perl-cleaner stopped due to:
Because postgresql-9.5 uses Perl somewhere, I *have* to rebuild it!
But postgresql-9.5 is out of the Portage tree and even if I had the
ebuild, it would probably not work because it would use some old
EAPI or old eclasses, or both. Now what? Am I forced to either

- not upgrade Perl as long as I use postgresql-9.5 (HOW IDIOTIC IS THAT?)

Not very. You don't want to upgrade postgresql, so you can't upgrade the things it depends on. What did you expect would happen? Did you expect that every version of postgres would be runtime-compatible with every version of perl, forever? That's the only way to avoid this rebuild.

- remove postgresql-9.5 just because I upgraded Perl (HOW EVEN MORE IDIOTIC IS THAT?)

Considering postgres-9.5 is so old it isn't even in the main tree anymore, also not very. You might have avoided this if you didn't install the Perl bindings for postgres.

How can a distant program (Perl) decide over the fate of another
distant program (Postgresql) and vice versa? HOW IDIOTIC IS THAT?

For the third time, not very. Portage is trying very hard to avoid having any installed files become broken as a result of upgrades. In this case, since upgrading perl is expected to break the perl bindings of the installed postgres, you need to rebuild postgres so that its perl bindings can be rebuilt to work with the new perl.

And all this, only because either postgresql had the 'perl' USE flag,
or (who knows) perl the 'postgresql' flag - give me a break!

Perl has no postgres USE flag. Postgres has a USE=perl flag, to enable the optional perl bindings, which you chose to have enabled.

So, to upgrade Perl, I need

- to uninstall postgreql:9.5

Or rebuild it in place, yes. It would probably work, if you had an ebuild for that version. Perl breaks are usually about path changes or simple ABI breaks, and an ebuild reinstall is usually sufficient.

- to upgrade openssl, which needs to upgrade

On a system this out of date, I would just remove openssl and break all the SSL using programs. It would be safer. Though, if you're willing to go that far, it might be easier just to unplug the network cable.

- all packages with the ssl USE flag too

That is not necessarily right. USE=ssl enables SSL bindings, but not necessarily through dev-libs/openssl. That aside, this would only be true if openssl has undergone an ABI change since the last time you installed it. Those are fairly rare, but not unheard of.

- to upgrade all virtuals in the above list of perl-cleaner

Since you are so unhappy, I will let you in on a dirty little secret: virtual packages don't install any files. Their dependencies are only there to keep you out of trouble. Since you're quite determined to get into trouble, you can probably blow past all the Portage safety mechanisms.

- to also upgrade all programs in the extra list of perl-cleaner

Do you think perl-cleaner is wrong to flag these programs as dependent on the installed perl version? If yes, please describe how. If no, then why would you not want to rebuild them? If perl-cleaner is correct, and you choose not to rebuild them, they will be broken. If they are broken, you may as well remove them, which would be easier than rebuilding them, and no less disruptive than letting them break.

Code: Select all

emerge -av1 --backtrack=200 --selective=n app-office/gnumeric:0 net-analyzer/net-snmp:0 www-servers/nginx-unit:0 dev-db/postgresql:11 dev-db/postgresql:12 dev-db/postgresql:10 media-gfx/imagemagick:0 media-gfx/graphviz:0 dev-tcltk/tclperl:0 net-nds/openldap:0 www-apache/mod_perl:1 app-editors/gvim:0 app-editors/vim:0

could not proceed due to vim asking for an upgrade of ruby,
just because vim has a 'ruby' USE flag:

Do you even use Vim's ruby bindings? USE=ruby is not a default-enabled flag for Vim. If you don't need its Ruby bindings, you should turn them back off.

Code: Select all

!!! All ebuilds that could satisfy "virtual/rubygems" have been masked.
!!! One of the following masked packages is required to complete your request:
- virtual/rubygems-16::gentoo (masked by: backtracking: slot conflict)
- virtual/rubygems-15::gentoo (masked by: backtracking: slot conflict)

(dependency required by "app-editors/gvim-8.2.3950::gentoo[ruby]" [ebuild])
(dependency required by "app-editors/gvim:0" [argument])

I will grant that one is hard to understand without further context. Was there really no other error output about this?

So to upgrade Perl, I have to upgrade Ruby first

You could also choose to remove Vim from the system, or to rebuild Vim without Ruby bindings.

Do you get the idea? That's upgrade hell! But it only started!

Sure. This is the consequence of leaving a complex system without upgrades for too long. We see these every so often on the forums, and the people who are nice about their problems are given a detailed walkthrough on how to fix it.

Here is the end of my notes regarding ruby:

"Add all things Ruby to package.mask and package.use.mask"?

Not even a whole hour's! This CANNOT be accepted as "normal"! This is NOT fun!

This is not normal. Normal maintenance would have given Portage a much simpler problem to solve, and it would have solved it on its own. You waited so long that you presented it an impossible set of requirements, so it gave up and required you to simplify the constraint set. That took a while for you to solve.

Code: Select all

emerge -1av $(equery list 'dev-perl/*' | sed -e 's/.*dev/dev/; s/-[0-9][0-9]*.*$//') $(equery list 'virtual/perl*' | sed -e 's/-[0-9][0-9]*.*$//')

I probably would have used some variant of eix --only-names -I --options here. It's much easier than using sed to clean up package names later.

This *almost worked* - and it would give me 500+ packages at once!
But it failed due to slot conflicts with installed packages that
required the same slot of Perl that there was at the time
*they* were built.

This was probably because you did not tell Portage to rebuild those packages, so it panicked at the conflicting requirements of: (1) don't rebuild packages with a subslot dependency on perl, (2) don't break packages by changing the active perl subslot, and (3) do upgrade perl. You can have any 2 of those, but not all 3.

* Note that upgrading Perl with the merge option
* --ignore-built-slot-operator-deps=y is not supported.

I believe this is unsupported in the sense of "If you do this, and it doesn't produce the results you wanted, we don't want to hear about it. We know it may fail. You're on your own if you try this." not in the sense of "This is guaranteed to fail and no one should ever try it."

and join them, if you like, into one line by repeatedly pressing 'J'.

:%j would be faster.

...but the last one did NOT work - there were packages that needed them...
...but what to do if there are no ebuilds AND they must be rebuilt?

There should be no packages in the tree, for which ebuilds exist, which depend on ebuilds which did not exist. If your depclean failed because a consuming package existed, then it is likely that the consuming package itself was not in the Portage tree and should have been added to the --depclean list.

The only one that now remained was virtual/perl-Pod-Parser,
which I *had* to hard-remove, because, even though all
packages that depended on it still had ebuilds, it itself
did not

There should be no ebuilds which still depend on this. If there are, then they need to be upgraded.

- so what else can one do, other than

Code: Select all

emerge -C virtual/perl-Pod-Parser

???

It is just a virtual, so that should be fine.

O.K., before continuing, I thought I would clean up the LUA MESS

How many different optional interpreters did you have? Maybe some PHP and Fortran for good measure?

...but lua pulls in the whole QT

Citation needed.

Code: Select all

$ grep -e -qt -r dev-lua/
$ 

So I *HAD* TO USE --nodeps,

This is a good way to make a bad situation worse.

Gentoo developers, maintainers, users and respective wannabes: unite! We have to make Gentoo fun again!

Indeed. Writing this line by line critique of your rant was less fun than I anticipated. Next time, if you make a mess, please rant less in your notes.

[segmentation-fault]

My dear Hu, where do I start now...

You made a very thorough effort and many thanks for this. I warned everybody from the start to not interpret my post as a rant, even if there are rants in my notes - I hope you did not get offended. If you did, please accept my apologies. As I said, I wanted to give a fresh, unfiltered view of my upgrade hell, not to rant and "not be nice". I have no reason not to be nice, because I am sure that everybody does his/her best here. However, we can do better - that's my message.

I have the feeling that we are talking past each other. So I will try to answer top-down, from general views to specifics. This is because we approach portage with fundamentally different expectations - and I guess that's where the problems start.

You say it is "best practice" to upgrade often. I say, no, no way, never! It's even worse: when I see that it takes me 2 full man-months to upgrade, next time I even think about an upgrade, I say to myself "no, not now!". But, am I "responsible" for the mess that I have when I do try to upgrade? You say yes, because I created such a complicated dependency graph, which is "impossible to resolve". I say no, because I expect my package manager to work even in that scenario!

How?

Here's how: I know that the perl package does not have a 'postgresql' flag - but what if it had one? What would portage do then? In my case, it would think "oh I have circular dependencies, I cannot do anything here!". My point is: when I say "upgrade perl", I don't care (and portage should not care either) if some installed package will break because it has the 'perl' flag set and needs to be rebuilt. It should upgrade perl, making sure that it pulls-in the direct dependencies of perl - and that's all, thank you.

At the very least, there should be an option (say, --ignore-use-flag-induced-rebuilts) that would make it ignore all packages that have the 'perl' use flag when it upgrades perl. It should print a warning that now a second emerge run is needed, this time with "--deep" and "--newuse", to ensure that packages who use perl will not be broken in their Perl functionality - and that's all. The rest is in the responsibility of the user.

Why?

Because, when I am upgrading, I am upgrading and don't work - at least not with the functionality I am currently upgrading. To stay on the example, when I upgrade Perl, I will not be using the Perl bindings of my PostgreSQL and when I upgrade Ruby, I will not be asking my vim to use Ruby at exactly that moment. I know I am upgrading. I will take care of PostgreSQL and vim and their bindings in a second and third step. All will be good again - and if it's not, let that be my problem. But refusing to upgrade Perl, because some remote, distant, irrelevant program has the 'perl' flag set is, well...counterproductive.

Trying to enforce a consistent state across the whole dependency graph, and fail hard if this is somehow harder than with the miniscule graphs you get when you update daily - that' s a recipe for disaster. Saying to the user "your fault that you have such a big graph, you are alone now" is outright user-hostile.

Give us the tools and we will finish the job!

We need a way to "upgrade perl and its modules only", "ruby and its modules only", "Qt only", without the packages that have a 'perl', 'ruby' or (possibly) 'qt' USE flag. If it is difficult to program a --ignore-use-flag-induced-rebuilts option that could calculate the list of the packages-to-be-excluded, then may be it is easier to copy the functionality of the --exclude option and make a --exclude-packages-with use-flag-set 'some user-provided list of USE flags, separated by space' option. Instead of having to disable the 'perl' use flag in all packages that have it set, upgrade Perl, then reset the flag in all those packages back, then re-rerun emerge with --newuse, I would just run 'emerge ... --exclude-packages-with use-flag-set "perl"', then - at some later point of my choice - I would run 'emerge ... --newuse ...'.

That's power to the user, responsibility to the user, tools for the user - all I ask for. Not indoctrinating him about supposedly "best" practices that have been defined to retroactively fit existing inadequacies of a stiff status quo.

[segmentation-fault]

Let me give some answers to "specifics" here...As I said, you have done a very thorough effort, dear Hu, and it's not going to be easy for me - but I'll try. As always, please remember: my purpose in opening such a Pandorra box is to make portage better - and ultimately Make Gentoo FUN Again (TM)! - no rants, I promise!

So let's see:

What did you expect would happen? Did you expect that every version of postgres would be runtime-compatible with every version of perl, forever?

By now it should be clear from my above general thoughts what I expect: I expect portage to exclude packages with the 'perl' flag set, if I ask it so (possibly through some extra option crafted for this purpose). And I am willing to take responsibility for the (presumably very restricted) breakage this might bring upon those excluded packages. I also expect to be able to remedy the situation with a subsequent "emerge ... --newuse ..." at a later time point of my choice.

This makes a "complicated" (you might call it "unsolvable") upgrade from an "all-or-nothing proposition" to a multi-step process. From one-step-that-MUST-succeed-or-else... to a more-small-steps-that-resolve-a-small-part-each-time procedure - the UNIX way, if you like! For this, we need the tools (some option here and there that will "cut the tree" to a manageable size, according to user's wishes).

Portage is trying very hard to avoid having any installed files become broken as a result of upgrades.

Exactly. And that's the problem. Even with smaller dependency graphs (my world file has more than 1700 packages and there are more than 4000 installed all together), taking USE flags into account always and by all means will make portage very inadequate, even if it is actually very potent. We must break loose with this tradition! At least with some extra option, when the user explicitly asks for it.

...but lua pulls in the whole QT

Citation needed.

I retried

Code: Select all

emerge -1av dev-lang/lua:5.1::gentoo dev-lang/lua:5.4::gentoo media-video/vlc dev-lua/lua-zlib app-admin/conky dev-lua/lpeg media-video/mpv app-text/podofo x11-misc/devilspie2 app-editors/vim app-misc/gpick dev-lua/luajson media-libs/libquvi dev-lua/LuaBitOp dev-lua/luaexpat dev-lua/luasocket

and, among a few blockers, it also tells me that

Code: Select all

!!! The slot conflict(s) shown above involve package(s) which may need to
!!! be rebuilt in order to solve the conflict(s). However, the following
!!! package(s) cannot be rebuilt for the reason(s) shown:

  (kde-apps/k3b-19.12.3:5/5::gentoo, installed): ebuild is masked or unavailable
  (media-video/mplayer-1.3.0-r6:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-video/transcode-1.1.7-r4:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-video/handbrake-1.3.0-r2:0/0::gentoo, installed): ebuild is masked or unavailable
  (media-libs/libmatroska-1.5.2:0/6::gentoo, installed): ebuild is masked or unavailable
  (media-video/mkvtoolnix-37.0.0:0/0::gentoo, installed): ebuild is masked or unavailable

and kde-apps/k3b is surely going to pull-in the whole Qt package set. Now, why is kde-apps/k3b pulled-in? As it turns out, kde-apps/k3b needs media-libs/libdvdread and media-libs/libdvdread must be rebuilt/upgraded because it is needed by media-video/vlc, which in turn is being pulled-in because...you guessed it! ...it has the 'lua' USE flag set!

Now, you will say "take that 'lua' flag off all packages that have it set, upgrade lua, put it back in and run emerge with --newuse". To which I answer: this is exactly what I meant by "retrofitting supposedly 'best' practices". It's not "best" practice having to put USE flags back-and-forth! It is, however, acceptable, from a user perspective, to be able to tell portage to (temporarily) exclude packages with a certain USE flag from the upgrade procedure.

[segmentation-fault]

More "specifics":

- to upgrade all virtuals in the above list of perl-cleaner

Since you are so unhappy, I will let you in on a dirty little secret: virtual packages don't install any files. Their dependencies are only there to keep you out of trouble. Since you're quite determined to get into trouble, you can probably blow past all the Portage safety mechanisms.

Well, thank you for sharing little dirty secrets with me , but what I was trying to do was replicate perl-cleaner manually. perl-cleaner takes quite some time to create its lists and there was no point in recalculating them. Since perl-cleaner told me it was going to upgrade those virtuals, I thought I could (and should) do the same manually. At no point did I try to undermine the way perl-cleaner would do it - only to mimick it. And remember: perl-cleaner would choke on programs that have the 'perl' flag set and no ebuild, so I had to somehow imitate its operation manually, taking care to exclude those packages - very logical IMO and by no means "determined to get into trouble"! The least thing one needs in such a situation is extra trouble!

But I see no other option - other than the "best" practice to squeeze my system to fit the needs of portage, that is.

[pietinger]

segmentation-fault,

I am using Gentoo nearly the past 20 years. I love Gentoo. Therefore I love everyone who loves it too and want to make it better

I am an old and paranoid (for IT security) man (and no native english speaker). I want to tell you some of my beliefs - also from a "user perspective" (= please forget my moderator status). Maybe you will think its great nonsense, but maybe you think about ...

You say it is "best practice" to upgrade often. I say, no, no way, never! It's even worse [...]

I love liberty in every way, but in a human society my liberty ends where it would harm the liberty of others. I have the freedom to not repair the lights of my car if I dont want. But I am not allowed driving my car in the night on a highway without lights. It would be to risky for others ...

If you now think, you will not harm others with an old computer system, I will agree with you ... IF ... you are not connected to the Internet. If your system have been hacked (because of old software) and sends now spam e-mails to me (as part of an evil botnet) I would say "thank you very much" to you ...

From a security point of view it is really "best practice" to upgrade as often as possible (of course it does not prevent against "new" bugs, but I think you know what I mean).

Now to a technical point ov view: You cannot have a racing Formula-1 car for transporting the same amount of things a truck can do ...

I think you know the differences between distributions with fixed versions and a distribution doing a rolling release. An "update" from a fixed Ubuntu version to the next one is almost an "invisible" "new installation" (all packages are removed by the new one). All we do here in Gentoo has been done also by the developers of Ubuntu. So, yes a rollings release has its advantages and disadvantages. One disadvantage is the complexity of software depending from other software (you know this already). Now you can go two ways:

1. Make it as easy as possible to update, WHEN users do all updates absolutely regular, or
2. Give full control over the update process with the risk users have to learn all about all used software ... (I suggest you cannot drive a formula-1 car ... )

My point of view is that our Gentoo developers try hard to make a comprimise between these. Look to "--nodeps" and "--keep-going" (I really hate both - you are able to shoot yourself in the knee with it). Now, when I combine (its only my point of view) the "best practice" with the possibilities we have, I would vote for option 1 (dont forget: even the linux kernel cannot be compiled with a too old gcc).

I love your approach to make Gentoo better ... but maybe you want something impossible (for a rolling distro).

Many greetings,
Peter

[pjp]

Split from the original thread. Not updating for nearly 2 years puts this into a category unrelated to normal scenarios.

Put in Gentoo Chat as it appears resolved. In addition, if it looks like a duck, walks like a duck, and quacks like a duck, its a duck (if you rant in your notes, then the post reads like a rant). Ranting isn't inherently a problem, but it can easily become inappropriate in support threads.

Actively making a choice to not update in such a long period of time does not warrant such ranting about tooling or how long support for updates should be provided.

Constructive discussion about how support could be provided for longer periods of time on old software is fine. The problem is ultimately one of human time required for maintenance and mostly not a technical problem. Although if you have a technical solution, I'm sure many people would love to see a demo.

[Hu]

You say it is "best practice" to upgrade often. I say, no, no way, never!

I will elaborate: recommended practice for maintaining a Gentoo system with minimal hassle is to upgrade often. The Gentoo developers adopted this position partly for their own convenience (because keeping the tree in a state that you can wake up a 2 year old system and have it all just work is a lot of extra trouble, and relatively few users will benefit from this) and partly as a reflection of some upstream projects that take a similar position.

It's even worse: when I see that it takes me 2 full man-months to upgrade

This is a common complaint / justification from users who have been burned by deferring an upgrade. The upgrade goes very poorly, and costs them a lot of personal time. They then resolve "I'm never doing that again," and proceed to upgrade as little as possible and as rarely as possible. That exacerbates the problems, so when they are finally forced to upgrade (say, because they need a new browser version because of poor web site compatibility), it is again a miserable and drawn out process. If the upgrade process were this hard for everyone every time, I think many users would have quit the distribution a long time ago. Instead, users who upgrade often find that the package manager is usually able to solve everything on its own. (We sometimes see tricky / complicated special events, like the libxcrypt migration. The Gentoo developers are good at recognizing these events and go out of their way to provide special handling to smooth the path.)

But, am I "responsible" for the mess that I have when I do try to upgrade? You say yes, because I created such a complicated dependency graph, which is "impossible to resolve". I say no, because I expect my package manager to work even in that scenario!

You can expect that, and it certainly would be nice if Portage could always solve every requirement. In practice, it doesn't, and at least for some of the scenarios you seem to have, it cannot.

Here's how: I know that the perl package does not have a 'postgresql' flag - but what if it had one? What would portage do then?

That would depend on how the ebuild reacts to USE=postgresql.

In my case, it would think "oh I have circular dependencies, I cannot do anything here!". My point is: when I say "upgrade perl", I don't care (and portage should not care either) if some installed package will break because it has the 'perl' flag set and needs to be rebuilt. It should upgrade perl, making sure that it pulls-in the direct dependencies of perl - and that's all, thank you.

I disagree here. I remember when Portage would happily let you break wget by upgrading libraries that it depended on. After an upgrade, you'd get:

Code: Select all

$ wget --help
wget: error while loading shared libraries: /usr/lib/libopenssl.so.0: No such file or directory

or similar. I prefer that, by default, Portage try very hard not to break any installed packages.

At the very least, there should be an option (say, --ignore-use-flag-induced-rebuilts)

This isn't actually USE flag related. You could have the same problem with a package that unconditionally uses Perl, and therefore has no USE=perl flag. However, I can see the merit of having an option to choose to let Portage break some packages in order to simplify updating others. --ignore-built-slot-operator-deps, which is intended only for debugging purposes may approximate this. --ignore-world, though dangerous, is another possibility for you. Both of these are documented in man emerge.

There is also the sledgehammer approach of disabling all dependency resolution.

Why?

Because, when I am upgrading, I am upgrading and don't work - at least not with the functionality I am currently upgrading.

I see this argument, but I would counter that what you propose would let Portage leave the system in the interim state indefinitely. If for any reason you don't come back and run that later emerge, the affected packages could be broken for weeks. Also, in some cases, the affected packages will be used by background/automated processes, so you might not even notice that it was being used unsuccessfully. We've seen this before when a PAM upgrade causes cron to fail to start any jobs until cron is restarted with the new PAM.

If you really want to break one package while upgrading another, you could emerge --depclean the one you are willing to lose. That will remove it, and remove the need to satisfy its requirements.

To stay on the example, when I upgrade Perl, I will not be using the Perl bindings of my PostgreSQL and when I upgrade Ruby, I will not be asking my vim to use Ruby at exactly that moment.

That's fair, but in the case of Vim, what if the Ruby bindings were load-time linked to Vim? (I don't think they are, but some packages may be done that way.) If they were, then if you break Ruby, you cannot use vim at all, even for basic text editing, until you repair Ruby. Personally, I would be rather annoyed to have my text editor break while I'm upgrading. Thankfully, Vim seems to prefer dynamic loading of many of its bindings, so it works more like you hypothesized - avoiding use of Ruby bindings would let you use the rest of the editor.

I know I am upgrading. I will take care of PostgreSQL and vim and their bindings in a second and third step.

Portage prefers to fix them all in one run. This is less flexible for you as a user, but has the useful property that if the run completes, your system is back in a good state.

All will be good again - and if it's not, let that be my problem. But refusing to upgrade Perl, because some remote, distant, irrelevant program has the 'perl' flag set is, well...counterproductive.

I agree that this is frustrating. The standard answer is that if your dependency graph was not so complicated, then Portage would not have refused. It would have scheduled a needed rebuild of PostgreSQL as part of the run.

Trying to enforce a consistent state across the whole dependency graph, and fail hard if this is somehow harder than with the miniscule graphs you get when you update daily

Daily is not necessary. I do less than daily, and it is fine.

Saying to the user "your fault that you have such a big graph, you are alone now" is outright user-hostile.

The overall goal is that the "obvious" way to use the tool makes it hard to put the system into a broken state. If the user wants to override those safeties and tolerate a system that is partially broken, that is the user's choice. In some cases, that can be a good tradeoff. However, having read the forums for many years now, I appreciate that the obvious path is to refuse to let the user break the system, absent explicit instructions requesting that.

If it is difficult to program a --ignore-use-flag-induced-rebuilts

I think --ignore-world would do what you want. As noted in the manual page, this can make a mess if used carelessly.

For this, we need the tools (some option here and there that will "cut the tree" to a manageable size, according to user's wishes).

That is fair, if it is a package manager option rather than a standard about what stays in the tree versus what is removed.

Portage is trying very hard to avoid having any installed files become broken as a result of upgrades.

Even with smaller dependency graphs (my world file has more than 1700 packages and there are more than 4000 installed all together)

Wow. You have more packages in @world than I have on my entire system.

I retried

Code: Select all

emerge -1av dev-lang/lua:5.1::gentoo dev-lang/lua:5.4::gentoo media-video/vlc dev-lua/lua-zlib app-admin/conky dev-lua/lpeg media-video/mpv app-text/podofo x11-misc/devilspie2 app-editors/vim app-misc/gpick dev-lua/luajson media-libs/libquvi dev-lua/LuaBitOp dev-lua/luaexpat dev-lua/luasocket

and, among a few blockers, it also tells me that

Code: Select all

!!! The slot conflict(s) shown above involve package(s) which may need to
!!! be rebuilt in order to solve the conflict(s). However, the following
!!! package(s) cannot be rebuilt for the reason(s) shown:

  (kde-apps/k3b-19.12.3:5/5::gentoo, installed): ebuild is masked or unavailable

and kde-apps/k3b is surely going to pull-in the whole Qt package set. Now, why is kde-apps/k3b pulled-in? As it turns out, kde-apps/k3b needs media-libs/libdvdread and media-libs/libdvdread must be rebuilt/upgraded because it is needed by media-video/vlc, which in turn is being pulled-in because...you guessed it! ...it has the 'lua' USE flag set!

So this effectively validates my original point. Lua does not impact Qt. I would have preferred getting the Portage blocker output, but I think you are mischaracterizing it. vlc was scheduled for upgrade because you requested it. libdvdread was not requested, but may have been required to support the newer vlc. Once libdvdread was pulled in, then k3b was impacted.

Now, you will say "take that 'lua' flag off all packages that have it set

I would tell you to remove lua and leave it off. You seem to have a huge number of optional features enabled, and it's not clear why you did this, but that makes the whole graph more complicated.

"retrofitting supposedly 'best' practices". It's not "best" practice having to put USE flags back-and-forth!

Right. It's best practice not to create graphs that require user intervention.

[Irre]

Why not temporary remove the world file. Then update your system. Then restore your world file, possibly in small steps.

[NeddySeagoon]

or do some of the intermediate steps that were missed by the neglect.

Like eating an elephant, do it one plateful at a time. :)

[segmentation-fault]

or do some of the intermediate steps that were missed by the neglect.

Like eating an elephant, do it one plateful at a time.

Neddy,

so nice to see you here!

Yes, I have used that one once, when I had not upgraded for...uhmm...6 years? I have experience in eating elephants, you see... My situation today pales in front of the one I was in back in those days...As Hu said, I had managed to destroy my PAM and could not even login...that was fun!

Off-topic: I still use FVWM, with a Motif-look and xeyes (of course!). I also use a GTK3 "CDE" theme in my whole X that brings you back 20 years and more. I still have ALL my distfiles ever installed, starting ~2002 - all interesting stuff for your historical Gentoo project, only that this is a "rocket Gentoo on steroids" (almost up-to-date! working on it... ) with historical looks. P.S. The running cat in the image is from app-misc/oneko.

[segmentation-fault]

If it is difficult to program a --ignore-use-flag-induced-rebuilts

I think --ignore-world would do what you want. As noted in the manual page, this can make a mess if used carelessly.

Hu,

you are the hero of the day! This indeed comes close to what I desperately needed! It is "broader" in its "pruning" than my proposed option (that would prune only USE flag dependencies), but still is a life-saver!

You should offer this option to everybody that is desperately looking for a "sword" to cut the Gordian Knot of his dependency tree.

Thank you!

[figueroa]

Yes, it was a rant. That's OK, as it was interesting. A log of electrons have been used to coax you along. Bless their hearts.

This has been said before -- Gentoo follows upstream and upstream continues to accelerate at an increasing rate.

What used to be OK, updating a few times a year has evolved into a major ordeal. This requires updating more frequently to avoid the ordeal.

My main desktop and main collocated server I now usually update daily. The three other Gentoo machines I maintain I update at least once a month. The daily updates take almost no time at all. The other machines take a little more thought, but I don't watch them compile and on a monthly basis isn't usually messy and takes very little of my time most months.

In the meantime, I am fastidious about reading the news and following the advice contained therein as doing so makes the major issues easier.

Don't be left behind. It's your choice. The problems are not in Gentoo. A poor workman blames his tools. https://thecontentauthority.com/blog/wh ... tools-mean

[segmentation-fault]

Why not temporary remove the world file. Then update your system. Then restore your world file, possibly in small steps.

Isn't this equivalent to the --ignore-world option proposed by Hu? In that case, I prefer using the option, as it looks less radical to me. I mean, next time when one omits the option the world file will be used again, so it's not as radical as totally removing and rebuilding it.

[segmentation-fault]

Yes, it was a rant.

Well, not exactly. Yes and no. If I tell you "please be my quest for a while, I am going to show you how much I have to rant each time I update" - is it a rant?

This has been said before -- Gentoo follows upstream and upstream continues to accelerate at an increasing rate.

Yes, this too is a problem - but I'll discuss it another time.

What used to be OK, updating a few times a year has evolved into a major ordeal. This requires updating more frequently to avoid the ordeal.

This is exactly the point! Now, let's sit quiet for a moment and ponder about it. Is what you say ("updating more frequently") a logical necessity, or the silent recognition that our tools cannot cope with this accelerated pace?

Suppose you update monthly today. If tomorrow the pace accelerates even more, will you then be telling me (and yourself) "now we must update daily"? And if it goes even faster, will you say "oh, now you must update hourly!"?

Will you never come to think "we must make portage faster"? Or "we must make portage better"? Or "we need some flexible tools/options to cope with this increasing complexity"? Will your only solution be "update more frequently and even more frequently!"?

I am where you all will be tomorrow[1]. What will you do then?

---

[1] No, I am not arrogant. It's the truth - Hu has said that my world file has more entries that all his installed programs. What will you do if one day you too find yourself with ~2000 files in your wold file? Or will you avoid it, out of fear of failing? That's where the system is curbing your possibilities - one should never allow that to happen. And even if your world file remains small, an accelerating pace means my 1 year 250 days today will be your 4 months tomorrow - what then?

I hope you (I mean: all) realize that the problems are also in Gentoo and something must be done about it (--ignore-world is a wonderful option, but is only a start) - to make Gentoo fun again!

[alamahant]

I was wondering if
https://wiki.gentoo.org/wiki/Upgrading_ ... ild_chroot
would have helped somehow.
Ease the pain a bit...

[pingtoo]

I agree with most of segmentation-fault point. It is a scale issue. when changes are few emerge feel ok, but for a very long delay update emerge just feel wrong. If emerge does not hold the concept to solve the problem in one shot and accept some kind of segmentation (no pun intended), for example use day range with git history. This may feel less painful.

For example I plan to build a binhost for my RPI4 desktop. I clone portage tree. branch out on each significant date on portage news and update binhost to that point then move on to next branch. This way I can have working environment for my everyday use and enjoy of solving emerge update problem at manageable scale.

This slow update idea is not mine, I lean from NeddySeagoon.

BTW, I never able to find where in Gentoo official document stated that user of Gentoo need frequent update or what is the frequency. I wonder if it is stated officially that use of Gentoo require perform high frequency update and each update might take up to a day to complete, few will want to use Gentoo.

[figueroa]

...
BTW, I never able to find where in Gentoo official document stated that user of Gentoo need frequent update or what is the frequency. I wonder if it is stated officially that use of Gentoo require perform high frequency update and each update might take up to a day to complete, few will want to use Gentoo.

Here: https://wiki.gentoo.org/wiki/Upgrading_Gentoo

[Hu]

you are the hero of the day! This indeed comes close to what I desperately needed!

Before these threads, I didn't even know that option existed. I found it while reading man emerge looking for a solution to your problem.

It is "broader" in its "pruning" than my proposed option (that would prune only USE flag dependencies), but still is a life-saver!

Broader, but necessary. As I described above, the conflicts you reported may have been gated behind USE flags that you could disable, but they could just as readily have been hard conflicts that can only be solved by emerge --unmerge or emerge --ignore-world.

Why not temporary remove the world file. Then update your system. Then restore your world file, possibly in small steps.

Isn't this equivalent to the --ignore-world option proposed by Hu?

Mostly, but not exactly. --ignore-world will always ignore everything in @world, so you must keep using it until all the relevant conflicts are solved. Irre's approach would let you initially ignore everything, then selectively ignore fewer things as the system approached a consistent state. This could be helpful if you want to solve problems piecemeal, and have Portage check that you are not regressing things as you go.

Saying that we need better tooling is fine, but I would then question: what exactly would make Portage "better" here? Do you want it to dump out a list of paths that will definitely break something, and let you pick through them to find the path that is least bad for your use case? I underline that qualifier, because different systems may have different choices for least bad. Consider a personal use system where the administrator has routine console access and does not need a GUI (but finds it convenient). On that system, temporarily breaking the entire Desktop Environment might be an acceptable trade to let Portage step forward a long distance quickly. Next, consider a system where the administrator logs in remotely from another city, and console access is used by a user who must have a working GUI at all times. On that system, breaking the Desktop Environment is not acceptable, so the administrator chooses a more difficult path that keeps the DE working. I can envision a variety of system usage models, each with their own rules for what packages can be freely broken for minutes or for days, and a core list of packages that must not be broken, no matter how much inconvenience is added by keeping them functional. I think Portage already partially supports this approach. Define two sets: @critical and @optional. Add both to @world. Load the sets with packages according to this system's use case. When you hit a hard block like this, remove @optional from @world. Keep @critical in @world. This should force Portage to avoid paths that break anything in @critical. For a more extreme, but definitely effective, version, you can emerge --unmerge outdated packages that cause dependency conflicts. Be careful here, as removing the wrong thing may make the system unusable.

Do you propose that the Gentoo developers should keep in-tree, and test for, upgrade paths that will allow a system to step forward by 18 months at a time? It's theoretically possible, but a lot of work to ask volunteers to undertake, and a lot of extra testing. If we support 18, why not 24, or 180? With enough old ebuilds, it should be possible to take a Gentoo from 15 years ago and bring it up to current, by stepping through all the major phases from that time. It would be very time consuming though.

[figueroa]

To assist with doing frequent (daily?) updates with the least possible waste of time, I automate certain preliminaries.

I have "emerge -- sync" followed by "emerge -uDU @world -p" run via root's crontab every morning at 7:15 am local. The output of "emerge --sync" is automatically emailed to me, and the output of "emerge -uDU @world -p" is also emailed to me with another small script. Here are the scripts. Adjust to suit your use case.

Code: Select all

$ cat ~/bin/emerge.scr
#!/bin/sh
emerge --sync
echo "emerge -uDU @world -p" > /home/USERNAME/bin/emergeauto.txt
emerge -uDU @world -p >> /home/USERNAME/bin/emergeauto.txt 2>&1
#Put this line in your user's crontab to run a few minutes after above script (i.e. 7:30).
#/home/USERNAME/bin/mailx.scr

Code: Select all

$ cat ~/bin/mailx.scr
#!/bin/sh
mail -s "bethel emerge output" USER@DOMAIN.COM < /home/USERNAME/bin/emergeauto.txt

Explanation: I keep my personal scripts and their temporary output files in my ~/bin directory. I find it convenient and informative to review the output of these files with my morning coffee. The "mail" command in the final script is from mailx.

Code: Select all

$ equery b /usr/bin/mail
 * Searching for /usr/bin/mail ... 
mail-client/mailx-8.1.2.20180807 (/usr/bin/mail)

This obviously requires your Gentoo systems to be set up to send mail. For those who don't have a full-blown email system installed, an easy way to work around that is to use the mutt email client, which once set up can easily send email from the command-line or a script.

[figueroa]

...
Exactly. And that's the problem. Even with smaller dependency graphs (my world file has more than 1700 packages and there are more than 4000 installed all together), ...

I think you should at least consider that your world file may be badly polluted and your overall system (more than 4000 packages) may be obese. Such comorbidities may easily be fatal.

[pingtoo]

@figueroa, Thanks for the information. I like your scripts.

Here: https://wiki.gentoo.org/wiki/Upgrading_Gentoo

[segmentation-fault]

figueroa,

thank you for your scripts. Although I don't think I will update daily, but who knows, maybe twice a year for the start...

I think you should at least consider that your world file may be badly polluted and your overall system (more than 4000 packages) may be obese. Such comorbidities may easily be fatal.

This sounds so much like Dr. Now and My 600-lb life reality show...

- "Sir, I would so much like to be accepted to your anti-obesity program for world files - I have meticulously followed your daily update instructions the past few months..."

- "I am sorry, your last depclean log shows that you managed to get rid of only 5 superfluous files - that's not enough! This shows me that your file intake must be much higher than prescribed! I must warn you that the state of your world file is now more critical than ever - your file BMI is too high. Your system is very obese - and such comorbidities may easily be fatal."

From: "My life with a 1700 lines world file" - Follow us on Gentoo.tv !

Sorry, with all due respect to those real-life heroes, but I could not resist...

[pingtoo]

A follow up in this gentoo update topic,

In old day it was suggested do emerge @system before emerge @world, Is this no longer required due to emerge capability improvement?

Will doing emerge ... @system May have helped in this case? at least reduce the frustration for performing update?

As Hu point out, using *set* concept could possible help in many situation, is there a way to promote (especially in handbook) more use of the *set* concept?

I personally use *set* concept to help in my binhost build process, so I can cut my emerge -uDN @world in to several days. I usually collect those failed ebuild into a set file so I will fix them in another day, and let thing that can be done finish. This way I can feel for each emerge involve I accomplish something.

[segmentation-fault]

what exactly would make Portage "better" here?

As I said from the start, I am looking at it from a user perspective. What does a user want? A painless update procedure. Let's assume the procedure is painless for update intervals of, say, 6 months up to, maximally, a year. But what do you have to offer to users who, for whatever reason, did not upgrade that often? Do you tell them "go untangle this unsolvable knot yourself?". Do you offer them endless cryptic commands to try?

Let me elaborate on the last question. I have seen threads here, with you and many others here spending an enormous amount of energy into resolving dependency problems. And I have thought "why don't we have better tools for this, why do highly qualified people have to spend so much energy on this?". I mean, we could all be programming the next-generation, multi-processor, CUDA-enabled dependency resolver for portage - but no, we have to spend our precious time on such problems!

So I was thinking of a toolset of small, but effective programs or options that, combined in series of operations, would resolve, in multiple, clever thought-out steps, every seemingly unresolvable dependency problem. And, starting from my own painful experience detailed at the start, I think something that enables the user to say:

- Update Perl, but Perl and its modules ONLY - don't care about programs with the 'perl' flag, for example.
- Update Lua ONLY - spare me the hassle to go through Qt, or anything else, just because some Qt program has the 'lua' USE flag set.
- Update icu ONLY.

Something like that. The general idea is being able to update X without "going too far". So that one can combine these to update parts of the system, until the rest is manageable with the standard procedure. But not like this

Code: Select all

emerge -1av --ignore-world --ignore-built-slot-operator-deps=y $(eix --only-names -IC dev-qt)

which, although it has worked fantastically to update all installed dev-qt packages on my system (just finished), is still too cryptic for the average user IMO.