Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Paludis last rites -- migrating back to Portage
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Sun Jun 24, 2018 1:58 am    Post subject: Paludis last rites -- migrating back to Portage Reply with quote

Looks like Paludis support in Gentoo is on its deathbed: https://bugs.gentoo.org/658278

Any tips on migrating back to Portage?

(Ironically, it was Paludis that told me that Paludis is now repo masked and on its way out. My Gentoo install was born in 2002, and first logged Paludis install was 2010.)
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Sun Jun 24, 2018 6:29 am    Post subject: Reply with quote

Almost every of these bugs is a showstopper for gentoo (some even for exherbo, but apparently nobody cares).

There is a sever problem when switching package managers which prevented me to do such experiments from the very beginning: The /var/db/pkg format differs.

I am afraid you will have to emerge -e @world after a change or you will risk all sort of subtle breakage.
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent

PostPosted: Mon Jun 25, 2018 1:57 am    Post subject: Reply with quote

mv wrote:
There is a sever problem when switching package managers which prevented me to do such experiments from the very beginning: The /var/db/pkg format differs.

That's interesting; when I installed Paludis ~8 years ago it used the Portage /var/db/pkg natively -- have the formats diverged?

Quote:
I am afraid you will have to emerge -e @world after a change or you will risk all sort of subtle breakage.

Might be a sensible precaution nonetheless.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Mon Jun 25, 2018 8:52 am    Post subject: Reply with quote

Hypnos wrote:
have the formats diverged?

I would expect. For instance, portage I know that obtained entries for "repository", BUILD_TIME, "NEEDED.ELF.2", and (in EAPI=7) BDEPEND. I think also DEFINED_PHASES and *FLAGS are relatively new. I have now idea what paludis obtained over time.
Moreover, environment.bz2 might be arbitrarily incompatible, i.e. there might be even problems with unmerging. So not even emerge -e @world is guaranteed to work reliably; a complete reinstall might be necessary (although I would use this only if you observe problems). Comparing lists of orphaned files before and after the re-emerge of world probably won't hurt, just in case. (And as usual: keep backups).
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5074

PostPosted: Mon Jun 25, 2018 5:42 pm    Post subject: Reply with quote

environment.bz2 is appropriately sanitized in Portage. Paludis just dumps whatever the output of `env` gives it; if you forgot to "su -l" at any point it's going to contain a copy of everything your user session sets in those world-readable files. Fun for security, when it contains things like tempfile names, socket paths, dbus tokens...

Also, the one big incompatibility I remember is that portage lowercases one of the filenames in vardb - paludis does not (ciaranm's excuse for wontfixing it was that the PMS standard, which he wrote, doesn't tell him not to).
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5354
Location: Removed by Neddy

PostPosted: Mon Jun 25, 2018 10:20 pm    Post subject: Reply with quote

Couldn't have happened to a nicer project
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
firefly
Advocate
Advocate


Joined: 31 Oct 2002
Posts: 4362

PostPosted: Tue Jul 10, 2018 4:52 pm    Post subject: Reply with quote

I'm currently in the process of migrating back from paludis to emerge.
I test it in a VM (which is a copy of my system)
And yes the environment.bz2 differs very between paludis and emerge

I got following error messages while doing an emty tree emerge:

Quote:
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: syntax error near unexpected token `('
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: ` for f in "${x}"/*.@(diff|patch);'
.[31;01m*.[0m ERROR: app-arch/bzip2-1.0.6-r9:: failed (prerm phase):
.[31;01m*.[0m error processing environment
.[31;01m*.[0m
.[31;01m*.[0m Call stack:
.[31;01m*.[0m ebuild.sh, line 562: Called die
.[31;01m*.[0m The specific snippet of code:
.[31;01m*.[0m __preprocess_ebuild_env || \
.[31;01m*.[0m die "error processing environment"
.[31;01m*.[0m
.[31;01m*.[0m If you need support, post the output of `emerge --info '=app-arch/bzip2-1.0.6-r9::'`,
.[31;01m*.[0m the complete build log and the output of `emerge -pqv '=app-arch/bzip2-1.0.6-r9::'`.
.[31;01m*.[0m The complete build log is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/build.log'.
.[31;01m*.[0m The ebuild environment file is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment'.
.[31;01m*.[0m Working directory: '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/homedir'
.[31;01m*.[0m S: '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/work/bzip2-1.0.6'
.....
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: syntax error near unexpected token `('
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: ` for f in "${x}"/*.@(diff|patch);'
.[31;01m*.[0m ERROR: app-arch/bzip2-1.0.6-r9:: failed:
.[31;01m*.[0m error sourcing environment
.[31;01m*.[0m
.[31;01m*.[0m Call stack:
.[31;01m*.[0m misc-functions.sh, line 17: Called source '/var/tmp/portage/._portage_reinstall_.nZeM3c/bin/ebuild.sh'
.[31;01m*.[0m ebuild.sh, line 570: Called die
.[31;01m*.[0m The specific snippet of code:
.[31;01m*.[0m source "${T}"/environment || \
.[31;01m*.[0m die "error sourcing environment"
.[31;01m*.[0m
.[31;01m*.[0m If you need support, post the output of `emerge --info '=app-arch/bzip2-1.0.6-r9::'`,
.[31;01m*.[0m the complete build log and the output of `emerge -pqv '=app-arch/bzip2-1.0.6-r9::'`.
.[31;01m*.[0m The complete build log is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/build.log'.
.[31;01m*.[0m The ebuild environment file is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment'.
.[31;01m*.[0m Working directory: '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/homedir'
.[31;01m*.[0m S: '/var/tmp/paludis/app-arch-bzip2-1.0.6-r9/work/bzip2-1.0.6'
!!! FAILED postrm: 1
.[31;01m * .[39;49;00mThe 'postrm' phase of the 'app-arch/bzip2-1.0.6-r9' package has failed
.[31;01m * .[39;49;00mwith exit value 1.
.[31;01m * .[39;49;00m
.[31;01m * .[39;49;00mThe problem occurred while executing the ebuild file named
.[31;01m * .[39;49;00m'bzip2-1.0.6-r9.ebuild' located in the '/var/db/pkg/app-
.[31;01m * .[39;49;00march/bzip2-1.0.6-r9' directory. If necessary, manually remove the
.[31;01m * .[39;49;00menvironment.bz2 file and/or the ebuild file located in that directory.
.[31;01m * .[39;49;00m
.[31;01m * .[39;49;00mRemoval of the environment.bz2 file is preferred since it may allow the
.[31;01m * .[39;49;00mremoval phases to execute successfully. The ebuild will be sourced and
.[31;01m * .[39;49;00mthe eclasses from the current portage tree will be used when necessary.
.[31;01m * .[39;49;00mRemoval of the ebuild file will cause the pkg_prerm() and pkg_postrm()
.[31;01m * .[39;49;00mremoval phases to be skipped entirely.

I deleted now all environment.bz2 files under /var/db/pkg and will rerun the empty tree emerge command
Also the ebuild files under /var/db/pkg must be deleted otherwise the same error occures. It seems with an existing ebuild file portage tries to re-create the environment.bz2 files with the data under /var/db/pkg/<category>/<package_name with version>/ and then it still creates a broken version of that file
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Tue Jul 10, 2018 8:05 pm    Post subject: Reply with quote

It looks like paludis is enabling the extglob shell option.
It might be sufficient to put
Code:
shopt extglob
at the beginning of /usr/lib/portage/python*/ebuild.sh
For your other problem: If environment.bz2 is missing, ebuild.sh will try to source the ebuild. Perhaps the ebuild stored in /var/db/pkg by paludis contains (or sources) also some paludis helper functions.
Back to top
View user's profile Send private message
firefly
Advocate
Advocate


Joined: 31 Oct 2002
Posts: 4362

PostPosted: Tue Jul 10, 2018 8:59 pm    Post subject: Reply with quote

mv wrote:
It looks like paludis is enabling the extglob shell option.
It might be sufficient to put
Code:
shopt extglob
at the beginning of /usr/lib/portage/python*/ebuild.sh

Hmm might help but i have already deleted the environment.bz2 and emerge -aev @world is currently running fine.
And it should work because before i started the empty tree merge i made sure that an normal emerge -avuND @world would not show any packages to be updated. (If an package showed up i did an re-merge of that package with paludis)
With this pre-caution portage should only do an re-merge of existing packages with the same settings as paludis. Ideally no binary or any other data copied from the build dir to the system differs in its content.

I hope that later i will only have to copy the updated files under /var/db/pkg into my live system.
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Wed Jul 11, 2018 5:56 am    Post subject: Reply with quote

firefly wrote:
Hmm might help but i have already deleted the environment.bz2

The problem is that in this way that pkg_{pre,post}rm functions do not get executed. If the ebuild has not changed, this should probably not hurt, but if it has you might get orphaned files which would have been removed in these functions.
Back to top
View user's profile Send private message
firefly
Advocate
Advocate


Joined: 31 Oct 2002
Posts: 4362

PostPosted: Wed Jul 11, 2018 4:57 pm    Post subject: Reply with quote

mv wrote:
firefly wrote:
Hmm might help but i have already deleted the environment.bz2

The problem is that in this way that pkg_{pre,post}rm functions do not get executed. If the ebuild has not changed, this should probably not hurt, but if it has you might get orphaned files which would have been removed in these functions.

Jupp, for such a case i made sure that all updates are installed with paludis. After migrating the configuration back to portage i didn't do an emerge --sync. So the portage-tree and overlays don't change.
And thus it should use the same ebuilds when doing the emerge -aev @world.

EDIT: I should find any orphaned files when i compare the new /var/db/pkg content in the VM with the live system version
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Thu Jul 12, 2018 8:13 am    Post subject: Reply with quote

firefly wrote:
I should find any orphaned files when i compare the new /var/db/pkg content in the VM with the live system version

Sure; there is find_cruft (e.g. from the mv overlay) which does this quite efficiently and configurable. However, beware that there are a lot more "desired" orphaned files than one might think first.
Back to top
View user's profile Send private message
firefly
Advocate
Advocate


Joined: 31 Oct 2002
Posts: 4362

PostPosted: Thu Jul 12, 2018 5:03 pm    Post subject: Reply with quote

mv wrote:
However, beware that there are a lot more "desired" orphaned files than one might think first.

Huh? Any example?
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Thu Jul 12, 2018 5:59 pm    Post subject: Reply with quote

firefly wrote:
mv wrote:
However, beware that there are a lot more "desired" orphaned files than one might think first.

Huh? Any example?

Code:
$ find_cruft /usr|wc -l
554

(and find_curft already filters obvious candidates like /usr/src)
Back to top
View user's profile Send private message
firefly
Advocate
Advocate


Joined: 31 Oct 2002
Posts: 4362

PostPosted: Thu Jul 12, 2018 6:29 pm    Post subject: Reply with quote

ah I guess we might have a misunderstanding here.

A hopefully bit more clearer explanation what I did and will do:

As already said some time. The system for which I test the migration from paludis back to portage has no pending updates. At least paludis had not shown any updates when running
Code:
cave resolve -U '\''*/*'\'' -d '\''*/*'\'' -P '\''*/*'\'' -c -Cs installed-packages

Also the portage-tree wasn't updated since the last world update via paludis.

First I migrated the configuration from paludis to portage.

Then I did a test
Code:
emerge -pvuND @world
to check if portage might find any package which needs an update (e.g. an use change)
If there were any updates shown I either fixed the configuration (e.g. an use-flag activation changed) or re-merged that package with paludis (e.g. linguas use-flag removal of that package)

Due the incompatibility of the formats in /var/db/pkg I had to delete all .ebuild and environment.bz2 files under /var/db/pkg

After this I started an empty world merge to fill /var/db/pkg with data which is parsable by portage.

Now regaring possible orphand files.
As you said due an ebuild change the ebuild in the portage tree might install not the same files as the version under /var/db/pkg.
Those changes might be occure unnoticed. In my case it could only be caused by an update in the portage tree without an rev-bump.

And with orphaned files I had only those files in mind which where installed by an ebuild and recorded in the CONTENTS file (stored under /var/db/pkg)
To catch those files my guess is that I have only to compare the CONTENTS files of the /var/db/pkg tree filled by paludis with the version filled by portage.
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12490

PostPosted: Fri Jul 13, 2018 1:43 am    Post subject: Reply with quote

In a strict sense, an orphaned file is any file for which no owning ebuild can be located. Due to implementation decisions in some tools, there are some surprising "orphans" on most systems. For example, /usr/bin/gcc is an orphan for me because it is not installed by Portage merging a package. It is generated separately by gcc-config as a side effect of choosing which version of gcc I want to be default on the system. That orphan can be rebuilt (if you know which tool made it), but removing it for being an orphan would be wrong because it is still in use and still maintained. There are likely other examples of files not owned by an ebuild, but still important for proper operation.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6107

PostPosted: Fri Jul 13, 2018 6:54 am    Post subject: Reply with quote

firefly wrote:
And with orphaned files I had only those files in mind which where installed by an ebuild and recorded in the CONTENTS file

I do not think that such files can occur because portage would remove those files even if it is not able to execute the pkg_{pre,post}rm hooks from the ebuild.
Anyway, a simple way to compare these (if you do not want to write a parser of the CONTENTS files manually) is to compare whether the output of corresponding find_cruft calls (with /var/db/pkg temporarily replaced by the respective tree) is identical.
Back to top
View user's profile Send private message
firefly
Advocate
Advocate


Joined: 31 Oct 2002
Posts: 4362

PostPosted: Fri Jul 13, 2018 4:29 pm    Post subject: Reply with quote

mv wrote:
firefly wrote:
And with orphaned files I had only those files in mind which where installed by an ebuild and recorded in the CONTENTS file

I do not think that such files can occur because portage would remove those files even if it is not able to execute the pkg_{pre,post}rm hooks from the ebuild.
Anyway, a simple way to compare these (if you do not want to write a parser of the CONTENTS files manually) is to compare whether the output of corresponding find_cruft calls (with /var/db/pkg temporarily replaced by the respective tree) is identical.

If this is the case then i don't know which files do you mean.
But i come to the conclusion that in my case that no orphaned files (which would be removed by "pkg_{pre,post}rm hooks from the ebuild.") will be there.
Because during the migration i use the same ebuilds which where used for the last update and should be identical to the portage tree version.
But if an ebuild without an rev-bump in the portage-tree would differ that much in case for file handling done by the pkg_{pre,post}rm hooks then i would say that is a very very bad habit!
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum