| View previous topic :: View next topic |
| Author |
Message |
Hypnos
Advocate
Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent
|
 Posted: Sun Jun 24, 2018 1:58 am Post subject: Paludis last rites -- migrating back to Portage |
 |
|
Looks like Paludis support in Gentoo is on its deathbed: https://bugs.gentoo.org/658278
Any tips on migrating back to Portage?
(Ironically, it was Paludis that told me that Paludis is now repo masked and on its way out. My Gentoo install was born in 2002, and first logged Paludis install was 2010.)
_________________
Personal overlay | Simple backup scheme |
|
| Back to top |
|
 |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jun 24, 2018 6:29 am Post subject: |
|
|
Almost every of these bugs is a showstopper for gentoo (some even for exherbo, but apparently nobody cares).
There is a sever problem when switching package managers which prevented me to do such experiments from the very beginning: The /var/db/pkg format differs.
I am afraid you will have to emerge -e @world after a change or you will risk all sort of subtle breakage. |
|
| Back to top |
|
|
Hypnos
Advocate
Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent
|
| Posted: Mon Jun 25, 2018 1:57 am Post subject: |
|
|
| mv wrote: | | There is a sever problem when switching package managers which prevented me to do such experiments from the very beginning: The /var/db/pkg format differs. |
That's interesting; when I installed Paludis ~8 years ago it used the Portage /var/db/pkg natively -- have the formats diverged?
| Quote: | | I am afraid you will have to emerge -e @world after a change or you will risk all sort of subtle breakage. |
Might be a sensible precaution nonetheless.
_________________
Personal overlay | Simple backup scheme |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Jun 25, 2018 8:52 am Post subject: |
|
|
| Hypnos wrote: | | have the formats diverged? |
I would expect. For instance, portage I know that obtained entries for "repository", BUILD_TIME, "NEEDED.ELF.2", and (in EAPI=7) BDEPEND. I think also DEFINED_PHASES and *FLAGS are relatively new. I have now idea what paludis obtained over time.
Moreover, environment.bz2 might be arbitrarily incompatible, i.e. there might be even problems with unmerging. So not even emerge -e @world is guaranteed to work reliably; a complete reinstall might be necessary (although I would use this only if you observe problems). Comparing lists of orphaned files before and after the re-emerge of world probably won't hurt, just in case. (And as usual: keep backups). |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Mon Jun 25, 2018 5:42 pm Post subject: |
|
|
environment.bz2 is appropriately sanitized in Portage. Paludis just dumps whatever the output of `env` gives it; if you forgot to "su -l" at any point it's going to contain a copy of everything your user session sets in those world-readable files. Fun for security, when it contains things like tempfile names, socket paths, dbus tokens...
Also, the one big incompatibility I remember is that portage lowercases one of the filenames in vardb - paludis does not (ciaranm's excuse for wontfixing it was that the PMS standard, which he wrote, doesn't tell him not to). |
|
| Back to top |
|
|
Naib
Watchman
Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy
|
| Posted: Mon Jun 25, 2018 10:20 pm Post subject: |
|
|
Couldn't have happened to a nicer project
_________________
| Quote: | | Removed by Chiitoo |
|
|
| Back to top |
|
|
firefly
Watchman
Joined: 31 Oct 2002
Posts: 5187
|
| Posted: Tue Jul 10, 2018 4:52 pm Post subject: |
|
|
I'm currently in the process of migrating back from paludis to emerge.
I test it in a VM (which is a copy of my system)
And yes the environment.bz2 differs very between paludis and emerge
I got following error messages while doing an emty tree emerge:
| Quote: | /var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: syntax error near unexpected token `('
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: ` for f in "${x}"/*.@(diff|patch);'
.[31;01m*.[0m ERROR: app-arch/bzip2-1.0.6-r9:: failed (prerm phase):
.[31;01m*.[0m error processing environment
.[31;01m*.[0m
.[31;01m*.[0m Call stack:
.[31;01m*.[0m ebuild.sh, line 562: Called die
.[31;01m*.[0m The specific snippet of code:
.[31;01m*.[0m __preprocess_ebuild_env || \
.[31;01m*.[0m die "error processing environment"
.[31;01m*.[0m
.[31;01m*.[0m If you need support, post the output of `emerge --info '=app-arch/bzip2-1.0.6-r9::'`,
.[31;01m*.[0m the complete build log and the output of `emerge -pqv '=app-arch/bzip2-1.0.6-r9::'`.
.[31;01m*.[0m The complete build log is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/build.log'.
.[31;01m*.[0m The ebuild environment file is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment'.
.[31;01m*.[0m Working directory: '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/homedir'
.[31;01m*.[0m S: '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/work/bzip2-1.0.6'
.....
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: syntax error near unexpected token `('
/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment: line 433: ` for f in "${x}"/*.@(diff|patch);'
.[31;01m*.[0m ERROR: app-arch/bzip2-1.0.6-r9:: failed:
.[31;01m*.[0m error sourcing environment
.[31;01m*.[0m
.[31;01m*.[0m Call stack:
.[31;01m*.[0m misc-functions.sh, line 17: Called source '/var/tmp/portage/._portage_reinstall_.nZeM3c/bin/ebuild.sh'
.[31;01m*.[0m ebuild.sh, line 570: Called die
.[31;01m*.[0m The specific snippet of code:
.[31;01m*.[0m source "${T}"/environment || \
.[31;01m*.[0m die "error sourcing environment"
.[31;01m*.[0m
.[31;01m*.[0m If you need support, post the output of `emerge --info '=app-arch/bzip2-1.0.6-r9::'`,
.[31;01m*.[0m the complete build log and the output of `emerge -pqv '=app-arch/bzip2-1.0.6-r9::'`.
.[31;01m*.[0m The complete build log is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/build.log'.
.[31;01m*.[0m The ebuild environment file is located at '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/temp/environment'.
.[31;01m*.[0m Working directory: '/var/tmp/portage/._unmerge_/app-arch/bzip2-1.0.6-r9/homedir'
.[31;01m*.[0m S: '/var/tmp/paludis/app-arch-bzip2-1.0.6-r9/work/bzip2-1.0.6'
!!! FAILED postrm: 1
.[31;01m * .[39;49;00mThe 'postrm' phase of the 'app-arch/bzip2-1.0.6-r9' package has failed
.[31;01m * .[39;49;00mwith exit value 1.
.[31;01m * .[39;49;00m
.[31;01m * .[39;49;00mThe problem occurred while executing the ebuild file named
.[31;01m * .[39;49;00m'bzip2-1.0.6-r9.ebuild' located in the '/var/db/pkg/app-
.[31;01m * .[39;49;00march/bzip2-1.0.6-r9' directory. If necessary, manually remove the
.[31;01m * .[39;49;00menvironment.bz2 file and/or the ebuild file located in that directory.
.[31;01m * .[39;49;00m
.[31;01m * .[39;49;00mRemoval of the environment.bz2 file is preferred since it may allow the
.[31;01m * .[39;49;00mremoval phases to execute successfully. The ebuild will be sourced and
.[31;01m * .[39;49;00mthe eclasses from the current portage tree will be used when necessary.
.[31;01m * .[39;49;00mRemoval of the ebuild file will cause the pkg_prerm() and pkg_postrm()
.[31;01m * .[39;49;00mremoval phases to be skipped entirely.
|
I deleted now all environment.bz2 files under /var/db/pkg and will rerun the empty tree emerge command
Also the ebuild files under /var/db/pkg must be deleted otherwise the same error occures. It seems with an existing ebuild file portage tries to re-create the environment.bz2 files with the data under /var/db/pkg/<category>/<package_name with version>/ and then it still creates a broken version of that file
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Tue Jul 10, 2018 8:05 pm Post subject: |
|
|
It looks like paludis is enabling the extglob shell option.
It might be sufficient to put at the beginning of /usr/lib/portage/python*/ebuild.sh
For your other problem: If environment.bz2 is missing, ebuild.sh will try to source the ebuild. Perhaps the ebuild stored in /var/db/pkg by paludis contains (or sources) also some paludis helper functions. |
|
| Back to top |
|
|
firefly
Watchman
Joined: 31 Oct 2002
Posts: 5187
|
| Posted: Tue Jul 10, 2018 8:59 pm Post subject: |
|
|
| mv wrote: | It looks like paludis is enabling the extglob shell option.
It might be sufficient to put at the beginning of /usr/lib/portage/python*/ebuild.sh |
Hmm might help but i have already deleted the environment.bz2 and emerge -aev @world is currently running fine.
And it should work because before i started the empty tree merge i made sure that an normal emerge -avuND @world would not show any packages to be updated. (If an package showed up i did an re-merge of that package with paludis)
With this pre-caution portage should only do an re-merge of existing packages with the same settings as paludis. Ideally no binary or any other data copied from the build dir to the system differs in its content.
I hope that later i will only have to copy the updated files under /var/db/pkg into my live system.
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Wed Jul 11, 2018 5:56 am Post subject: |
|
|
| firefly wrote: | | Hmm might help but i have already deleted the environment.bz2 |
The problem is that in this way that pkg_{pre,post}rm functions do not get executed. If the ebuild has not changed, this should probably not hurt, but if it has you might get orphaned files which would have been removed in these functions. |
|
| Back to top |
|
|
firefly
Watchman
Joined: 31 Oct 2002
Posts: 5187
|
| Posted: Wed Jul 11, 2018 4:57 pm Post subject: |
|
|
| mv wrote: | | firefly wrote: | | Hmm might help but i have already deleted the environment.bz2 |
The problem is that in this way that pkg_{pre,post}rm functions do not get executed. If the ebuild has not changed, this should probably not hurt, but if it has you might get orphaned files which would have been removed in these functions. |
Jupp, for such a case i made sure that all updates are installed with paludis. After migrating the configuration back to portage i didn't do an emerge --sync. So the portage-tree and overlays don't change.
And thus it should use the same ebuilds when doing the emerge -aev @world.
EDIT: I should find any orphaned files when i compare the new /var/db/pkg content in the VM with the live system version
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Thu Jul 12, 2018 8:13 am Post subject: |
|
|
| firefly wrote: | | I should find any orphaned files when i compare the new /var/db/pkg content in the VM with the live system version |
Sure; there is find_cruft (e.g. from the mv overlay) which does this quite efficiently and configurable. However, beware that there are a lot more "desired" orphaned files than one might think first. |
|
| Back to top |
|
|
firefly
Watchman
Joined: 31 Oct 2002
Posts: 5187
|
| Posted: Thu Jul 12, 2018 5:03 pm Post subject: |
|
|
| mv wrote: | | However, beware that there are a lot more "desired" orphaned files than one might think first. |
Huh? Any example?
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Thu Jul 12, 2018 5:59 pm Post subject: |
|
|
| firefly wrote: | | mv wrote: | | However, beware that there are a lot more "desired" orphaned files than one might think first. |
Huh? Any example? |
| Code: | $ find_cruft /usr|wc -l
554 |
(and find_curft already filters obvious candidates like /usr/src) |
|
| Back to top |
|
|
firefly
Watchman
Joined: 31 Oct 2002
Posts: 5187
|
| Posted: Thu Jul 12, 2018 6:29 pm Post subject: |
|
|
ah I guess we might have a misunderstanding here.
A hopefully bit more clearer explanation what I did and will do:
As already said some time. The system for which I test the migration from paludis back to portage has no pending updates. At least paludis had not shown any updates when running
| Code: | | cave resolve -U '\''*/*'\'' -d '\''*/*'\'' -P '\''*/*'\'' -c -Cs installed-packages |
Also the portage-tree wasn't updated since the last world update via paludis.
First I migrated the configuration from paludis to portage.
Then I did a test | Code: | | emerge -pvuND @world |
to check if portage might find any package which needs an update (e.g. an use change)
If there were any updates shown I either fixed the configuration (e.g. an use-flag activation changed) or re-merged that package with paludis (e.g. linguas use-flag removal of that package)
Due the incompatibility of the formats in /var/db/pkg I had to delete all .ebuild and environment.bz2 files under /var/db/pkg
After this I started an empty world merge to fill /var/db/pkg with data which is parsable by portage.
Now regaring possible orphand files.
As you said due an ebuild change the ebuild in the portage tree might install not the same files as the version under /var/db/pkg.
Those changes might be occure unnoticed. In my case it could only be caused by an update in the portage tree without an rev-bump.
And with orphaned files I had only those files in mind which where installed by an ebuild and recorded in the CONTENTS file (stored under /var/db/pkg)
To catch those files my guess is that I have only to compare the CONTENTS files of the /var/db/pkg tree filled by paludis with the version filled by portage.
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Fri Jul 13, 2018 1:43 am Post subject: |
|
|
| In a strict sense, an orphaned file is any file for which no owning ebuild can be located. Due to implementation decisions in some tools, there are some surprising "orphans" on most systems. For example, /usr/bin/gcc is an orphan for me because it is not installed by Portage merging a package. It is generated separately by gcc-config as a side effect of choosing which version of gcc I want to be default on the system. That orphan can be rebuilt (if you know which tool made it), but removing it for being an orphan would be wrong because it is still in use and still maintained. There are likely other examples of files not owned by an ebuild, but still important for proper operation. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Jul 13, 2018 6:54 am Post subject: |
|
|
| firefly wrote: | | And with orphaned files I had only those files in mind which where installed by an ebuild and recorded in the CONTENTS file |
I do not think that such files can occur because portage would remove those files even if it is not able to execute the pkg_{pre,post}rm hooks from the ebuild.
Anyway, a simple way to compare these (if you do not want to write a parser of the CONTENTS files manually) is to compare whether the output of corresponding find_cruft calls (with /var/db/pkg temporarily replaced by the respective tree) is identical. |
|
| Back to top |
|
|
firefly
Watchman
Joined: 31 Oct 2002
Posts: 5187
|
| Posted: Fri Jul 13, 2018 4:29 pm Post subject: |
|
|
| mv wrote: | | firefly wrote: | | And with orphaned files I had only those files in mind which where installed by an ebuild and recorded in the CONTENTS file |
I do not think that such files can occur because portage would remove those files even if it is not able to execute the pkg_{pre,post}rm hooks from the ebuild.
Anyway, a simple way to compare these (if you do not want to write a parser of the CONTENTS files manually) is to compare whether the output of corresponding find_cruft calls (with /var/db/pkg temporarily replaced by the respective tree) is identical. |
If this is the case then i don't know which files do you mean.
But i come to the conclusion that in my case that no orphaned files (which would be removed by "pkg_{pre,post}rm hooks from the ebuild.") will be there.
Because during the migration i use the same ebuilds which where used for the last update and should be identical to the portage tree version.
But if an ebuild without an rev-bump in the portage-tree would differ that much in case for file handling done by the pkg_{pre,post}rm hooks then i would say that is a very very bad habit!
_________________
Ein Ring, sie zu knechten, sie alle zu finden,
Ins Dunkel zu treiben und ewig zu binden
Im Lande Mordor, wo die Schatten drohn. |
|
| Back to top |
|
|
|
Portage & Programming
