| View previous topic :: View next topic |
| Author |
Message |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
 Posted: Wed Sep 13, 2017 8:41 am Post subject: public overlays and arch-specific USE masks |
 |
|
Is it possible to set arch-specific USE masks for a public overlay?
Assume, for instance, package cat/foo is ARCH="~amd64 ~x86", and in an overlay, there is a package cat/bar with optional support for package foo with ARCH="~amd64 ~x86 ~sparc".
Of course, one could do in the ebuild of cat/bar:
| Code: | IUSE="foo"
RDEPEND="foo? ( !sparc? ( cat/foo ) )" |
but this is not recommended, because on sparc the USE-flag foo is silently ignored (and also a change of that flag would imply a completely redundant recompilation).
Of course, one could do in addition something like
| Code: | | REQUIRED_USE="sparc? ( !foo )" |
but this is not nice, either, because it requires manual setting of USE-flags by the user of the ebuild.
Moreover, although semantically one could do after the latter the simpler
| Code: | | RDEPEND="foo? ( cat/foo )" |
in practice, one cannot, because repoman will throw warnings.
The "clean" thing would be to use only the latter simple RDEPEND, avoid REQUIRED_USE, and instead to mask USE=foo for the sparc profile.
Within the gentoo repository, this is not a problem. In a local "sparc" profile, this is also not a problem, but this is already not nice, because it should be done in the local overlay instead of the profile.
In particular, how can this be done for a public overlay? Is it possible to make even repoman happy?
Edit: Portage enhancement bug |
|
| Back to top |
|
 |
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Sat Sep 16, 2017 9:44 pm Post subject: |
|
|
My understanding is that there is no need to do any of this at the ebuild level (whether that ebuild is in the main repo or in an overlay). This is already handled at the profile level.
For example, USE="altivec" is supported only in the ppc/ppc64 profiles, and not the others. So if I am using the amd64 profile and want to emerge media-video/ffmpeg (which does have altivec as a USE flag), turning this flag on or off has no effect, and does not cause any rebuilds either. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Sep 17, 2017 6:25 am Post subject: |
|
|
| cyrillic wrote: | | This is already handled at the profile level. |
That's how it is handled in the main gentoo repository, that is, by masking or forcing a USE-flag, either globally or for a specific ebuild in the corresponding arch-specific profile.
The problem is that an overlay has no way to do this, because the arch-specific profiles exist only in the main gentoo repository and cannot be extended/modified by an overlay AFAIK:
Although a local overlay can e.g. have a profiles/use.{mask,force} file, this works only for all profiles (i.e. arches) simultaneously.
It seems to be that this is really a missing feature in portage/pms; that's why the mentioned bug was reported. |
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Sun Sep 17, 2017 8:32 pm Post subject: Re: public overlays and arch-specific USE masks |
|
|
I reread your example
| mv wrote: | | Assume, for instance, package cat/foo is ARCH="~amd64 ~x86", and in an overlay, there is a package cat/bar with optional support for package foo with ARCH="~amd64 ~x86 ~sparc". |
and if your ebuild does not include any sparc-related checks, it should still do the right thing.
If you run this command on your sparc machine
| Code: | | # USE="foo" emerge cat/bar |
then portage will calculate dependencies and respond with
| Code: | | emerge: there are no ebuilds to satisfy "cat/foo". |
|
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Tue Sep 19, 2017 6:44 am Post subject: Re: public overlays and arch-specific USE masks |
|
|
| cyrillic wrote: | If you run this command on your sparc machine
| Code: | | # USE="foo" emerge cat/bar |
then portage will calculate dependencies and respond with
| Code: | | emerge: there are no ebuilds to satisfy "cat/foo". |
|
It would rather respond with something like
| Code: | emerge: cat/foo is masked (masked by: missing keyword)
pulled in by cat/bar |
Moreover, repoman would already throw an error of type error.keyword. Actually, the latter is the main annoyance for a public overlay: It practically means that layman becomes useless because it throws hundreds of errors which cannot be avioded (except by building the arch check into the ebuilds which is not how such issues are solved in the main gentoo repository) |
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Wed Sep 20, 2017 12:34 am Post subject: |
|
|
Even though I have been around Gentoo for a while, I am not a programmer, just a regular user. So when you propose something that looks like it will make life more difficult for regular users, I reserve the right to object.
For the record, I am not really happy with the way Gentoo does ARCH-specific keywords since the vast majority of packages are ARCH-agnostic (or should be). Notable exceptions are bootloaders, binary blobs, and defective packages (and the defective packages should have bugs filed against them).
Fortunately there is an easy way to bypass this by adding this to your make.conf | Code: | | ACCEPT_KEYWORDS="* ~*" |
If you put additional ARCH-specific checks in your ebuilds, then this just adds an additional layer of nuisance, so I am begging you, please don't do this. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9530
Location: beyond the rim
|
| Posted: Wed Sep 20, 2017 8:14 am Post subject: |
|
|
Pragmatic option if you want to avoid user intervention would be to use the sparc conditional in *DEPEND and add a warning about the skipped use flag in pkg_setup or pkg_pretend (or whatever the name of the depphase callback is).
Not a big fan of trying to formalize every possible option under the sun in the PM, just leads to the mess that was caused with the overuse of REQUIRED_USE initially. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Thu Sep 21, 2017 11:08 am Post subject: |
|
|
| cyrillic wrote: | | So when you propose something that looks like it will make life more difficult for regular users |
The opposite is the case: If the overlay could extend arch-specific profiles, users of such a profile would have auto-enabled/-disabled the USE-flags in the correct way for the overlay (what happens currently only for the main gentoo repository) instead of forcing the user to manually enable/disable it (or tacitly ignore a USE-flag, perhaps only printing a warning a Genone suggests).
| Genone wrote: | | add a warning about the skipped use flag in pkg_setup or pkg_pretend |
It is not good practice to informi users about severe issues (and not installing a big functionality despite the USE-flag is set is a major issue) only through a message during installation.
Of course, if no proper solution is found, this is probably about the only thing which can be done, but it is very hackish.
| Quote: | | Not a big fan of trying to formalize every possible option under the sun in the PM |
++
However, in this case the thing is: There has been an agreement how to deal with forced/disabled USE-flags, and this has a good practice since years in gentoo. It is only that this whole mechanism is completely unavailable for overlays. This is not an issue for gentoo developers (who can add to the gentoo's repositories profiles whatever they want) but only for users/contributors of overlays. |
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Thu Sep 21, 2017 10:39 pm Post subject: |
|
|
Ok, now I am curious why package cat/foo is missing the sparc keyword if said package provides functionality that would be important to sparc users.
If package cat/foo is known to be broken on sparc, then maybe someone could provide a patch to fix it, or if it has never been tested, then see if it works by overriding the keywords, and file a bug to have the keyword added.
| mv wrote: |
If the overlay could extend arch-specific profiles, users of such a profile would have auto-enabled/-disabled the USE-flags in the correct way for the overlay ...
|
As far as I know, new and/or modified profiles can be added to an overlay, but it is still up to the users to select your profile (perhaps manually) over one of the official profiles (with eselect profile). |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Fri Sep 22, 2017 2:10 am Post subject: |
|
|
| I know exactly what you're talking about, and AFAIK it's impossible to do sanely. There really needs to be a way for overlay profiles to inherit from their master repository, would make things so much simpler. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Sep 22, 2017 7:34 am Post subject: |
|
|
| cyrillic wrote: | | Ok, now I am curious why package cat/foo is missing the sparc keyword |
Typical cases when this happens is if cat/bar is a front-end supporting many optional backends (e.g. a wrapper for various encoders, archivers, or other sorts of data converters) which should be configured at install time.
Then often cat/bar and most backends run on most architectures, but of course one neither wants to block an architecture only because some backends are not (yet) available, nor does one want to support only those backends which surely run on every exotic architecture.
| Quote: | | As far as I know, new and/or modified profiles can be added to an overlay |
Yes, but first, these would have to be complete profiles (i.e. only to mask some USE-flags for certain architectures, one would practically have to fork [i.e.: regularly update] the whole profiles/ tree). I also doubt that any user would switch to such an overlay-provided profile "only" to get some USE-flags masking: I would not do this.
Second, these profiles are "hidden" to portage: They are just like any other directory which the user can use as his profile. Portage does not know that a certain such directory is supposed to be used for a certain architecture: In layman only the arch profiles from the gentoo repository are hard-coded as a basis. In particular, layman still cannot be used to check the sanity of such an overlay, and IMHO this is correct: the overlay is not sane, if it cannot be "correctly" used without switching to a special overlay-only profile.
This full-profile-in-overlay concept latest breaks down in the moment when the user has 2 such overlays: Then he is forced to merge and maintain these profiles manually. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9530
Location: beyond the rim
|
| Posted: Fri Sep 22, 2017 10:31 am Post subject: |
|
|
| mv wrote: | | Genone wrote: | | add a warning about the skipped use flag in pkg_setup or pkg_pretend |
It is not good practice to informi users about severe issues (and not installing a big functionality despite the USE-flag is set is a major issue) only through a message during installation.
Of course, if no proper solution is found, this is probably about the only thing which can be done, but it is very hackish. |
Which is why I called it pragmatic, not ideal.
| Ant P. wrote: | | I know exactly what you're talking about, and AFAIK it's impossible to do sanely. There really needs to be a way for overlay profiles to inherit from their master repository, would make things so much simpler. |
Nah, that's the wrong way to approach things. A "normal" overlay that just wants to provide some extra packages should never have to deal with profiles at all. The key problem is that ebuilds implicitly depend on profiles in the first place. The repoman check that is causing the actual problem for mv is based on that dependency. I've never been a big fan how the whole profile stuff is handled, but then I don't really have a better plan either without duplicating a lot of stuff. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Sep 22, 2017 12:15 pm Post subject: |
|
|
| Genone wrote: | | I've never been a big fan how the whole profile stuff is handled |
++
The package.use.{mask,force} should IMHO be mainly be done in the ebuild, at least, for the purpose of masking/unmasking due to dependencies. (That special-purpose profiles like hardened, uclib, or systemd have this possibility in addition is fine, but it should not be the only available mechanism for masking).
Actually, even better, the package manager could do the use-masking automatically according to conditional dependencies (so that e.g. a stable use mask is removed automatically for an architecture once the dependency becomes stable). |
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Fri Sep 22, 2017 3:56 pm Post subject: |
|
|
| mv wrote: | | ... but of course one neither wants to block an architecture only because some backends are not (yet) available, nor does one want to support only those backends which surely run on every exotic architecture. |
Then don't block anything. Simply let the user decide which backend(s) they want. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Sep 22, 2017 6:23 pm Post subject: |
|
|
| cyrillic wrote: | | Then don't block anything. Simply let the user decide which backend(s) they want. |
Then the user might get pulled in a masked-by-keyword package dependency from a non-masked package. Portage will try to solve this dependency by suggesting to unmask the package, this package will not compile, and the user will end up with a mess unless he knows clumsy details about the ebuild which is actually causing the trouble (which is not the ebuild about which he will eventually receive the error). That's exactly the sort of confusing problems which should be avoided by using repoman. |
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Sat Sep 23, 2017 12:32 am Post subject: |
|
|
| mv wrote: | | Portage will try to solve this dependency by suggesting to unmask the package, this package will not compile, and ... |
In most cases, this package would compile just fine unless it is a defective package like I mentioned above. I am also pretty sure portage would never suggest unmasking a package for the wrong ARCH ( I only do this because I am fearless  ). |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sat Sep 23, 2017 6:55 am Post subject: |
|
|
| cyrillic wrote: | | In most cases, this package would compile just fine |
It would probably already have been marked ~ARCH if it and all dependencies it requires would compile and run fine.
In any case, this is an issue/decision of the main repository (if the non-keyworded package is in the main repository), and except for very exceptional situations, an overlay should not mess up the decisions of the main repository; if it does, it must include the corresponding ebuild and possibly its dependencies.
| Quote: | | I am also pretty sure portage would never suggest unmasking a package |
| emerge -v app-shells/bb wrote: | These are the packages that would be merged, in order:
Calculating dependencies . ... done!
[ebuild N *] app-shells/aa-1::xx 0 KiB
[ebuild N ] app-shells/bb-1::xx USE="depend-on-aa" 0 KiB
Total: 2 packages (2 new), Size of downloads: 0 KiB
The following keyword changes are necessary to proceed:
(see "package.accept_keywords" in the portage(5) man page for more details)
# required by app-shells/bb-1::xx[depend-on-aa]
# required by app-shells/bb (argument)
=app-shells/aa-1 ** |
But it plays no role how portage reacts to a broken tree: Having an KEYWORDS="~ARCH" or KEYWORDS="ARCH" ebuild which under some (non-masked) USE situations depends on an ebuild without the corresponding keyword is by definition a broken tree. That's why repoman does not only give a warning but a fatal error:
| repoman wrote: | dependency.bad [fatal] 48
app-shells/bb/bb-1.ebuild: DEPEND: amd64(default/linux/amd64/13.0)
['app-shells/aa']
...
47 other fatal messages due to the same issue removed from this posting |
|
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Sun Sep 24, 2017 12:08 am Post subject: |
|
|
| mv wrote: | | Having an KEYWORDS="~ARCH" or KEYWORDS="ARCH" ebuild which under some (non-masked) USE situations depends on an ebuild without the corresponding keyword is by definition a broken tree. |
I agree with you there.
I was looking forward to doing some testing, possibly patching cat/foo, and submitting a keyword request to return the tree to non-broken status ... but there is an easier way.
Think of cat/foo as an optional package, and not a dependency. Remove foo from USE and DEPEND in the cat/bar ebuild and add a notice in pkg_postinst() like this one.
| sys-boot/grub wrote: | elog
elog "You may consider installing the following optional packages:"
optfeature "Detect other operating systems (grub-mkconfig)" sys-boot/os-prober
optfeature "Create rescue media (grub-mkrescue)" dev-libs/libisoburn
optfeature "Enable RAID device detection" sys-fs/mdadm
|
That way, the users are happy with the choices available to them and repoman is happy too. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Sep 24, 2017 5:45 am Post subject: |
|
|
| cyrillic wrote: | | Think of cat/foo as an optional package, and not a dependency |
If cat/foo is an optional dependency, this is of course what should be done - this has nothing to do with arches.
However, being an optional dependency means that cat/bar checks at the beginning of each run whether cat/foo is installed and changes its behaviour correspondingly, and that this is desired (e.g. there is a runtime interface in which the user can decide whether to use that feature).
This does not work for true dependencies, i.e. if it is decided at install time whether support for cat/foo is built into cat/bar (e.g. because the binary [or at least a plug-in] links agains a library from cat/foo, or because it chooses in a hard-coded manner between cat/foo or another backend).
Edits: By definition, a dependency is non-optional if the content of at least one installed file depends on it.
That opt_feature currently is the best/only solution for optional dependencies (and currently keeps the package manager out of the game) is a different story. The better solution would be runtime switchable USE-flags which is hopefully introduced in EAPI 7. In this case the arch-specific masking problem would return... |
|
| Back to top |
|
|
cyrillic
Watchman
Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA
|
| Posted: Sun Sep 24, 2017 11:09 pm Post subject: |
|
|
Hey, isn't optional dependency an oxymoron ...
Hold that thought, I think my jumbo shrimp are almost done cooking.  |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Sep 25, 2017 6:37 am Post subject: |
|
|
| cyrillic wrote: | | Hey, isn't optional dependency an oxymoron ... |
It's the terminology used in gentoo for the topic. Debian calls these things "recommended" (or "recommended dependencies"? One would have to check)
Edit: I am not a native English speaker, but concerning the "oxymoron", I think that the problem is that "dependency" is actually a misnomer: It should perhaps better be called "used package" or something similar. Then also "optionally used package" does not become an oxymoron anymore. However, {R,P,}DEPEND already have a long history in Gentoo, and linguistic bikeshedding is certainly not a valid reason to rename them. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
