| View previous topic :: View next topic |
| Author |
Message |
whjeon
Tux's lil' helper
Joined: 20 Nov 2017
Posts: 82
Location: Seoul,Korea(Republic of)
|
 Posted: Wed May 16, 2018 3:54 pm Post subject: Is it OK to use additional repositories? |
 |
|
Dear Mentors!
I have a question related to a layman.
Is it recommended or proved by Gentoo upstream to use additional repositories?
Just curious because I want to keep my system clean and safe.
(P.S. I couldn't find good forum for this, sorry if this doesn't fit here) |
|
| Back to top |
|
 |
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9675
Location: almost Mile High in the USA
|
| Posted: Wed May 16, 2018 3:59 pm Post subject: |
|
|
Yeah, probably the right subforum.
The overlay system was added for convenience and it should work just fine, but it's hard to debug overlays outside of the developers' control.
Outside of my control too. What I can do to help people with their Gentoo problems sometimes depends on what I can do to reproduce the problem on my own system - and if I have to use an overlay that I may not trust, then my debug effort stops. I suspect other devs may have the same view unless they're also supporting their own overlay.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
whjeon
Tux's lil' helper
Joined: 20 Nov 2017
Posts: 82
Location: Seoul,Korea(Republic of)
|
| Posted: Wed May 16, 2018 4:03 pm Post subject: |
|
|
| eccerr0r wrote: | Yeah, probably the right subforum.
The overlay system was added for convenience and it should work just fine, but it's hard to debug overlays outside of the developers' control.
Outside of my control too. What I can do to help people with their Gentoo problems sometimes depends on what I can do to reproduce the problem on my own system - and if I have to use an overlay that I may not trust, then my debug effort stops. I suspect other devs may have the same view unless they're also supporting their own overlay. |
Thanks for your quick reply! And I decided not to use overlay.
How about Masked packages? According to Wiki, it says a few hours or days to be unmasked, and is it true?
So, I think I should wait for days. I have atom editor, and some gnome themes to install which are masked for amd64( ~amd64).
Is it recommended not to unmask manually and wait? Am I doing right? |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Wed May 16, 2018 4:26 pm Post subject: |
|
|
whjeon ...
to add to what eccerr0r has said, while you may want x,y,z from one or other overlay you probably don't want other packages selected for merger simply because they exist and have a higher priority/version. For this reason I consider the following as the safest method of managing what packages are selected, and from where.
Firstly mask each overlay with a category/package::repo wildcard ...
| /etc/portage/package.mask/overlay.mask: | | */*::<name_of_repo> |
Then selectively unmask those packages you want selected ...
| /etc/portage/package.unmask: | | category/package::<name_of_repo> |
You may still have to 'package.accept_keywords' depending on what keywords are set in the ebuild by the repo maintainer, but with the above you only get those packages that you've explictly unmasked.
HTH & best ... khay |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9675
Location: almost Mile High in the USA
|
| Posted: Wed May 16, 2018 4:29 pm Post subject: |
|
|
It depends on the mask on what the situation is with that package.
If it was masked because it was never keyworded, then these were never tested. It doesn't mean it won't work though there's no guarantee. However typically it may not be usable for that architecture (like running nvidia-drivers on hppa... good luck getting that to work.)
The "~" keyworded are 'experimental' but someone likely has tested it lightly or have some random issues with it, but it could work just fine for people.
If there was an outright mask in package.mask, usually there's an explanation that should show up when trying to use it.
It sounds like you're in the second case. You're free to unmask it and see if it works for you. Since it is in portage/main tree, people have copies of what you unmasked so it isn't a problem to try to replicate your issues.
The unmask policy depends on the situation with the package. Sometimes it just lingers there waiting for more in-depth testing, sometimes it's because it's waiting for additional testing with other packages. There appears really no schedule on when something may get unmasked in portage, so if you need a particular package or version, just unmask and use, let others know what you find, maybe we can fix it.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Wed May 16, 2018 4:29 pm Post subject: Re: Is it OK to use additional repositories? |
|
|
| whjeon wrote: | | to use additional repositories? |
If the maintainers are not malevolent, it can hardly harm your system: They usually provide different packages which cannot influence the functionality of the other packages (some easy-to-detect cases like plugins left aside). There are only very few overlays which replace a gentoo package. This has to be decided on a case-by-case basis. Unless the package is a library or a basic tool it can hardly do any damage. Of course, for a stable system I would not recommend to replace a basic toolchain packages by one from an overlay.
The only "breakage" in connection with overlays which you might observe in practice is that they might contain a dependency bug which can cause the portage dependency resolver to fail. This is rare, but it can happen, especially if there are lot of gentoo dependency changes and the overlay is not yet adapted to it. However, these problems can happen also without overlays, and they are not critical. Usually, they are rather easy to solve (and you can always ask in the forum).
| Quote: | | which are masked for amd64( ~amd64). |
~amd64 is not masked, it is "testing". Officially, running a mixed amd64/~amd64 system is not supported. In practice, the only problems which it causes is again some problems with dependency resolving. (E.g. having one package ~amd64 usually forces you to have some other dependent packages ~amd64 as well.)
From my experience, the system is safer (and sometimes even more stable) if you use the "right" testing packages: Quite often, these testing packages contain bugfixes (including sometimes undetected security fixes). Of course, experiences and opinions will probably differ, here.
OTOH, using packages which are really masked (and not only testing) or which are not even marked ~amd64 is really experimental, and I would recommend to avoid this if you can.
Edit: Trivial reformulations and layout changes.
Last edited by mv on Wed May 16, 2018 4:33 pm; edited 2 times in total |
|
| Back to top |
|
|
whjeon
Tux's lil' helper
Joined: 20 Nov 2017
Posts: 82
Location: Seoul,Korea(Republic of)
|
| Posted: Wed May 16, 2018 4:29 pm Post subject: |
|
|
| khayyam wrote: | whjeon ...
to add to what eccerr0r has said, while you may want x,y,z from one or other overlay you probably don't want other packages selected for merger simply because they exist and have a higher priority/version. For this reason I consider the following as the safest method of managing what packages are selected, and from where.
Firstly mask each overlay with a category/package::repo wildcard ...
| /etc/portage/package.mask/overlay.mask: | | */*::<name_of_repo> |
Then selectively unmask those packages you want selected ...
| /etc/portage/package.unmask: | | category/package::<name_of_repo> |
You may still have to 'package.accept_keywords' depending on what keywords are set in the ebuild by the repo maintainer, but with the above you only get those packages that you've explictly unmasked.
HTH & best ... khay |
]
Thanks! That was awesome! I didn't think about that! |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Wed May 16, 2018 4:39 pm Post subject: |
|
|
| khayyam wrote: | | but with the above you only get those packages that you've explictly unmasked. |
emerge -a (which you should always use) or also eix will show you anyway which packages from the overlay are/would be used, so this is somewhat superfluous work.
But what is more important: This way you will usually miss if the overlay maintainer masks a package for some reason. This might even be critical if that reason is related to security considerations. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54210
Location: 56N 3W
|
| Posted: Wed May 16, 2018 4:59 pm Post subject: |
|
|
whjeon,
The quality of overlays varies from "mostly harmless" to "will drink all the beer in your fridge".
In practice a lot of early work on new ebuilds is performed in overlays as it simplifies testing.
These ebuilds get moved to the gentoo repo later.
~arch ebuilds have had some testing. They may be stuck in ~arch as they have dependencies that are also ~arch.
Its a bug for a stable package to depend on an ~arch package, so the ~arch package you would like cannot be stabilsed until all its ~arch dependencies are stabilised too.
You find out about that sort of thing when you try to emerge a testing package and portage tells you that you need two different versions of the same dependency installed. One to support something stable, already installed, the other to support the testing package you would like to install.
You can run an all ~arch system. It mostly works most of the time.
Post back if you are considering that. Its not something to be undertaken lightly as there is no way back to stable short of a reinstall.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Wed May 16, 2018 5:58 pm Post subject: |
|
|
| khayyam wrote: | | but with the above you only get those packages that you've explictly unmasked. |
| mv wrote: | | emerge -a (which you should always use) or also eix will show you anyway which packages from the overlay are/would be used, so this is somewhat superfluous work. |
mv ... I disagree, the time taken to repeatedly review, and then --exclude, is probably greater than the time taken to mask/unmask once. With the above I know what will happen, it's basically fire & forget.
| mv wrote: | | But what is more important: This way you will usually miss if the overlay maintainer masks a package for some reason. This might even be critical if that reason is related to security considerations. |
If masked then it's not selected, or emerge will complain about the mask.
best ... khay |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Thu May 17, 2018 12:55 pm Post subject: |
|
|
| khayyam wrote: | | mv ... I disagree, the time taken to repeatedly review, and then --exclude, is probably greater than the time taken to mask/unmask once. |
The problem is that packages in an overlay are usually not independent. For instance, app-shells/schily-tools::mv contains some files/libraries which are in a different version in app-cdr/cdrtools::gentoo. So only unmasking app-shells/schily-tools is not necessarily the "right" thing. (This is a somewhat bad example since here the problem was solved with USE-dependencies which are not in cdrtools::gentoo. However, adding an artificial USE-flag is not always the sane approach. Note that ::overlay deps are not supported (which is IMHO correct, but can lead to problems with your approach)).
| Quote: | | If masked then it's not selected, or emerge will complain about the mask. |
No, it will not: If you masked the overlay globally and then unmask a single package, then this unmask will override any mask of that package (or of a version of it) of the overlay maintainer. The problem here is that the package might not be masked once you added it, but was masked later (or a certain masked version was added later). It is exactly the "fire and forget" which can fall on your feet here. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu May 17, 2018 2:51 pm Post subject: |
|
|
| khayyam wrote: | | mv ... I disagree, the time taken to repeatedly review, and then --exclude, is probably greater than the time taken to mask/unmask once. |
| mv wrote: | | The problem is that packages in an overlay are usually not independent. For instance, app-shells/schily-tools::mv contains some files/libraries which are in a different version in app-cdr/cdrtools::gentoo. So only unmasking app-shells/schily-tools is not necessarily the "right" thing. (This is a somewhat bad example since here the problem was solved with USE-dependencies which are not in cdrtools::gentoo. However, adding an artificial USE-flag is not always the sane approach. Note that ::overlay deps are not supported (which is IMHO correct, but can lead to problems with your approach)). |
mv ... well, that "problem" is no different to the one involved with using repos per se, and while you might conceivably run into such issues, the mask/unmask isn't the underlying cause. So, what is the greater "problem", encountering the issue as you present it, or having packages from overlays selected in preference to ::gentoo? I would argue the later, because what you tend to find in most overlays is one or two packages specific to that overlay, and then a whole slew of packages from ::gentoo, mostly updates waiting merger upstream. I'm not saying an overlay mask is perfect, but it does prevent the bigger problem as it applies to me, and without my having to constantly look see what packages get selected on --update, and take action against them being installed.
| khayyam wrote: | | If masked then it's not selected, or emerge will complain about the mask. |
| mv wrote: | | No, it will not: If you masked the overlay globally and then unmask a single package, then this unmask will override any mask of that package (or of a version of it) of the overlay maintainer. The problem here is that the package might not be masked once you added it, but was masked later (or a certain masked version was added later). It is exactly the "fire and forget" which can fall on your feet here. |
You're right, my bad, but couldn't this argument be applied to package.unmask in general, is it "safe" to use package.unmask at all given this concern? Further, with the level of trust involved in using overlays in general who's to say that such masks are ever applied, or how much attention is given to such considerations, and multiply this by the x number of packages you might never use, but that could get selected, when any given overlay is in use. I have exactly 5 packages installed from as many overlays (and those overlays contain hundreds of packages) so I'm placing some level of trust in these maintainers, how many of these do you think I can safely say would package.mask a problem package? So, again, it's not perfect but the greater issue for me is having to constantly watch what is getting installed from where, and so its a trade off between that, and the possible, or less likely, situation where I'm open to the problem as you've stated it.
I should probably note that I started doing this prior to the introduction of repos.conf ... and I might be better served by adjusting priority wrt to ::gentoo ... so my suggestion may be more or less obsolete for that reason, and I should probably reconsider doing it this way.
EDIT: corrected attribution ... again :P
best ... khay
Last edited by khayyam on Thu May 17, 2018 4:32 pm; edited 1 time in total |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Thu May 17, 2018 3:47 pm Post subject: |
|
|
| khayyam wrote: | | I should probably note that I started doing this prior to the introduction of repos.conf ... and I might be better served by adjusting priority wrt to ::gentoo ... so my suggestion may be more or less obsolete for that reason, and I should probably reconsider doing it this way. |
Man, then you get into underlays, which I like for introducing ebuilds, but it's a bit trickier to reason about the dependencies.
Personally, I prefer your approach, because I'm not interested in taking much from overlays; specific packages only, which I can then watch for in emerge -pv output.
I can see the issue if the overlay maintainer masks her own package, but I'd hope she'd also remove the versions from her overlay when doing so (or wtf is going on?) at which point equal-versioned Gentoo packages will be preferred.
And bear in mind, wrt the best-visible calculation, later-versioned (and not masked) takes precedence over repo ordering.
So, in summary, I can see the technical issue, and while it might be nice to have that addressed, it seems like a much less usual corner-case that could only ever show up with a badly-maintained overlay. And we'd have much more chance of spotting it, with khay's approach.
Obviously, general approaches don't fit every situation, and there might well be a specific overlay which we just use as-is. Only one I can think of where I'd even consider it would be mv. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Thu May 17, 2018 6:33 pm Post subject: |
|
|
| steveL wrote: | | I can see the issue if the overlay maintainer masks her own package, but I'd hope she'd also remove the versions from her overlay when doing so (or wtf is going on?) at which point equal-versioned Gentoo packages will be preferred. |
$REPO/profile/package.mask refers only to the versions from $REPO. (Inconsistentely, $MAIN_REPO/profile/package.mask refers to all repositories.)
Reasons for masking in an overlay are the same as for masking in the gentoo repository: It might be for very experimental testing, a known breakage in a certain setting, a known unresolved security issue, or dead upstream (hence unknown security state). Of course, opinions might differ whether some of these justifies removal of the package. My opinion in general is to keep packages as long as possible: A masked package does not hurt anybody. Of course, if the mask should not be seen by the majority of users, this opinion becomes problematic. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri May 18, 2018 12:18 am Post subject: |
|
|
| mv wrote: | $REPO/profile/package.mask refers only to the versions from $REPO. (Inconsistentely, $MAIN_REPO/profile/package.mask refers to all repositories.)
Reasons for masking in an overlay are the same as for masking in the gentoo repository: It might be for very experimental testing, a known breakage in a certain setting, a known unresolved security issue, or dead upstream (hence unknown security state). Of course, opinions might differ whether some of these justifies removal of the package. My opinion in general is to keep packages as long as possible: A masked package does not hurt anybody. Of course, if the mask should not be seen by the majority of users, this opinion becomes problematic. |
Hard-masked for testing is on a new ebuild; the others are all cases for removal, especially in an overlay.
And yes, hard-masked ebuilds are going to be an issue, if the user is unmasking the package from overlay.
I think overlay maintainers must expect that users are not going to allow every package on to their machine; it's far more common to want to use only a few.
For instance, there is no way I'd want every package from pro-audio to land on my machine.
WRT development overlays like gentoo-haskell, where users tend to use the IRC channel, I still wouldn't want everything.
I'm not saying everything's perfect; there's the inconsistency you mention, and the wider issue of overlay masks, where the user has the overlay itself masked, and is unmasking certain packages. I'd certainly like to see both addressed, the latter with more priority.
Not sure "main profile" masks are that big an issue -- certainly there needs to be a base distro tree, and the user can unmask -- but I'd be interested to hear a better idea.
Hard-masks in overlay I think should require a specific =versioned unmask; again, I'd be interested to hear what others think can or should (not) be done. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri May 18, 2018 4:36 am Post subject: |
|
|
| steveL wrote: | | the others are all cases for removal |
I do not agree here as I already never agree with the trigger-happy mentality in the gentoo repository. For instance, there are always valid reasons why users might want to keep certain packages which are no longer developed upstream. Also known security bugs are not necessarily an issue: It always depends on the environment in which a package is used.
Edit: For the gentoo repository, I can understand the problem if nobody maintains the ebuild anymore and if eventually it is not compiling anymore or depends on packages no longer in the tree, has a deprecated EAPI or something similar. But as long as the overlay maintainer agrees to continue maintaining the ebuild such that it can be used after unmasking (with possibly known restrictions), I see no reason for removal. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri May 18, 2018 6:34 pm Post subject: |
|
|
| steveL wrote: | | the others are all cases for removal |
| mv wrote: | I do not agree here as I already never agree with the trigger-happy mentality in the gentoo repository. For instance, there are always valid reasons why users might want to keep certain packages which are no longer developed upstream. Also known security bugs are not necessarily an issue: It always depends on the environment in which a package is used.
For the gentoo repository, I can understand the problem if nobody maintains the ebuild anymore and if eventually it is not compiling anymore or depends on packages no longer in the tree, has a deprecated EAPI or something similar. But as long as the overlay maintainer agrees to continue maintaining the ebuild such that it can be used after unmasking (with possibly known restrictions), I see no reason for removal. |
Heh, I agree with you about not removing in the Gentoo tree (and keeping usage possible), but not for hard-masked packages in overlays, which are supposed to work alongside the base distro tree.
Certainly not so long as there isn't a specific mechanism, such as requiring a specific versioned unmask for hard-masked packages from overlay, to deal with the situation:
An overlay maintainer MUST expect that users are going to mask the overlay by default, and selectively unmask specific packages (which will then take priority over equal-versioned pkgs in main tree.)
In fact, I'd personally prefer that as a default (in the mangler) simply because quality-control is so variable with overlays, by definition.
Wrt keeping ebuilds, a user can always shove the ebuild into local overlay, once it's hard-masked upstream; at which point they're far more aware of what's happening, and can take responsibility for the specific package.
But yeah, I agree older ebuilds should not be ditched just because there's a newer version out; it's especially annoying when it's going up an ABI series.
I think that applies even more to the main tree, than overlays; and that it is (otherwise) important to keep the distinction in mind.
Known breakage and known security problems are not what we need from overlays; they should go in the git repo/attic, afaic.
I'd make a possible exception for dead upstreams, where the overlay maintainer is in effect maintaining the package (because it never needs much code to change.) And if known breakage is only under a specific USE-flag (which should then just be removed from IUSE.)
But hey, that's what overlays are about: you're trusting the maintainer not to mess up your system with inadequate care. Which brings us back to how most people mask the overlay, and only use ebuilds they need and (hopefully) have reviewed, certainly on first usage.
Anything else would be a reckless approach, and bad advice to give, since the standard line is that in using overlays, people are on their own. (Gentoo has no control over a random website or repo.)
I love the pro-audio crowd, but there is no way that I will ever use that overlay without a default blanket mask.
About the only one I can think of where I'd lift the default restriction, is the haskell overlay: principally because any snafus cannot affect my @system set; also because I trust the team, based on lurking in their IRC channel for a couple of years, and knowing the actual language channels and the discipline they instil (because they're mathematicians.)
Feels like there's another approach that doesn't get bogged down in masks, and the implications of unmasking; just disabling overlays by default, and enabling certain packages (or the repo) in the same way as unmask, that applies earlier on. |
|
| Back to top |
|
|
arnvidr
l33t
Joined: 19 Aug 2004
Posts: 629
Location: Oslo, Norway
|
| Posted: Sun May 27, 2018 1:13 pm Post subject: |
|
|
Just curious what parts of the pro-audio overlay makes you bring them up as something you don't completely trust more than once? I ran that for almost 10 years on my previous computer, without much trouble. Any special packages they often broke for you?
_________________
|
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Mon May 28, 2018 4:39 pm Post subject: |
|
|
| arnvidr wrote: | | Just curious what parts of the pro-audio overlay makes you bring them up as something you don't completely trust more than once? I ran that for almost 10 years on my previous computer, without much trouble. Any special packages they often broke for you? |
God, I haven't looked at it for years, because I've been too caught up in other work to spend weeks playing with audio (which is not an indication of broken software, but a misspent youth. ;)
There was no issue of "more than once" where I got burnt; as usual I reviewed the overlay ebuilds and too many of them were gunk purely in terms of bash, for me to trust them. (Implementation language must be learnt well enough that practitioners don't barf; or YDIW, and doing your users a disservice, imo.)
I also spent a few years passively reading the list, and time and again basic mistakes (with real end-user consequences) were coming up; yes, they were being corrected, but it meant I never felt comfortable enough to layman install it.
Masking in overlay wasn't a big thing back then; I don't think we even had directory masks at that point. So I accept that time has moved on, and there's been a steady of influx of new developers actually from the community.
Point being, it's good to hear that you've done well with it.
Current project is over this year (whether we're ready or not;) so I'm planning some downtime with audio soonish, and will take another look. (I am still going to mask the overlay by default, mind.. ;)
Thanks for the feedback on your experience with pro-audio. |
|
| Back to top |
|
|
arnvidr
l33t
Joined: 19 Aug 2004
Posts: 629
Location: Oslo, Norway
|
| Posted: Tue May 29, 2018 8:21 am Post subject: |
|
|
No problem.
It seems most audio packages I used before is now in the main repo, as I don't have anything installed from the (now named) audio-overlay. I still have it synced for some reason, might be able to get rid of that completely without impacting my audio stuff.
_________________
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
