Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Let Others Manage Java and Python in Gentoo
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
tld
Veteran
Veteran


Joined: 09 Dec 2003
Posts: 1380

PostPosted: Tue Jan 01, 2019 3:07 pm    Post subject: Re: Gentoo, please step back Reply with quote

pjp wrote:
Putting "better" aside, does any distro do this well?
This was the point I made in my first reply to this thread, and no they don't. It's always sucked when you needed a perl/python/java/whatever module that you're distribution doesn't happen to have. However just as Hu pointed out, it's also impossible to have a dependable system wide package manager if there's some other external package manager shoe horning stuff into the system.

Tom
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17773

PostPosted: Tue Jan 01, 2019 5:24 pm    Post subject: Reply with quote

I didn't think so, but wasn't sure. Given nokilli's other comments, I thought maybe he had some insight

As for conflicting package managers, it seems possible to have a separate area. So, hypothetically, pip, npm, emacs stuff go into /usr/local or maybe /opt and nowhere else. Of course, that then requires each of those items behaves well, which more or less requires them to be sandboxed. Which starts looking like externally managed magic black boxes, such as linking live production code to jquery (I believe it was they who performed an upgrade which broke a lot of people's webshlock, but I can't find a reference).

I've been surprised a specification which generically defined requirements hasn't "taken over." Upstream publishes their requirements, downstream then produce mostly the same relative results.
_________________
I honestly think you ought to sit down calmly, take a stress pill, and think things over.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5595

PostPosted: Tue Jan 01, 2019 6:43 pm    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
Arch Linux is looking to be my main development system owing to pacman being written in c. You have no idea how good that sounds right now.

Sounds almost as good as Paludis. Good luck, you'll need it.
Back to top
View user's profile Send private message
nokilli
Apprentice
Apprentice


Joined: 25 Feb 2004
Posts: 195

PostPosted: Tue Jan 01, 2019 6:58 pm    Post subject: Re: Gentoo, please step back Reply with quote

pjp wrote:
nokilli wrote:
But Python… man, let me tell you it’s been nothing but a trail of tears developing using Python under Gentoo over the last nearly 20 years.
I thought Python support had been handled well. Is this just a matter of not having the latest and greatest available in the tree?

That's a different issue. Recent changes have made it nearly a non-issue, but in the past it could get tricky.

No, I'm referring to pip. Running pip under Python produces an admonition to run specifying the option --user, which is a Gentoo creation and that forces newly installed packages into the user's home directory. Looks for documentation on --user in the docs, finds nothing. Only documented on the wiki page as far as I can tell.

The problem comes when you use some other tool that relies on Python. mod_wsgi for instance, which will let you tell it which version of Python (via path) to use, but which doesn't let you get any more elaborate than that and so the packages installed in the user's directory are missed. Ok, so we can do this via venv, and if we have other tools written in Python we can be careful to make sure everybody is pointed at the same version and so I spend the day on it and it seems to work. For now. My understanding is that this situation doesn't happen on something like Arch Linux because it has no dependency on Python at all, which means, I'm free to set up a single Python installation under Arch and do with it as I please and there's no special workarounds or elaborate hacks necessary, no hard-coding of paths, no custom shebangs, no config file nonsense, it just works because there's only one Python.

Python comes out with 3.7! Everybody else shouts hurray! I sort of grimace and am actually relieved to work through the release notes and see there's nothing there I want to use because if there was I know I'd have to go through the SLOT dance and yes it is working (or was for 3.6 when release was 3.5, 3.5 for 3.4) but it feels like every time I do this I'm risking upsetting portage and then there goes the day trying to figure out why. Or maybe this time it's python-exec that gets annoyed. Happened before. Are all three Jython developers happy with python-exec? Good.

It may be that my use case is more elegantly addressed with using something like Docker or a chroot or even an overlay. Work that I probably would do at some point down the road anyways. I've sort of taken to using qemu for that kind of stuff, but of course, on these guests lives portage because I put Gentoo on there. Docker would let me have just /usr/bin/python without portage I'm thinking. This might be a better way forward for me, I don't know.

Rewriting portage in c++ and sqlite? Imagine being able to do emerge -C python. Priceless. Probably belongs in a new thread.
_________________
Today is the first day of the rest of your Gentoo installation.
Back to top
View user's profile Send private message
nokilli
Apprentice
Apprentice


Joined: 25 Feb 2004
Posts: 195

PostPosted: Tue Jan 01, 2019 7:00 pm    Post subject: Re: Gentoo, please step back Reply with quote

Ant P. wrote:
nokilli wrote:
Arch Linux is looking to be my main development system owing to pacman being written in c. You have no idea how good that sounds right now.

Sounds almost as good as Paludis. Good luck, you'll need it.

Because nobody reports having a good experience with Arch Linux.
_________________
Today is the first day of the rest of your Gentoo installation.
Back to top
View user's profile Send private message
nokilli
Apprentice
Apprentice


Joined: 25 Feb 2004
Posts: 195

PostPosted: Tue Jan 01, 2019 7:10 pm    Post subject: Re: Gentoo, please step back Reply with quote

tld wrote:
pjp wrote:
Putting "better" aside, does any distro do this well?
This was the point I made in my first reply to this thread, and no they don't. It's always sucked when you needed a perl/python/java/whatever module that you're distribution doesn't happen to have. However just as Hu pointed out, it's also impossible to have a dependable system wide package manager if there's some other external package manager shoe horning stuff into the system.

Even if it's being shoe horned into a known location like site-packages, site-lisp or node_modules?

I find this very hard to believe.

It sounds to me like you guys are mixing up how a package gets installed with whether a package should be installed. Of course, undoubtedly I have benefited from this countless times and not even known it; I even now only understand a fraction of what is going on inside my box.

But there needs to be a way for me to say that for this set of packages, I'm willing to take the risk. The safe default, but with the option of user override. Like ~amd64 for instance, or the -9999 ebuilds, the latter which grabs software directly from a git repository, yes? Gentoo is all about choices, including the choice to hose your own system.

And then there's the question of whether all of this effort needs to be duplicated in an ebuild. And what it would mean for Gentoo to see its developers suddenly relieved of a burden they were never really in a position to properly bear in the first place.

Less is more?
_________________
Today is the first day of the rest of your Gentoo installation.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2546

PostPosted: Tue Jan 01, 2019 7:11 pm    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
Ant P. wrote:
nokilli wrote:
Arch Linux is looking to be my main development system owing to pacman being written in c. You have no idea how good that sounds right now.

Sounds almost as good as Paludis. Good luck, you'll need it.

Because nobody reports having a good experience with Arch Linux.
Because Arch routinely breaks during upgrades. For all its flaws, Gentoo's testing branch is far more stable than Arch's main (and only) branch. Instability in Gentoo seems to be nearly 100% user error.

I literally had an arch update destroy the installation and I had to start over from scratch. If using Arch floats your boat, go for it. But don't be shocked to learn it isn't perfect.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5595

PostPosted: Tue Jan 01, 2019 8:09 pm    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
Rewriting portage in c++ and sqlite? Imagine being able to do emerge -C python. Priceless. Probably belongs in a new thread.

No need to waste a thread on it, it's been tried already and we know the results. Paludis was removed from the tree actually because it was unmaintainable and about 10x slower than portage after being around for 10 years.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6276

PostPosted: Tue Jan 01, 2019 8:47 pm    Post subject: Re: Gentoo, please step back Reply with quote

Ant P. wrote:
Paludis was removed from the tree actually because it was unmaintainable and about 10x slower than portage

No. It was removed because it was unmaintained: The reasons were mainly social, not technical. Certainly they were not related to the implementation language.
Concerning speed: This is exclusively the fault of some paludis concepts. You can be sure that it would be much slower if the same concepts would have been written in python instead of C++.

Actually, having portage being implemented in C++ would speed it up quite a lot and would also have other advantages (no breakage if a python lib or one of its dependencies breaks). There were some plans to pass at least the dependency resolver (the slowest NP-complete part) to some SAT-solver library (which could be written in C++ or C or some other fast language), but I don't know what has become of this. The current portage resolver is apparently a mess with lots of known bugs and hackish workarounds which probably nobody fully understands.

Concerning python: I think that it was a huge mistake to make more than 2 python slots (2.* and 3.*) since the incompatibilities within python 3.* are negligible and should be handled like minor incompatibilities in all other packages - without slotting.
Anyway, giving control away from the package manager, you end exactly here.

The same holds for all the other languages which bring their own "package managements": perl, TeX(live), emacs, ...

These "internal" packagers are useful only for bleeding-edge developers who need the latest version of all packages immediately on their machines, no matter whether it breaks things or even their whole system. Proper packaging just costs time and goes much at a much slower pace. Of course, tools to convert such language's internal hackish package management to ebuilds are useful, though usually the generated ebuilds need to be polished manually. Anyway, for a sane user experience there is no way around installation through portage (i.e. through ebuilds) (on a gentoo system).

BTW: If it is really true that arch has no python2 it means that quite a lot of packages will not run (fully) under arch. An example which come to mind immediately is fvwm-crystal.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2546

PostPosted: Tue Jan 01, 2019 8:53 pm    Post subject: Reply with quote

I would argue there is absolutely no reason one could not write a portage replacement in any language of your choice.

However, a thread here would be absolutely pointless. If you really want to see it open the development tool of your choice and start writing.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42596
Location: 56N 3W

PostPosted: Tue Jan 01, 2019 9:03 pm    Post subject: Reply with quote

No need to start from scratch.
There is pkgcore.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17773

PostPosted: Tue Jan 01, 2019 11:34 pm    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
It sounds to me like you guys are mixing up how a package gets installed with whether a package should be installed.

nokilli wrote:
But there needs to be a way for me to say that for this set of packages, I'm willing to take the risk.
So you want to hide something from Portage and at the same time you still expect Portage to know what you did behind its back?

Then what happens with pip needs to be predictable so that Portage can be informed that it needs to inspect system changes made by pip. So my next question is, what does native pip do with stuff when it "installs" something?
_________________
I honestly think you ought to sit down calmly, take a stress pill, and think things over.
Back to top
View user's profile Send private message
Dr.Willy
Guru
Guru


Joined: 15 Jul 2007
Posts: 470
Location: NRW, Germany

PostPosted: Tue Jan 01, 2019 11:47 pm    Post subject: Reply with quote

NeddySeagoon wrote:
No need to start from scratch.
There is pkgcore.

Which does little to solve the "package manager depends on python"-problem. :P
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2516

PostPosted: Wed Jan 02, 2019 2:08 am    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
Ant P. wrote:
nokilli wrote:
Arch Linux is looking to be my main development system owing to pacman being written in c. You have no idea how good that sounds right now.

Sounds almost as good as Paludis. Good luck, you'll need it.

Because nobody reports having a good experience with Arch Linux.


In my limited experience with Arch, nobody reports having a good experience because Arch's forum is, by direction, only for people who have problems. Not for chatting about an idea, or any other random discussion. To me this discourages reporting of any positive results.

While every time I mention this some people seem to jump on me and deny it, I find Arch's forum to be inhospitable. Which is why I don't run Arch. I installed it once, got my forum ID and then after lurking awhile I installed Gentoo on that box. Come to think of it, the very fact that you can't get a forum id until you've installed Arch tells you everything you need to know about their community.

Arch, as many of you probably know, has documentation second to none. I use it all the time. The distro clearly has lots of experts. In my opinion though, there's more to a good distro than having a bunch of experts.
Back to top
View user's profile Send private message
Goverp
l33t
l33t


Joined: 07 Mar 2007
Posts: 639

PostPosted: Wed Jan 02, 2019 9:02 am    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
[...
Because nobody reports having a good experience with Arch Linux.
I'll report having a good experience with Arch.

(It's my go-to distro for Raspberry Pi. (I do have a Gentoo setup too, but only on one card; the other 5 or so are all Arch). Arch seems to me to break no more often than Gentoo - though there's a lot more to break on my Gentoo AMD64 boxes).
_________________
Greybeard
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2516

PostPosted: Wed Jan 02, 2019 3:33 pm    Post subject: Re: Gentoo, please step back Reply with quote

Goverp wrote:
nokilli wrote:
[...
Because nobody reports having a good experience with Arch Linux.
I'll report having a good experience with Arch.

(It's my go-to distro for Raspberry Pi. (I do have a Gentoo setup too, but only on one card; the other 5 or so are all Arch). Arch seems to me to break no more often than Gentoo - though there's a lot more to break on my Gentoo AMD64 boxes).


But isn't it curious that you see more people reporting a good experience with Arch on non-Arch forums than you see on Arch forums? Monitoring Arch forums is like waiting for the pin to drop.
Back to top
View user's profile Send private message
nokilli
Apprentice
Apprentice


Joined: 25 Feb 2004
Posts: 195

PostPosted: Wed Jan 02, 2019 5:54 pm    Post subject: Re: Gentoo, please step back Reply with quote

pjp wrote:
nokilli wrote:
It sounds to me like you guys are mixing up how a package gets installed with whether a package should be installed.

nokilli wrote:
But there needs to be a way for me to say that for this set of packages, I'm willing to take the risk.
So you want to hide something from Portage and at the same time you still expect Portage to know what you did behind its back?

I don't see why portage needs to know. The package lives in the site-packages directory and it was installed by explicit user request.

Python... yeah, it can get out there and do stuff. But look, you guys have the same concern with emacs! I count 203 emacs packages in app-emacs vs. 211 in ELPA. WTF! These packages contain only lisp files that execute in the editor and have little in the way of troublemaking opportunities for the system. And the editor is always(?) run under the user account. Nevertheless, Gentoo decides that no, it needs to be there to duplicate the effort of ELPA which is maintained by of all people the gnu guys!

What are we afraid is going to happen here?

Python is more complicated I admit owing to the possibility of c extensions and of course it's ability to do almost anything on the system by design. A bad Python program can be very bad, granted. But the principle is still the same. Gentoo embraced Python for a reason. Python has its own foundation and appear to be very responsible in the work that they put out. And at some point the user needs to be trusted with his own system and do things behind portage's back.

pjp wrote:
Then what happens with pip needs to be predictable so that Portage can be informed that it needs to inspect system changes made by pip. So my next question is, what does native pip do with stuff when it "installs" something?

It's python code that it goes into the site-packages directory packaged as a module. And then the user is free to import the module into his own code at will. Maybe there are packages that somehow go beyond that but every time I've added something to Python that is how it works. I have to explicitly install it, then I have to explicitly call it.

That's also been my experience with node.js. Yes there was a time when they were trying to integrate more deeply into the system but now they just create their own directory and get the user to prepend their bin directory to PATH. node.js lives in my home directory with the name 'node'. I do a rm -rf node and it's gone, all of it.

For me that's the test. Does this stuff go into a known location that can be easily deleted when the containing environment is removed? If the answer is yes then as a purely package management issue there doesn't seem to be a problem. Why worry about doing a rm -rf /usr/lib64/python3.6/site-packages/whizbang3000 when a rm -rf /usr/lib64/python3.6/site-packages/ will delete it just the same?

Whether or not these packages are safe is an entirely different question, and of course Gentoo should provide a safe environment by default, but to work at protecting the user from himself? This is not why I run Gentoo. And it's not why you guys run it either.
_________________
Today is the first day of the rest of your Gentoo installation.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17773

PostPosted: Wed Jan 02, 2019 11:04 pm    Post subject: Re: Gentoo, please step back Reply with quote

nokilli wrote:
I don't see why portage needs to know. The package lives in the site-packages directory and it was installed by explicit user request.
Using your mod_wsgi example, I thought you were referring to managing python yourself, without Portage. And then at the same time, you expected Portage to install mod_wsgi and magically know about the python you installed. If that's not what you meant, and you would still expect Portage to install whatever python mod_wsgi needed, then that makes more sense. At which point, you should be using /usr/local to "take the risk" and burden of manually maintaining non system packages. If neither of those, then I just don't understand what you're trying to accomplish.

nokilli wrote:
It's python code that it goes into the site-packages directory packaged as a module.
As long as that site-packages is in either /usr/local or /opt, then fine. Otherwise, not on any system I'll be using.

nokilli wrote:
That's also been my experience with node.js. Yes there was a time when they were trying to integrate more deeply into the system but now they just create their own directory and get the user to prepend their bin directory to PATH. node.js lives in my home directory with the name 'node'.
I'm not opposed to relying on upstream to do their package management thing, but it must be segregated from everything else. /usr/local/lib/{npm,python} etc.

nokilli wrote:
but to work at protecting the user from himself? This is not why I run Gentoo.
Nothing is stopping you from manually managing software outside of Portage. But if you conflict with what Portage does, the mess is yours to clean up.

nokilli wrote:
And it's not why you guys run it either.
I run with very few changes from a "normal" install. The only off thing I can think of is blocking Python 3.7.
_________________
I honestly think you ought to sit down calmly, take a stress pill, and think things over.
Back to top
View user's profile Send private message
Syl20
Guru
Guru


Joined: 04 Aug 2005
Posts: 553
Location: France

PostPosted: Thu Jan 03, 2019 4:16 pm    Post subject: Reply with quote

pjp wrote:
As Anaconda is an installer, it doesn't have packages, so "Anaconda packages" appears to refer to RPMs, which are decidedly not distro independent. Or is there another Anaconda?

Wikipedia says : "Not to be confused with Anaconda (Python distribution)."
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17773

PostPosted: Fri Jan 04, 2019 12:59 am    Post subject: Reply with quote

Thanks, obviously missed that (and never heard of it). Wonderful name choice.
_________________
I honestly think you ought to sit down calmly, take a stress pill, and think things over.
Back to top
View user's profile Send private message
ervin.peters
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2003
Posts: 108
Location: Weimar, Germany

PostPosted: Mon Apr 15, 2019 6:34 am    Post subject: hint: distributed responsibility... universal PM interfaces Reply with quote

To be a good package manager of the system, it has to be responsible for some core aspects: consistency of dependencies, distinguishing between user and system, making current releases available, tracking installation files and defining locations.

Let's call the good package manager for the system 'rootPM', and let's call the domainspecific package Managers 'assistantPM'.

To make them collaborate, which is necessary for any aspect of stable systems, they need to have a defined responsibility and the need to have a defined interface to communicate with the rootPM, not only the root user, about dependencies, location etc., the frame conditions in which it can operate and how it should inform the rootPM.

That comes to the idea of a distributed hierarchical package managing, could be implemented as a plugin architecture.

Is someone thinking in that direction? Is there a community working on this? Are there defined Interfaces ready to use?

I assume every package manager community is suffering at this point, but has to accept that it might be better to delegate some parts of responsibility and force over the system.

just my 3ct.

Ervin

P.S. Im suffering from the bad relations between rootPM and the domain specific managers: g-cpan, pip, npm,...
...trying to upgrade conan leads to write a ton of local python ebuilds...
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5595

PostPosted: Mon Apr 15, 2019 12:50 pm    Post subject: Reply with quote

Gentoo currently has 435 accessible overlays via eselect-repository, and 60937 packages between them. I'd say the delegation part already works fine.
Back to top
View user's profile Send private message
ervin.peters
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2003
Posts: 108
Location: Weimar, Germany

PostPosted: Mon Apr 15, 2019 5:22 pm    Post subject: qfindoverlay? Reply with quote

Ant P. wrote:
Gentoo currently has 435 accessible overlays via eselect-repository, and 60937 packages between them. I'd say the delegation part already works fine.


Your kidding.

Maybe you didn't get the point: I meant some similar package manager Inferfaces for all Systems, since pip and kind of this or widely OS-independent.

And the overlay approach has some drawback, the stable tree in main portage went older and older, some parts are prehistoric and bzw. where to find an overlay proposing the right package?

Time for a qfindoverlay?

Or ome emerge hints: The dependencies could be fullfilled by overlay xy. Should It be configured?

That's one job of a packageManager: To know about the packages a user might want or might need.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5595

PostPosted: Mon Apr 15, 2019 5:58 pm    Post subject: Re: qfindoverlay? Reply with quote

ervin.peters wrote:
where to find an overlay proposing the right package?

eix -R
Back to top
View user's profile Send private message
ervin.peters
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2003
Posts: 108
Location: Weimar, Germany

PostPosted: Wed Apr 17, 2019 6:38 am    Post subject: Re: qfindoverlay? Reply with quote

Ant P. wrote:
ervin.peters wrote:
where to find an overlay proposing the right package?

eix -R


does not work. To find an aktual ebuild vor conan (1.14.3) in the wild world, I tried

$ eix -R conan

gives an error: Datenbank-Datei /var/cache/eix/remote.eix kann nicht geöffnet werden.
Wurde sie mit „eix-remote add1/update1“ erzeugt?

$ eix-remote add1
* Datei /var/cache/eix/remote.tar.bz2 nicht lesbar

It seems all searchable overlays have to be configured before, that is not really sufficient on 435 Overlays.

Just to clarify, a description of the process:

1. Need a software package

2. Like to find that package in main portage for quality reasons

3. If not, would like to know which overlays / assistant package manager provide my needed package an the dependencies

4. Would like to try to install package and dependencies, checking errors, providing fixes

And the other point is still the os independent handling of dependencies by pip, cpan, npm, et al.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum