Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
strategy for installing python packages not in portage
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
its1louder
n00b
n00b


Joined: 03 Jul 2003
Posts: 68
Location: Santa Barbara CA

PostPosted: Thu Nov 20, 2014 8:18 pm    Post subject: strategy for installing python packages not in portage Reply with quote

Hi there, I figure this must be a pretty common problem and I think there is no right answer, just a bunch of opinions with various pros and cons. I'd like to solicit some of those opinions right now. Here's what I can come up with.

So I have gentoo systems with the science overlay, but there are still some scientific programming libraries not available as ebuilds, eg, seaborn, mpld3 latest versions of Bokeh etc. I can think of a couple ways to deal with this.

One is to make my own local overlay and make my own ebuild. Pro: I could be a good citizen and pass the ebuild on to science overlay or someone. Con: I don't know how to write ebuilds and learning looks hard.

two is that I could install pip or something and get it that way. Pro: easy to use package manager that I am familiar with because I use it on Mac & windows machines. Con: seems like portage and pip would step on each others toes maintaining parallel dependencies. A variation on this is that i might build some sort of sandbox to keep a pip/python installation in I'd probably have to learn about virtualenv or something then and that sounds overly complicated.

three is I can just download source by hand and hand install it. Pro: This is what I do and it seems to work OK. Con: I guess its not as elegant at maintaining dependencies.

so ... Am I doing it wrong?
_________________
These go to eleven.
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 9464

PostPosted: Fri Nov 21, 2014 2:53 am    Post subject: Reply with quote

Yes, you are doing it wrong. Always install via the system package manager, even if that means writing an ebuild for it first. If you know how to install it by hand and the package is well behaved, writing an ebuild for it is usually quite simple. If you need help writing one, various posters here can critique your ebuild and some may even walk you through writing an ebuild based on the instructions for a manual installation. I suggest posting the identifying data and installation instructions for a package you need, so that someone can walk you through writing an ebuild that will manage it for you. You can then use that as a reference for future ebuilds.
Back to top
View user's profile Send private message
dataking
Apprentice
Apprentice


Joined: 20 Apr 2005
Posts: 212

PostPosted: Fri Nov 21, 2014 7:43 am    Post subject: Reply with quote

Hu wrote:
Yes, you are doing it wrong. Always install via the system package manager, even if that means writing an ebuild for it first.

I partially disagree*. Since you asked for opinions, I'm simply offering mine. ;-)

Emerge dev-python/pip. eix (search) for the portage package(s) you want. Search multiple times with different variations of the python package name. If you still can't find it, use pip to install the package. pip is a package manager in and of itself.

Python dependencies aren't that well resolved in portage, for some reason. At least not that I've experienced -- here's an example: I recently tried out the package "tuned", which has several python requirements. The "tuned" package installed fine, but the daemon failed to start because of certain missing (pre-)requirements. Emerged the latest versions of pygobject and dbus-python(?) and the daemon starts just fine. (No pip involved.) Whereas I had an occasion to use maltrieve, on another distro, and all dependencies had to be installed with pip.

Bottom line: use portage if you can.....exhaustively. Otherwise, use pip.



* Hu generally has great advice. I don't want to detract from his comments, but sometimes there are easier ways, or at least "paths of least resistance". ;-)
_________________
-= the D@7@k|n& =-
Back to top
View user's profile Send private message
jauhien
n00b
n00b


Joined: 27 Sep 2012
Posts: 11
Location: Cracow, Poland

PostPosted: Fri Nov 21, 2014 10:48 am    Post subject: Reply with quote

You can use gs-pypi (https://github.com/jauhien/gs-pypi) available in the tree. Just emerge it, and you will see new pypi repo in the layman list of overlays. You have two options now: generate ebuilds for all the stuff in pypi (rather lots of them), or add only necessary ebuilds to /etc/g-sorcery/g-sorcery.cfg (see link that I have given). After you add pypi overlay with layman, ebuilds will be generated and they will be updated every time you sync this overlay with layman.
Back to top
View user's profile Send private message
ayvango
n00b
n00b


Joined: 08 Feb 2012
Posts: 74

PostPosted: Fri Nov 21, 2014 11:30 am    Post subject: Reply with quote

I'm afraid of writing a python ebuild. Python packages use heavy magic with python targets, single target, use_python variable and so on. And even more python can have native c insertions and I'm not aware how to deal with them when using uncommon runtime such as pypy or jython. It is too easy to make an error in a python package ebuild and too hard to test all configurations. So I leave working with python to system maintainers and assume that they provide all needed system-wide libraries.

All other python libraries I install locally for my user. I allocate two folders for it: /home/ayvango/local and /home/ayvango/venv
Back to top
View user's profile Send private message
its1louder
n00b
n00b


Joined: 03 Jul 2003
Posts: 68
Location: Santa Barbara CA

PostPosted: Fri Nov 21, 2014 8:32 pm    Post subject: Reply with quote

Thanks for the opinions and ideas, its just what I'm looking for. I like the idea of adding my tarball downloaded and built libraries to some sort of "local" hierarchy and then putting that in PYTHONPATH, something like /usr/local/lib/python2.7/site-packages/. that gets me going quick and dirty without too much fuss or breakage. Also a good way to make my personal python modules available system-wide, if its not something I plan to distribute it doesn't need an ebuild.

I know I should be a nice guy and give back and make an ebuild but I just perused the writing ebuilds page and my eyes glazed over halfway through EAPI=3. I don't think I am going to be an ebuild developer when I grow up but gs-pypi looks like it could be a reasonable crutch (if it works). I'll give it a try.

THX!
_________________
These go to eleven.
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 9464

PostPosted: Fri Nov 21, 2014 11:48 pm    Post subject: Reply with quote

dataking wrote:
Emerge dev-python/pip. eix (search) for the portage package(s) you want. Search multiple times with different variations of the python package name. If you still can't find it, use pip to install the package. pip is a package manager in and of itself.
True, he can use pip. However, any post advocating pip is incomplete without the obligatory warning that running pip as root on Gentoo will, under some circumstances, trash the Python indirection wrapper and turn all Python scripts into Pip. Running pip as an unprivileged user is fine if you are comfortable with their weaker security model.
Back to top
View user's profile Send private message
Faraclas
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2014
Posts: 80

PostPosted: Sun Dec 21, 2014 9:10 pm    Post subject: Reply with quote

This is a good discussion and I need to do the same thing. Specifically, I need to install pyvisa and I can't seem to find it in portage or any of the overlays. I just emerged pip but now am worrying about using it.

Another library that would work in this case is vxi-11
Code:
https://pypi.python.org/pypi/python-vxi11


Can someone share an ebuild that works for something off of the pypi site? It would be a great template and then we can all make ebuilds for wnatever we need and figure out a way to share them.
Back to top
View user's profile Send private message
Faraclas
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2014
Posts: 80

PostPosted: Sun Dec 21, 2014 9:26 pm    Post subject: Reply with quote

I think the method will lie somewhere here:

Code:
g-pypi - Create ebuilds from PyPI automatically

g-pypi is a tool for installing Python packages using information from PyPI without having to write an ebuild by hand. You can read more about the tool here
http://g-pypi.readthedocs.org/en/latest/
Back to top
View user's profile Send private message
Faraclas
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2014
Posts: 80

PostPosted: Sun Dec 21, 2014 11:58 pm    Post subject: Reply with quote

Confirmed.

Use this method: http://g-pypi.readthedocs.org/en/latest/userguide.html#installation

I was successfully able to install pyvisa and its dependencies using portage, a local overlay, and the ebuilds that g-pypi created.

One correction to the document though. It states to do this:

Code:
# sudo emerge -av gpypi


when actually you should do this:

Code:
# sudo emerge -av g-pypi
Back to top
View user's profile Send private message
its1louder
n00b
n00b


Joined: 03 Jul 2003
Posts: 68
Location: Santa Barbara CA

PostPosted: Thu May 07, 2015 11:24 pm    Post subject: Reply with quote

OK I just made my first ebuild for a package called pySAL. Even as simple and basic as my ebuild was, and with all the blatant copy-pasta from the netCDF-python ebuild, I still found it excruciating. A couple notes:

* test doesn't seem to work.
* should probably add example and doc options but that sounds even harder
* I don't really run gentoo its a funtoo system. Not sure how this makes the ebuild differ.

If there is a way to pitch this back to the community I am all ears. Here it is:
https://github.com/its1louder/funtoo-its1louder/blob/master/dev-python/pySAL/pySAL-1.9.1.ebuild

I'll probably do a similarly rough and ready version for geopandas and descartes soon.
_________________
These go to eleven.
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2702
Location: Richmond, BC, Canada

PostPosted: Fri May 08, 2015 12:51 pm    Post subject: Reply with quote

I hate to say it, but g-pypi is not really maintained/developed anymore.

app-portage/g-sorcery and it's app-portage/gs-pypi backend is what you want to use.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum