View previous topic :: View next topic |
Author |
Message |
its1louder Tux's lil' helper
Joined: 03 Jul 2003 Posts: 75 Location: Santa Barbara CA
|
Posted: Thu Nov 20, 2014 8:18 pm Post subject: strategy for installing python packages not in portage |
|
|
Hi there, I figure this must be a pretty common problem and I think there is no right answer, just a bunch of opinions with various pros and cons. I'd like to solicit some of those opinions right now. Here's what I can come up with.
So I have gentoo systems with the science overlay, but there are still some scientific programming libraries not available as ebuilds, eg, seaborn, mpld3 latest versions of Bokeh etc. I can think of a couple ways to deal with this.
One is to make my own local overlay and make my own ebuild. Pro: I could be a good citizen and pass the ebuild on to science overlay or someone. Con: I don't know how to write ebuilds and learning looks hard.
two is that I could install pip or something and get it that way. Pro: easy to use package manager that I am familiar with because I use it on Mac & windows machines. Con: seems like portage and pip would step on each others toes maintaining parallel dependencies. A variation on this is that i might build some sort of sandbox to keep a pip/python installation in I'd probably have to learn about virtualenv or something then and that sounds overly complicated.
three is I can just download source by hand and hand install it. Pro: This is what I do and it seems to work OK. Con: I guess its not as elegant at maintaining dependencies.
so ... Am I doing it wrong? _________________ These go to eleven. |
|
Back to top |
|
|
Hu Moderator
Joined: 06 Mar 2007 Posts: 21633
|
Posted: Fri Nov 21, 2014 2:53 am Post subject: |
|
|
Yes, you are doing it wrong. Always install via the system package manager, even if that means writing an ebuild for it first. If you know how to install it by hand and the package is well behaved, writing an ebuild for it is usually quite simple. If you need help writing one, various posters here can critique your ebuild and some may even walk you through writing an ebuild based on the instructions for a manual installation. I suggest posting the identifying data and installation instructions for a package you need, so that someone can walk you through writing an ebuild that will manage it for you. You can then use that as a reference for future ebuilds. |
|
Back to top |
|
|
dataking Apprentice
Joined: 20 Apr 2005 Posts: 251
|
Posted: Fri Nov 21, 2014 7:43 am Post subject: |
|
|
Hu wrote: | Yes, you are doing it wrong. Always install via the system package manager, even if that means writing an ebuild for it first. |
I partially disagree*. Since you asked for opinions, I'm simply offering mine.
Emerge dev-python/pip. eix (search) for the portage package(s) you want. Search multiple times with different variations of the python package name. If you still can't find it, use pip to install the package. pip is a package manager in and of itself.
Python dependencies aren't that well resolved in portage, for some reason. At least not that I've experienced -- here's an example: I recently tried out the package "tuned", which has several python requirements. The "tuned" package installed fine, but the daemon failed to start because of certain missing (pre-)requirements. Emerged the latest versions of pygobject and dbus-python(?) and the daemon starts just fine. (No pip involved.) Whereas I had an occasion to use maltrieve, on another distro, and all dependencies had to be installed with pip.
Bottom line: use portage if you can.....exhaustively. Otherwise, use pip.
* Hu generally has great advice. I don't want to detract from his comments, but sometimes there are easier ways, or at least "paths of least resistance". _________________ -= the D@7@k|n& =- |
|
Back to top |
|
|
jauhien n00b
Joined: 27 Sep 2012 Posts: 11 Location: Cracow, Poland
|
Posted: Fri Nov 21, 2014 10:48 am Post subject: |
|
|
You can use gs-pypi (https://github.com/jauhien/gs-pypi) available in the tree. Just emerge it, and you will see new pypi repo in the layman list of overlays. You have two options now: generate ebuilds for all the stuff in pypi (rather lots of them), or add only necessary ebuilds to /etc/g-sorcery/g-sorcery.cfg (see link that I have given). After you add pypi overlay with layman, ebuilds will be generated and they will be updated every time you sync this overlay with layman. |
|
Back to top |
|
|
ayvango Tux's lil' helper
Joined: 08 Feb 2012 Posts: 118
|
Posted: Fri Nov 21, 2014 11:30 am Post subject: |
|
|
I'm afraid of writing a python ebuild. Python packages use heavy magic with python targets, single target, use_python variable and so on. And even more python can have native c insertions and I'm not aware how to deal with them when using uncommon runtime such as pypy or jython. It is too easy to make an error in a python package ebuild and too hard to test all configurations. So I leave working with python to system maintainers and assume that they provide all needed system-wide libraries.
All other python libraries I install locally for my user. I allocate two folders for it: /home/ayvango/local and /home/ayvango/venv |
|
Back to top |
|
|
its1louder Tux's lil' helper
Joined: 03 Jul 2003 Posts: 75 Location: Santa Barbara CA
|
Posted: Fri Nov 21, 2014 8:32 pm Post subject: |
|
|
Thanks for the opinions and ideas, its just what I'm looking for. I like the idea of adding my tarball downloaded and built libraries to some sort of "local" hierarchy and then putting that in PYTHONPATH, something like /usr/local/lib/python2.7/site-packages/. that gets me going quick and dirty without too much fuss or breakage. Also a good way to make my personal python modules available system-wide, if its not something I plan to distribute it doesn't need an ebuild.
I know I should be a nice guy and give back and make an ebuild but I just perused the writing ebuilds page and my eyes glazed over halfway through EAPI=3. I don't think I am going to be an ebuild developer when I grow up but gs-pypi looks like it could be a reasonable crutch (if it works). I'll give it a try.
THX! _________________ These go to eleven. |
|
Back to top |
|
|
Hu Moderator
Joined: 06 Mar 2007 Posts: 21633
|
Posted: Fri Nov 21, 2014 11:48 pm Post subject: |
|
|
dataking wrote: | Emerge dev-python/pip. eix (search) for the portage package(s) you want. Search multiple times with different variations of the python package name. If you still can't find it, use pip to install the package. pip is a package manager in and of itself. | True, he can use pip. However, any post advocating pip is incomplete without the obligatory warning that running pip as root on Gentoo will, under some circumstances, trash the Python indirection wrapper and turn all Python scripts into Pip. Running pip as an unprivileged user is fine if you are comfortable with their weaker security model. |
|
Back to top |
|
|
Faraclas Tux's lil' helper
Joined: 08 Dec 2014 Posts: 117
|
Posted: Sun Dec 21, 2014 9:10 pm Post subject: |
|
|
This is a good discussion and I need to do the same thing. Specifically, I need to install pyvisa and I can't seem to find it in portage or any of the overlays. I just emerged pip but now am worrying about using it.
Another library that would work in this case is vxi-11 Code: | https://pypi.python.org/pypi/python-vxi11 |
Can someone share an ebuild that works for something off of the pypi site? It would be a great template and then we can all make ebuilds for wnatever we need and figure out a way to share them. |
|
Back to top |
|
|
Faraclas Tux's lil' helper
Joined: 08 Dec 2014 Posts: 117
|
Posted: Sun Dec 21, 2014 9:26 pm Post subject: |
|
|
I think the method will lie somewhere here:
Code: | g-pypi - Create ebuilds from PyPI automatically
g-pypi is a tool for installing Python packages using information from PyPI without having to write an ebuild by hand. You can read more about the tool here | http://g-pypi.readthedocs.org/en/latest/ |
|
Back to top |
|
|
Faraclas Tux's lil' helper
Joined: 08 Dec 2014 Posts: 117
|
Posted: Sun Dec 21, 2014 11:58 pm Post subject: |
|
|
Confirmed.
Use this method: http://g-pypi.readthedocs.org/en/latest/userguide.html#installation
I was successfully able to install pyvisa and its dependencies using portage, a local overlay, and the ebuilds that g-pypi created.
One correction to the document though. It states to do this:
Code: | # sudo emerge -av gpypi |
when actually you should do this:
Code: | # sudo emerge -av g-pypi |
|
|
Back to top |
|
|
its1louder Tux's lil' helper
Joined: 03 Jul 2003 Posts: 75 Location: Santa Barbara CA
|
Posted: Thu May 07, 2015 11:24 pm Post subject: |
|
|
OK I just made my first ebuild for a package called pySAL. Even as simple and basic as my ebuild was, and with all the blatant copy-pasta from the netCDF-python ebuild, I still found it excruciating. A couple notes:
* test doesn't seem to work.
* should probably add example and doc options but that sounds even harder
* I don't really run gentoo its a funtoo system. Not sure how this makes the ebuild differ.
If there is a way to pitch this back to the community I am all ears. Here it is:
https://github.com/its1louder/funtoo-its1louder/blob/master/dev-python/pySAL/pySAL-1.9.1.ebuild
I'll probably do a similarly rough and ready version for geopandas and descartes soon. _________________ These go to eleven. |
|
Back to top |
|
|
dol-sen Retired Dev
Joined: 30 Jun 2002 Posts: 2805 Location: Richmond, BC, Canada
|
Posted: Fri May 08, 2015 12:51 pm Post subject: |
|
|
I hate to say it, but g-pypi is not really maintained/developed anymore.
app-portage/g-sorcery and it's app-portage/gs-pypi backend is what you want to use. _________________ Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch... |
|
Back to top |
|
|
its1louder Tux's lil' helper
Joined: 03 Jul 2003 Posts: 75 Location: Santa Barbara CA
|
Posted: Fri Nov 20, 2015 11:11 pm Post subject: |
|
|
Its not necromancy if its auto-necrotic.
Just for completeness and posterity here is what I have finally settled with. It will be unsatisfying to hardliners who can't countenance any software that didn't "emerge". But for me it strikes a good balance between ideological purity and Git-R-Dun.
Basically, I try to rely on portage and keep it up to date. If I need the latest package that hasn't got an ebuild I first try rolling my own in my local overlay. I set the egg timer and bang on it for up to an hour. If it won't build at the end of that I give up and go to plan B.
Plan B is, I start a new env for my user with miniconda. I was using pyenv/virtualenv/pip for a while and I still do because its what I had already set up one server. Actually p/v/p might be closer to gentoo because it compiles from source and miniconda just downloads sorts binaries.
here are the advantages of miniconda for my workflow:
- get all of your requirements installed and working in like 10 minutes*
- emphasizes up to date scientific computing packages which are the same ebuilds that tend to be not so fresh in portage and even science overlay
- plays well with pip for packages that are not in the conda repositories
- really slick management of environments so you can handle packages with conflicting requirements without getting confused
The main disadvantage is that I don't see an option to switch to the native python environment on the machine. Since my first choice is to use all the system wide python for whatever project, it would have to be under a different user than one that can do projects with conda dependent environments. That or I have to go in and manually handle my $PATH variable which is less slick.
The best case scenario would be if something like g-pypi worked, but it didn't happen for me before the egg timer went off. So until then, this is my workaround.
* this is a BFG. I've had projects stall for weeks while I try to figure out how to install a needed library without an ebuild. _________________ These go to eleven. |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Sat Nov 21, 2015 2:14 pm Post subject: |
|
|
its1louder wrote: | The best case scenario would be if something like g-pypi worked, but it didn't happen for me before the egg timer |
Did you look at "app-portage/g-sorcery and its app-portage/gs-pypi backend"? Sounds like it'd be worth getting that running reliably for everyone.
I'll just echo what Hu said: Never run pip as root.
I've lost count of the number of threads where people have fubared their system doing that (eg: the most recent.) |
|
Back to top |
|
|
|