strategy for installing python packages not in portage

[its1louder]

Hi there, I figure this must be a pretty common problem and I think there is no right answer, just a bunch of opinions with various pros and cons. I'd like to solicit some of those opinions right now. Here's what I can come up with.

So I have gentoo systems with the science overlay, but there are still some scientific programming libraries not available as ebuilds, eg, seaborn, mpld3 latest versions of Bokeh etc. I can think of a couple ways to deal with this.

One is to make my own local overlay and make my own ebuild. Pro: I could be a good citizen and pass the ebuild on to science overlay or someone. Con: I don't know how to write ebuilds and learning looks hard.

two is that I could install pip or something and get it that way. Pro: easy to use package manager that I am familiar with because I use it on Mac & windows machines. Con: seems like portage and pip would step on each others toes maintaining parallel dependencies. A variation on this is that i might build some sort of sandbox to keep a pip/python installation in I'd probably have to learn about virtualenv or something then and that sounds overly complicated.

three is I can just download source by hand and hand install it. Pro: This is what I do and it seems to work OK. Con: I guess its not as elegant at maintaining dependencies.

so ... Am I doing it wrong?

[Hu]

Yes, you are doing it wrong. Always install via the system package manager, even if that means writing an ebuild for it first. If you know how to install it by hand and the package is well behaved, writing an ebuild for it is usually quite simple. If you need help writing one, various posters here can critique your ebuild and some may even walk you through writing an ebuild based on the instructions for a manual installation. I suggest posting the identifying data and installation instructions for a package you need, so that someone can walk you through writing an ebuild that will manage it for you. You can then use that as a reference for future ebuilds.

[dataking]

Yes, you are doing it wrong. Always install via the system package manager, even if that means writing an ebuild for it first.

I partially disagree*. Since you asked for opinions, I'm simply offering mine.

Emerge dev-python/pip. eix (search) for the portage package(s) you want. Search multiple times with different variations of the python package name. If you still can't find it, use pip to install the package. pip is a package manager in and of itself.

Python dependencies aren't that well resolved in portage, for some reason. At least not that I've experienced -- here's an example: I recently tried out the package "tuned", which has several python requirements. The "tuned" package installed fine, but the daemon failed to start because of certain missing (pre-)requirements. Emerged the latest versions of pygobject and dbus-python(?) and the daemon starts just fine. (No pip involved.) Whereas I had an occasion to use maltrieve, on another distro, and all dependencies had to be installed with pip.

Bottom line: use portage if you can.....exhaustively. Otherwise, use pip.



* Hu generally has great advice. I don't want to detract from his comments, but sometimes there are easier ways, or at least "paths of least resistance".

[jauhien]

You can use gs-pypi (https://github.com/jauhien/gs-pypi) available in the tree. Just emerge it, and you will see new pypi repo in the layman list of overlays. You have two options now: generate ebuilds for all the stuff in pypi (rather lots of them), or add only necessary ebuilds to /etc/g-sorcery/g-sorcery.cfg (see link that I have given). After you add pypi overlay with layman, ebuilds will be generated and they will be updated every time you sync this overlay with layman.

[ayvango]

I'm afraid of writing a python ebuild. Python packages use heavy magic with python targets, single target, use_python variable and so on. And even more python can have native c insertions and I'm not aware how to deal with them when using uncommon runtime such as pypy or jython. It is too easy to make an error in a python package ebuild and too hard to test all configurations. So I leave working with python to system maintainers and assume that they provide all needed system-wide libraries.

All other python libraries I install locally for my user. I allocate two folders for it: /home/ayvango/local and /home/ayvango/venv

[its1louder]

Thanks for the opinions and ideas, its just what I'm looking for. I like the idea of adding my tarball downloaded and built libraries to some sort of "local" hierarchy and then putting that in PYTHONPATH, something like /usr/local/lib/python2.7/site-packages/. that gets me going quick and dirty without too much fuss or breakage. Also a good way to make my personal python modules available system-wide, if its not something I plan to distribute it doesn't need an ebuild.

I know I should be a nice guy and give back and make an ebuild but I just perused the writing ebuilds page and my eyes glazed over halfway through EAPI=3. I don't think I am going to be an ebuild developer when I grow up but gs-pypi looks like it could be a reasonable crutch (if it works). I'll give it a try.

THX!

[Hu]

Emerge dev-python/pip. eix (search) for the portage package(s) you want. Search multiple times with different variations of the python package name. If you still can't find it, use pip to install the package. pip is a package manager in and of itself.

True, he can use pip. However, any post advocating pip is incomplete without the obligatory warning that running pip as root on Gentoo will, under some circumstances, trash the Python indirection wrapper and turn all Python scripts into Pip. Running pip as an unprivileged user is fine if you are comfortable with their weaker security model.

[Faraclas]

This is a good discussion and I need to do the same thing. Specifically, I need to install pyvisa and I can't seem to find it in portage or any of the overlays. I just emerged pip but now am worrying about using it.

Another library that would work in this case is vxi-11

Code: Select all

https://pypi.python.org/pypi/python-vxi11

Can someone share an ebuild that works for something off of the pypi site? It would be a great template and then we can all make ebuilds for wnatever we need and figure out a way to share them.

[Faraclas]

I think the method will lie somewhere here:

Code: Select all

g-pypi - Create ebuilds from PyPI automatically

g-pypi is a tool for installing Python packages using information from PyPI without having to write an ebuild by hand. You can read more about the tool here

http://g-pypi.readthedocs.org/en/latest/

[Faraclas]

Confirmed.

Use this method: http://g-pypi.readthedocs.org/en/latest ... stallation

I was successfully able to install pyvisa and its dependencies using portage, a local overlay, and the ebuilds that g-pypi created.

One correction to the document though. It states to do this:

Code: Select all

# sudo emerge -av gpypi 

when actually you should do this:

Code: Select all

# sudo emerge -av g-pypi

[its1louder]

OK I just made my first ebuild for a package called pySAL. Even as simple and basic as my ebuild was, and with all the blatant copy-pasta from the netCDF-python ebuild, I still found it excruciating. A couple notes:

* test doesn't seem to work.
* should probably add example and doc options but that sounds even harder
* I don't really run gentoo its a funtoo system. Not sure how this makes the ebuild differ.

If there is a way to pitch this back to the community I am all ears. Here it is:
https://github.com/its1louder/funtoo-it ... 9.1.ebuild

I'll probably do a similarly rough and ready version for geopandas and descartes soon.

[dol-sen]

I hate to say it, but g-pypi is not really maintained/developed anymore.

app-portage/g-sorcery and it's app-portage/gs-pypi backend is what you want to use.

[its1louder]

Its not necromancy if its auto-necrotic.

Just for completeness and posterity here is what I have finally settled with. It will be unsatisfying to hardliners who can't countenance any software that didn't "emerge". But for me it strikes a good balance between ideological purity and Git-R-Dun.

Basically, I try to rely on portage and keep it up to date. If I need the latest package that hasn't got an ebuild I first try rolling my own in my local overlay. I set the egg timer and bang on it for up to an hour. If it won't build at the end of that I give up and go to plan B.

Plan B is, I start a new env for my user with miniconda. I was using pyenv/virtualenv/pip for a while and I still do because its what I had already set up one server. Actually p/v/p might be closer to gentoo because it compiles from source and miniconda just downloads sorts binaries.

here are the advantages of miniconda for my workflow:

• get all of your requirements installed and working in like 10 minutes*
• emphasizes up to date scientific computing packages which are the same ebuilds that tend to be not so fresh in portage and even science overlay
• plays well with pip for packages that are not in the conda repositories
• really slick management of environments so you can handle packages with conflicting requirements without getting confused

The main disadvantage is that I don't see an option to switch to the native python environment on the machine. Since my first choice is to use all the system wide python for whatever project, it would have to be under a different user than one that can do projects with conda dependent environments. That or I have to go in and manually handle my $PATH variable which is less slick.

The best case scenario would be if something like g-pypi worked, but it didn't happen for me before the egg timer went off. So until then, this is my workaround.



* this is a BFG. I've had projects stall for weeks while I try to figure out how to install a needed library without an ebuild.

[steveL]

The best case scenario would be if something like g-pypi worked, but it didn't happen for me before the egg timer

Did you look at "app-portage/g-sorcery and its app-portage/gs-pypi backend"? Sounds like it'd be worth getting that running reliably for everyone.

I'll just echo what Hu said: Never run pip as root.

I've lost count of the number of threads where people have fubared their system doing that (eg: [topic=1006016]the most recent[/topic].)