build 32bit stuff on mostly-stable amd64 without emul-linux

[mmogilvi]

abi_x86_32 is now stable!

I will probably stop maintaining this post after this update.

Steps 1, 2, and 4 through 6 are mostly obsolete now. Kept mostly just for historical reference.

For the transition, the list of packages that need abi_x86 in step 3 (using limited wildcards) may still be a useful compromise between enabling it globally, and needing to set it individually on something like 200 packages (as the "eselect news" release suggests).

Also, until Qt 4.8.6 is stabilized, you might prefer to unmask emul-linux-x86-qtlibs (in /etc/package.unmask) if you want to install some out-of-tree or unstable package (like skype) that depends on 32 bit Qt. See also the "Qt Exception" note in step 1.

OLD: Problem/Background

It is now very practical to remove emul-linux libraries in favor of directly enabling 32 bit builds of source packages instead, even on a "stable" system. There are currently only three issues blocking official stabilization in bug 525254, none of which I currently need myself.

My solution, detailed further below, involves uninstalling emul-linux-x86-* packages and using natively compiled 32-bit packages instead.

OLD: Other Possible 32-bit Solutions: (which I haven't tested)

Although the native 32-bit builds is the way of the future (hopefully any day now...), in the short term it isn't officially supported, and so here are some alternative short-term strategies:

A 32-bit chroot gives you the maximum flexibility in configuring what you want, but requires extra effort to setup, use, and maintain. See https://www.gentoo.org/proj/en/base/amd64/howtos/chroot.xml

As of 2014-05-12, the official way to get a fully-functional, mostly stable system working is still to leave ABI_X86=-32 masked/disabled for most packages (except actual applications like wine), and do a world update to pick up the latest versions of the emul-linux packages. Although be aware that this will probably be a relatively short-term solution only: it could be masked any week [or day?] now.

Another alternative I haven't tried is to switch to a mostly unstable ~amd64 install with ABI_X86=32 globally enabled. This might still install emul-linux libraries, but only for the parts that can't be supplied by individual package ebuilds. I think this may be where most developers focus their efforts. Since development occurs here, it may break random things (including 64-bit stuff) moderately often, but any one break will likely be fixed quickly. See https://wiki.gentoo.org/wiki/Multilib_System_without_emul-linux_Packages for detailed instructions.

Or see comment 28 of bug 499952 to try a (probably fully obsolete) hybrid/combined native (where possible) and emul-linux (to fill in the gaps) solution that installs more unstable packages than my solution, but fewer than the full-on ~amd64 install. But I expect its list of ~amd64 packages is probably out of date. There are similar lists elsewhere, but they are probably also out of date.

OLD: Some related links:

http://unix.stackexchange.com/questions/89678/using-abi-x86-in-gentoo
The best explanation of the ABI_X86 stuff I've found.

http://wiki.gentoo.org/wiki/Project:Multilib
Relatively new project page for the multilib conversion effort. Roadmap, how to adapt ebuilds, etc.

https://wiki.gentoo.org/wiki/Multilib_System_without_emul-linux_Packages
How to eliminate emul-linux when running an unstable (~amd64) system. Lists abi_x86_32 packages individually (no wildcards), which has pluses and minuses. May eventually replace this page, once abi_x86_32 is unmasked for stable.

http://wiki.gentoo.org/wiki/Multilib/gx86-multilib
Very brief description of how to enable abi_x86_32. Should probably be merged with above.

https://wiki.gentoo.org/wiki/Multilib_porting_status
During the main portion of the multilib conversion effort, this was the main status page. But these days it is mostly obsolete.

--------

My Solution

The following steps should allow you to install wine without emul-linux packages. If you don't care about wine, it can still be used as a proxy for "can build most 32-bit software, with or without portage".

One line summary: My solution is to completely uninstall emul-linux-x86-* packages, enable ABI_X86=32 fairly widely but selectively, and (main gotcha:) disable anything that still has missing dependencies in this configuration.

---
Step 1: Find and disable anything that still requires emul-linux-x86*:

The number of unstable packages that need unmasked ~amd64 explodes (below) if you allow anything to pull in emul-linux-x86-baselibs with abi_x86_32 enabled.

Main gotcha: If you need something that still requires emul-linux*, then you obviously can't remove them, and this solution is not for you. But see some of the "optional"s below, or other solutions above. Qt exception: emul-linux-x86-qtlibs-20140508-r1 (only) can now be installed without the rest of the emul-linux packages, if you enable abi_x86_32 and ~amd64 in appropriate native dependencies, and stop using a wildcard in step 5 below. This should be an acceptable alternative solution until qt multilib is stabilized (bug 530238). I don't think any other emul-linux packages have dependencies setup to allow this.

Make sure the emul-linux-* packages aren't in /var/lib/portage/world. Use "emerge --deselect emul-linux-WHATEVER".

Use "equery d emul-linux-WHATEVER" to see what else may depend on them. The dependencies it prints are relatively useless; go look at the original ebuild (as well as other versions) to see if they can be satisfied with multilib packages instead. Then either uninstall such packages, or disable use flags for individual packages (if the dependency is conditional on use flags).

Recently I only need to disable a couple of use flags on my machine:

[pinion]

I followed your post, and I'm not sure how to get around this... trying to emerge -avuDN world

[mmogilvi]

I'm not sure, but if I am interpreting the output correctly, it looks like abi_x86_32 is somehow not enabled for zlib.

Does "emerge -pv zlib" provide any useful information? Especially regarding the ABI_X86=32 flag it is trying to use?

The /etc/portage/profile/use.mask change (step 2) should have unmasked it, and the sys-libs/* abi_x86_32 line in package.use (step 3) should have enabled it for zlib.

----

Also, in today's update I had to add =media-libs/libvorbis-1.3.3-r1 ~amd64 to the package.keywords file, because a newly-stabilized libsndfile depends on it when abi_x86_32 is enabled. I'll edit the original post appropriately.

[pinion]

Yeah, I'm not sure why it isn't working correctly for me. Perhaps I am missing something...

[TomWij]

[mmogilvi]

This should be /etc/portage/profile/use.mask, not /etc/portage/use.mask:

[pinion]

Thanks.. that was it. I had the use.mask file in the wrong location.

[pinion]

Here is where I am currently at; not sure how to remove those hard blocks in this case. If I add emul-linux-x86* abi_x86_32 to package.use, it removes 41 hard blocks, but I'm not sure if that's the best way to go. My thought was to emerge those packages with abi_x86_32 in order to more easily replace them with the new packages since it will not let me just unemerge them.

Currently:
http://pastebin.com/RQjET16C

With emul-linux-x86* abi_x86_32:
http://pastebin.com/LBka2ztq

Thanks

[mmogilvi]

General Notes:

My going-in goal was to uninstall all emul-linux-x86* packages completely. If you have anything that still requires them when abi_x86_32 is enabled (they lack alternative dependency options that use individual packages with abi_x86_32), your options include:

• Uninstall whatever it is, if you can do without it.
  
• Use package.keyword to upgrade to an unstable version that doesn't require emul-linux-x86*. (Possible if you can find a version that can use individual packages with abi_x86_32 instead of emul-linux, using "alternative option" dependencies.)
  
• Disable use flags for that package, if the dependency is conditional on particular flags. I used this option for stuff I'm using, via the flags in Step 1.
  
• Go ahead and pull in emul-linux-x86 after all. If you go with this option, then you are outside the scope of what I've tried myself. However, some tips for installing emul-linux with abi_x86_32 enabled:
  
  
  
  • You need to enable abi_x86_32 basically everywhere: At least in emul-linux-x86 packages, and in ALL individual packages they depend on.
    
  • Note that emul-linux-x86 packages can basically be installed two ways: Either everything WITHOUT abi_x86_32, or everything WITH abi_x86_32. Mixing and matching abi_x86_32 settings isn't allowed by emul-linux-x86 dependency rules and build scripts.
    
  • I think the list of packages that need to be put into packages.keywords (Step 4) is much larger when installing emul-linux-x86 with abi_x86_32 enabled, because emul-linux-x86 itself depends on them (even though they aren't needed much) and abi_x86_32 aware versions haven't been stabilized.
    
  • I was rather careful building the list in Step 4: Instead of using the latest version that emerge recommended, I usually looked at the availabe .ebuild files and found the smallest upgrade from stable that supported abi_x86_32.
    

On my machine, I think I uninstalled some of the emul-linu-x86 packages manually before my big update, while some of the others were automatically unstalled by the big update.

Specific Ideas:

Make sure you've masked out the emul-linux packages (Step 5).

Try "grep emul-linux /var/lib/portage/world". If found, remove them from the world set with "emerge --deselect emul-linux-WHATEVER", and/or completely uninstall them with "emerge -Cva emul-linux-WHATEVER". (Note: Uninstalling them will likely break 32-bit programs until you get the big update working, or until you reinstall them, which might require commenting out a lot of configuration file changes...)

Find out what packages may depend on them using "equery d emul-linux-x86-baselibs" or similar for the other emul-linux packages. On my machine, it only lists wine-1.6.2, grub-0.97-r12, adobe-flash-11.2.202.341, and libSM-1.2.2. It also seems to try to indicate the conditions when each of those actually depend on emul-linux-x86-baselibs, but it looks like it oversimplified the dependencies to the point of uselessness.

I often find it useful to look at actual .ebuild files to really understand what their dependencies are.

[pinion]

Thanks for the help.

I identified about 4 programs that required emul-linux-x86* libs explicitly, and 4 more that had an or condition for them that I removed anyway to get rid of the hard blocks. Right now, I have half the emul-linux-x86* libs removed, and am doing an update to remove the rest.. I hope

So.. I know it might be messy, but for the time being I can create an overlay that pulls in the individual libraries for the programs I need that require emul-linux-x86 stuff, right? For instance, net-misc/teamviewer has:

[steveL]

[megaflow]

what about qt apps like skype.

[mmogilvi]

@megaflow: According to https://wiki.gentoo.org/wiki/Multilib_porting_status (or manual inspection of ebuilds), there hasn't been any progress in adding native support for abi_x86_32 to Qt. I'm sure new ebuilds contributed to bugzilla would be appreciated, though.

@pinion: Any help you want to provide would be appreciated. However, I suspect you may be slightly misinterpreting those dependencies. The "||" applies to the two parts inside the following parentheses, either the list of x11-libs packages OR the emul-linux-x86-xlibs. Which means that emul-linux-x86-xlibs should already be taken care of, and any improvements need to focus on baselibs and soundlibs.

It is possible that the dependencies are out of date, and you could simply discover and add some OR dependencies for baselibs and soundlibs, and it would just work without any more substantive changes. If you have an executable, "ldd" can tell you what shared libraries it is trying to load. Or start with the x86 section as a guide...

Also, that whole section appears to be conditional on the system-wine use flag being disabled. Perhaps you could just enable "system-wine" instead?

Regarding a VM, you could always run a 32 bit gentoo in a VM instead of windows. Or you don't even need a VM: You could do it with a 32-bit chroot, as described here: https://www.gentoo.org/proj/en/base/amd64/howtos/chroot.xml

[pinion]

I thought I may have misinterpreted the OR statement.. thanks for clarifying that.

I like the 32bit chroot idea.. I think that may the road I want to go to get some of these programs running in the meantime.

I have used overlays a ton, and maintained simple ones.. so once I refresh my knowledge on setting it up, I will post a link to it.

Thanks!

[XQZS]

Some questions:
1. Why use wine-1.6.2? In this thread, you say wine-1.6.1 depends on lcms:0, but why not use wine-1.7 or other latest ~amd64 version?
2. grub:2 have no "ncurses" USE flag, so why not use grub2?
3. If my system is stable now, just no ~amd64 packages in it, then I want to install wine, could I just emerge ~amd64 packages as new slots, preserve and not update the stable version?

[mmogilvi]

Some Answers:

1. It should work with any version of wine larger than 1.6.1. But with these instructions I try to stay as close to the stabilized version as I can, for both wine and other packages. And with the way the package.keywords entries are done, as soon as a newer version is stabilized, the new version will be installed instead, and the old version entry will essentially be ignored.

2. grub2 didn't exist when I first setup many of my machines, and the upgrade isn't trivial (new config file format, etc). I intend to use grub2 on new machines (I've already experimented with it some on a USB drive), but don't want to mess with older ones. If you are already using grub2, you should continue using it. Also, having a stray grub:0 entry in your files won't touch grub:2.

3. The only one of the 6 packages that need ~amd64 that is slotted at all is lcms, and both the multilib (~amd64) version and the main stable (amd64) version install into SLOT=2, so they can't be installed at the same time. One or the other is probably also true of all of the other stable packages that need to be rebuilt with AMD_X86=32 enabled.


Alternative:

Although I haven't tried it recently, if you want a completely stable system, the simplest technique for now is probably to use the default configuration (NOT all the changes I describe in my various steps), run emerge wine, let it pick up emul-linux packages, etc, as I outline as the first of my "Other Possible Solutions" near the top. There have been various issues with this option this year (hence my instructions as a somewhat involved but hopefully future-proof alternative), but I think it is probably working currently. One thing to watch out for is enabling AMD_X86=32 for native packages that are also supplied by emul-linux; doing so can cause confusing conflict messages.

[SeeksTheMoon]

Great posting, OP!
I tried it out myself because my RadeonSI performance deeply sucked: mesa, drivers, kernel were bleeding edge, but emul-linux-opengl was so old that no games in wine worked at all.
Some years ago I compiled the mesa library and such in a 32bit chroot and replaced the emul libs with these but having a proper useflag is really elegant.
So I followed your example yesterday and had immediate success. The framerate went up from about 1fps-creepy-stuttering-cannot-even-leave-the-menu to really-playable-darn-good-looking-fast

To give you all something back, I wrote this article:
https://wiki.gentoo.org/wiki/Multilib_System_without_emul-linux_Packages

Feel free to improve it.

[archenroot]

This is great new. I tried already this once upon a time without success:-) ... I jumped today in my KDE dekstop system and was able to recompile everything with abi 32 use flag. The one remaining stuff is only proprietary nvidia-drivers...

Superior job Gentoo devs. Thank you.
_________________
Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

[thumper]

There are several references in this thread to AMD_X86.

Is that actually AMD_X86 or were they meant to be ABI_X86?

George