Binary Gentoo project (tried)

[dE_logics]

Looks pretty -

Code: Select all

emerge -pv --getbinpkg xorg-server
!!! It seems that /proc is not mounted. You have been warned.
!!! PORTAGE_BINHOST unset, but use is requested.

These are the packages that would be merged, in order:

Calculating dependencies         ... done!       
[binary  N    ] x11-proto/xproto-7.0.17  [0]
[binary  N    ] x11-libs/xtrans-1.2.5  USE="-debug"  [0]
[binary  N    ] x11-proto/kbproto-1.0.4  [0]
[binary  N    ] x11-proto/xextproto-7.1.1  [0]
[binary  N    ] dev-libs/libpthread-stubs-0.1  USE="-debug"  [0]
[binary  N    ] x11-apps/rgb-1.0.3  USE="-debug"  [0]
[binary  N    ] x11-misc/xbitmaps-1.1.0  [0]
[binary  N    ] x11-libs/pixman-0.18.2  USE="mmx sse2 (-altivec) -static-libs"  [0]
[binary  N    ] x11-proto/recordproto-1.14  [0]
[binary  N    ] x11-proto/inputproto-2.0  [0]
[binary  N    ] sys-apps/dmidecode-2.10  [0]
[binary  N    ] sys-apps/miscfiles-1.4.2-r1  USE="minimal"  [0]
[binary  N    ] x11-proto/fontsproto-2.1.0  [0]
[binary  N    ] sys-libs/talloc-2.0.1-r1  USE="-compat -doc -static-libs -swig"  [0]
[binary  N    ] x11-proto/damageproto-1.2.0  [0]
[binary  N    ] x11-proto/xf86vidmodeproto-2.3  [0]
[binary  N    ] dev-libs/libusb-0.1.12-r5  USE="-debug -doc -nocxx"  [0]
[binary  N    ] sys-apps/eject-2.1.5-r2  USE="-nls"  [0]
[binary  N    ] x11-libs/libXau-1.0.5  USE="-static-libs"  [0]
[binary  N    ] x11-libs/libXdmcp-1.0.3  USE="-static-libs"  [0]
[binary  N    ] x11-libs/libICE-1.0.6  USE="-debug -ipv6"  [0]
[binary  N    ] virtual/libusb-0  [0]
[binary  N    ] x11-libs/libpciaccess-0.11.0  USE="minimal zlib -debug"  [0]
[binary  N    ] sys-apps/pciutils-3.1.4  USE="zlib -network-cron"  [0]
[binary  N    ] x11-libs/libfontenc-1.0.5  USE="-debug"  [0]
[binary  N    ] x11-libs/libdrm-2.4.21-r1  USE="-static-libs" VIDEO_CARDS="radeon -intel -nouveau -vmware"  [?]
[binary  N    ] x11-proto/fixesproto-4.1.1  [0]
[binary  N    ] virtual/eject-0  [0]
[binary  N    ] sys-apps/usbutils-0.86-r1  USE="zlib -network-cron"  [0]
[binary  N    ] x11-libs/libSM-1.1.1  USE="uuid -debug -ipv6"  [0]
[binary  N    ] x11-base/xorg-drivers-1.7  INPUT_DEVICES="evdev keyboard mouse synaptics -acecad -aiptek -fpit -joystick -penmount -tslib -virtualbox -vmmouse -void -wacom" VIDEO_CARDS="fbdev radeon vesa -apm -ark -ast -chips -cirrus -dummy -epson -fglrx (-geode) -glint -i128 (-i740) (-impact) -intel -mach64 -mga -neomagic (-newport) -nouveau -nv -nvidia -r128 -radeonhd -rendition -s3 -s3virge -savage -siliconmotion -sis -sisusb (-sunbw2) (-suncg14) (-suncg3) (-suncg6) (-sunffb) (-sunleo) (-suntcx) -tdfx -tga -trident -tseng -v4l -via -virtualbox -vmware (-voodoo) (-xgi)"  [0]
[binary  N    ] app-admin/eselect-opengl-1.1.1-r2  [0]
[binary  N    ] dev-libs/libgamin-0.1.10-r2  USE="python -debug"  [0]
[binary  N    ] app-admin/eselect-mesa-0.0.5  [?]
[binary  N    ] x11-libs/libxcb-1.6  USE="-doc (-selinux) -static-libs"  [0]
[binary  N    ] x11-libs/libX11-1.3.4  USE="xcb -doc -ipv6 -static-libs -test"  [0]
[binary  N    ] x11-libs/libXext-1.1.2  USE="-static-libs"  [0]
[binary  N    ] x11-libs/libxkbfile-1.0.6  USE="-debug"  [0]
[binary  N    ] x11-libs/libXt-1.0.8  USE="-static-libs"  [0]
[binary  N    ] media-libs/freetype-2.4.2  USE="X -auto-hinter -bindist -debug -doc -fontforge -utils"  [0]
[binary  N    ] x11-apps/iceauth-1.0.3  USE="-debug"  [0]
[binary  N    ] x11-libs/libXfixes-4.0.5  USE="-static-libs"  [0]
[binary  N    ] x11-libs/libXi-1.3  USE="-debug"  [0]
[binary  N    ] x11-libs/libXmu-1.0.5  USE="-debug -ipv6"  [0]
[binary  N    ] x11-apps/xkbcomp-1.1.1  USE="-debug"  [0]
[binary  N    ] sys-apps/dbus-1.2.24  USE="X -debug -doc (-selinux) -test"  [0]
[binary  N    ] x11-libs/libxkbui-1.0.2  USE="-debug"  [0]
[binary  N    ] x11-libs/libXres-1.0.4  USE="-debug"  [0]
[binary  N    ] x11-libs/libXfont-1.4.2  USE="-ipv6 -static-libs"  [0]
[binary  N    ] x11-libs/libXxf86vm-1.1.0  USE="-debug"  [0]
[binary  N    ] x11-libs/libXdamage-1.1.3  USE="-static-libs"  [0]
[binary  N    ] x11-apps/xauth-1.0.4  USE="-debug -ipv6"  [0]
[binary  N    ] x11-misc/xkeyboard-config-1.9  [0]
[binary  N    ] media-libs/mesa-9999  USE="classic gallium nptl xcb -debug -gles -llvm -motif -pic (-selinux)" VIDEO_CARDS="r300 radeon -i810 -i915 -i965 -intel -mach64 -mga -none -nouveau -r100 -r128 -r200 -r600 -radeonhd -savage -sis -tdfx -via -vmware"  [?]                                                                                  
[binary  N    ] x11-libs/libXtst-1.1.0  USE="-debug"  [0]
[binary  N    ] x11-apps/xrdb-1.0.6  USE="-debug"  [0]
[binary  N    ] x11-apps/xinit-1.2.0-r3  USE="minimal pam -debug"  [0]
[binary  N    ] app-shells/bash-completion-1.2  [0]
[binary  N    ] app-shells/gentoo-bashcomp-20090613  [0]
[binary  N    ] app-admin/gamin-0.1.10  [0]
[binary  N    ] dev-libs/glib-2.24.1-r1  USE="fam -debug -doc -hardened (-selinux) -xattr"  [0]
[binary  N    ] app-admin/gam-server-0.1.10  USE="-debug"  [0]
[binary  N    ] dev-libs/dbus-glib-0.86  USE="bash-completion -debug -doc -test"  [0]
[binary  N    ] sys-auth/policykit-0.9-r1  USE="bash-completion pam -doc (-selinux) -zsh-completion"  [0]
[binary  N    ] dev-libs/eggdbus-0.6  USE="largefile -debug -doc -test"  [0]
[binary  N    ] sys-auth/polkit-0.96-r1  USE="pam -debug -doc -examples -expat -nls"  [0]
[binary  N    ] sys-auth/consolekit-0.4.1  USE="pam policykit -debug -doc"  [0]
[binary  N    ] sys-apps/hal-0.5.14-r2  USE="X acpi apm consolekit policykit -crypt -debug -dell -disk-partition -doc -laptop (-selinux)"  [0]                                                                                      
[binary  N    ] app-misc/hal-info-20090716  [0]
[binary  N    ] x11-base/xorg-server-1.7.7-r1  USE="hal nptl sdl xorg -debug -dmx -ipv6 -kdrive -minimal -tslib"  [0]                                                                                                               
[binary  N    ] x11-drivers/xf86-video-fbdev-0.4.2  [0]
[binary  N    ] x11-drivers/xf86-input-evdev-2.4.0  USE="-debug"  [0]
[binary  N    ] x11-drivers/xf86-video-ati-6.13.1  [0]
[binary  N    ] x11-drivers/xf86-input-keyboard-1.4.0  USE="-debug"  [0]
[binary  N    ] x11-drivers/xf86-input-mouse-1.5.0  USE="-debug"  [0]
[binary  N    ] x11-drivers/xf86-video-vesa-2.3.0  USE="-debug"  [0]
[binary  N    ] x11-drivers/xf86-input-synaptics-1.2.1  USE="hal -debug"  [0]

Total: 77 packages (77 new, 77 binaries), Size of downloads: 0 kB
Portage tree and overlays:
 [0] /usr/portage
 [?] indicates that the source repository could not be determined

 * IMPORTANT: 2 news items need reading for repository 'gentoo'.
 * Use eselect news to read news items.

I've heard it has been discussed before, but I did not get such discussion.

So, first let's first take the advantages of the binary Gentoo project -

1) Easiest to maintain Binary Distro... Gentoo users will be capable enough to maintain the binary distribution and thus will involve very less people or less skilled people.

2) Fame. Gentoo is well known, if it starts such a binary project people will be VERY much interested and so we wont have much problems of fame like with Sabayon (discussed later); soon Gentoo community will expand major and even attract more devs since this this project will the first of it's kind. Speaking of which, there're already many package maintainers around (Gentoo users).

3) Larger community. In a single project, there will be 2 sets of users involved. Possibly this will make the largest Linux community, accelerating it's development.

4) Hybrid disto -- Gentoo can be both a source and binary disto (if using Generic make.conf); this will be first time in Linux history. So if one wants to compile cups with with dubbing support, he can add the debug use in package.use and compile instead of downloading the binaries.

5) Higher stability for Binary Gentoo. The packages will be tested by the community using the pure source Gentoo.

6) Higher reliability for Gentoo. Many of the bugs will be reported by the Binary Gentoo users and devs, furthermore any compile time failures realized by the Binary Gentoo team will be reported to bugzilla, saving compile time failures for Gentoo users.

7) The binary distro that will be made will be easier to debug since there's an underlying source distro on which it'll actually work. Changing USE will be a major help in debugging or updating the binaries to support more things to solve a bug/enhancement.

This will save time configuring and compiling the kernel for source users... there'll be a generic binary kernel available always.

9) The community wont be pouted. There will be different user groups of Binary Gentoo and Gentoo (even a different forum)... the Binary Gentoo users wont be that much of an expert so their bug reports will be inferior to the ones made by the source Gentoo users, so some Gentoo users will be 'qualified' by Gentoo foundation to review the bug reports and will be given right to close them if they are not valid. We can have a better validation system by using the experienced Gentoo community. This will save the tragedy of Ubuntu where there are no experienced users at all... as a result the community has turned to trash, you never get answers to real questions there + the packages are the most buggiest of any other distro...

10) The larger community will consists of experts and normal users so 2 the 2 can help each other... this will be a very healthy mix of users. The real questions will be migrated to Gentoo forums which the easy questions will be moved to the Binary Gentoo forums.

11) Known USE and CFLAGS -- The same advantage we have in Sabayon.

So how do you do it?

Generic make.conf and kernel -- Simple.

The maintainers will have the original Gentoo installed with make.conf and kernel for the binary Gentoo... it's just a matter of upgrading and quickpkg and a few other things; overall it's not that hard and time consuming.

Since non advanced users are incapable of configuring the system, there will be additions of new packages in portage (maybe in a new category app-config) who's sore purpose is to configure the installed main package (for e.g. autostart of KDM). For this the main package (for e.g. kdm) will have a new 'config' user flag which if enabled will merge app-config/kdm automatically which will configure the autostart. The config USE will be there by default in make.conf for Binary Gentoo to ensure that the newbie users don't get bewildered.

The portage tree will be a bit special, the prebuild binaries and the Portage tree that Binary Gentoo will get will be corresponding (i.e all the packages in the tree will have prebuild binaries and we will have PORTAGE_BINHOST on Gentoo server. --getbinpkg and --getbinpkgonly will be set as default.).

In case of an upgrade of libraries, revdep will suggest a recompilation, if so, the corresponding packages will be recompiled and a new binary will be made off it. We have to take care of the nomenclature though, same packages, different libraries against which they have been compiled.

Any modifications to portage?

Yes, in the hosting system we will have a different folder named Binary-Gentoo which PORTAGE_BINHOST will point to. With this the minimum requirements of the Binary Gentoo project will be complete IMO, after that portage will can include enhancements for binaries.... i.e. if the devs agree and don't fight among each other and make another Exerbo (is the spelling right?) or 5 more new package managers for the purpose.

We have Sabayon for this.

Apparently the Gentoo foundation don't give a damn about Sabayon, furthermore if the database format of portage changes... entropy will lag behind, there's always a problem of the API. Here I propose a single package managers -- portage and even a single client -- emerge to be both binary and source package manager so there wont be such problems + we wont have another new package manager involved to make things more confusing.

Ok so when are you starting the project

At best, I'm a good admin, from no where I'm a dev. You need to have moderate experience with coding to do this. Furthermore I don't even have the resources. This is more of a Gentoo foundation thing, if it has to be started Gentoo foundation has to start it... that will be the advantage. If another private project will be started off this, it'll be another Sabayon.

Any attempts you made?

I've installed Gentoo from the latest stage3 running kdebase-meta without a single package compilation using the same strategy as stated above. I've installed mplayer... lots of codecs and it all works well. Done in less than 0.5 hrs

If you want I can upload the whole FS compressed to a zpaq archive, but it's -march=native and uses my own make.conf and fstab which are VERY tweaked for my purpose.

[NeddySeagoon]

dE_logics,

I think some of your discussion is fatally flawed. Basically, Gentoo is not a distro at all. Its a set of tools that allows users to build their own distro.
As a result, no two Gentoo installs are the same.

Given that Gentoo is a toolkit, not a distro, the concept of binary-gentoo falls apart, as you would in effect, use the Gentoo toolkit to make a binary distro, that is not at all binary-Gentoo, as its what you (or binary-Gentoos devs) will decide to make it. Whatever that is, it won't be Gentoo.

[tomk]

Moved from Portage & Programming to Gentoo Chat as it's not a support request.

[Kollin]

dE_logics,

I think some of your discussion is fatally flawed. Basically, Gentoo is not a distro at all. Its a set of tools that allows users to build their own distro.
As a result, no two Gentoo installs are the same.

Given that Gentoo is a toolkit, not a distro, the concept of binary-gentoo falls apart, as you would in effect, use the Gentoo toolkit to make a binary distro, that is not at all binary-Gentoo, as its what you (or binary-Gentoos devs) will decide to make it. Whatever that is, it won't be Gentoo.

Sabayon?

[AllenJB]

(This post is written by someone who considers Gentoo to be a "distro", just a different kind of distro from every other distro he's ever encountered. While there may be things that make Gentoo a meta-distro, explaining this to everyone is just too time consuming, so IMO Gentoo is a "distro" because it just makes life easier to call it a distro).

So how do you do it?

Generic make.conf and kernel -- Simple.

The maintainers will have the original Gentoo installed with make.conf and kernel for the binary Gentoo... it's just a matter of upgrading and quickpkg and a few other things; overall it's not that hard and time consuming.

Since non advanced users are incapable of configuring the system, there will be additions of new packages in portage (maybe in a new category app-config) who's sore purpose is to configure the installed main package (for e.g. autostart of KDM). For this the main package (for e.g. kdm) will have a new 'config' user flag which if enabled will merge app-config/kdm automatically which will configure the autostart. The config USE will be there by default in make.conf for Binary Gentoo to ensure that the newbie users don't get bewildered.

The portage tree will be a bit special, the prebuild binaries and the Portage tree that Binary Gentoo will get will be corresponding (i.e all the packages in the tree will have prebuild binaries and we will have PORTAGE_BINHOST on Gentoo server. --getbinpkg and --getbinpkgonly will be set as default.).

It's not that simple. There can't be one make.conf for everyone - you can't take into account architectures (CHOST / CFLAGS), etc. You'd need a seperate BINHOST for each architecture you wish to support. You then have to produce all of those binaries in a timely manner, test them to make sure they work to at least some degree, host them and provide the bandwidth for them to be downloaded.

Then there's the issue of USE flags, which are a core part of Gentoo, in my opinion. As far as I can see, you're proposing that there's one set of USE flags that everyone uses and no one ever changes. This simply doesn't work.

To take an example, look at PHP. Some users will want the Oracle extensions, which require the commercially licensed Oracle packages, but not everyone will. How do you deal with this?

Another example: MySQL - Amarok (and other applications) use embedded MySQL, so some people will want that while others won't. Then you have people who use seperate machines for the server and client sides of MySQL (ie, some people will want minimal on some machines while not on others) - How do you deal with that?

Some people use Gentoo on a networked setup, with LDAP or Kerberos or some other network authentication method, while others use Gentoo on a single desktop / server and don't want networked authentication at all. Some people use a desktop environment and want all of the GTK / Qt / KDE / Gnome / etc frontends, while others use a desktop with only Qt/KDE or GTK/Gnome. Others use Gentoo on a headless or server environment and want no graphical frontends at all. How do you deal with that?

Oh, and one really really big thing you don't even mention in your whole plan: How do you deal with the concept of a rolling tree in a binary distro?

We have Sabayon for this.

Apparently the Gentoo foundation don't give a damn about Sabayon, furthermore if the database format of portage changes... entropy will lag behind, there's always a problem of the API. Here I propose a single package managers -- portage and even a single client -- emerge to be both binary and source package manager so there wont be such problems + we wont have another new package manager involved to make things more confusing.

The Gentoo Foundation / Council / developers don't care about Sabayon because it's not Gentoo. It's run be a completely different set of people with different aims and different methodologies. It uses Gentoo as a base, but only in the same way that Ubuntu used Debian as a base to launch from.

To clear up some terminology: The Gentoo Foundation specifically only exists to deal with the legal / money side of Gentoo. It's the Gentoo Council and the developers who run Gentoo as a distro. The Gentoo Foundation exists as nothing more than an entitity necessary to be able to take donations, bank accounts and deal with various other legal / procedural issues.

Ok so when are you starting the project

At best, I'm a good admin, from no where I'm a dev. You need to have moderate experience with coding to do this. Furthermore I don't even have the resources. This is more of a Gentoo foundation thing, if it has to be started Gentoo foundation has to start it... that will be the advantage. If another private project will be started off this, it'll be another Sabayon.

And here you hit on exactly another reason there's no such project. That aren't enough people interested in this, within or outside of the Gentoo developers. There is exactly one resource that is needed to get this project off the ground: interested people with enough time to devote to it.

This is unlikely to happen within the existing group of Gentoo developers because their time is taken up with existing tasks on the existing Gentoo. If they were interested in working on a bianry distro, they'd be working on a binary distro already. They use Gentoo because they like Gentoo, not some binary distro version of Gentoo (which would end up very similar to existing binary distros, so why not just go use one of them instead as they've already done a lot of the hard graft in getting a distro off the ground and have plenty of existing developer resources, methodologies, etc already established to maintain said distro in the long term)

So, first let's first take the advantages of the binary Gentoo project -

1) Easiest to maintain Binary Distro... Gentoo users will be capable enough to maintain the binary distribution and thus will involve very less people or less skilled people.

But only if you ignore:
- the package management system wasn't designed to be used as a binary distro package management system
- the concept of USE flags
- architectures
- CFLAGS / CXXFLAGS / LDFLAGS
- The ability to use different compilers

2) Fame. Gentoo is well known, if it starts such a binary project people will be VERY much interested and so we wont have much problems of fame like with Sabayon (discussed later); soon Gentoo community will expand major and even attract more devs since this this project will the first of it's kind. Speaking of which, there're already many package maintainers around (Gentoo users).

Gentoo is well known as a from-source distro. It's well known for its flexibility, power and customizability. All of the things you want to get rid of to try and shoe-horn it into being a binary distro. There are a large number fo distros which are already binary distros, whose package management systems are far better suited to being binary distros and who have the resources, system, methodologies and interested developers already in place. Why do you insist that Gentoo should try to be a binary distro instead of using one of them as a launch platform for your binary distro?

3) Larger community. In a single project, there will be 2 sets of users involved. Possibly this will make the largest Linux community, accelerating it's development.

Gentoo has a large community, but it is one that is interested in runnign and developing a powerful, flexible, customizable from-source distro, not a binary distro built using an unsuitable package management system.

4) Hybrid disto -- Gentoo can be both a source and binary disto (if using Generic make.conf); this will be first time in Linux history. So if one wants to compile cups with with dubbing support, he can add the debug use in package.use and compile instead of downloading the binaries.

But in trying to do this it will confuse potential users - you basically have 2 distros using the same name: One is a powerful, flexible, customizable from-source distro with a community of knowledgeable users, the other is a binary distro with no user or developer community trying to use a package management system not suited to the task and trying to piggy-back on the Gentoo name and reputation.

5) Higher stability for Binary Gentoo. The packages will be tested by the community using the pure source Gentoo.

By what community? The binary distro branch of Gentoo has no community. No users. And testing by users alone isn't enough. You need developers to build and test the packages initially. You complain about Ubuntu packages, but if Binary Gentoo doesn't even test packages to make sure they've compiled correctly, have the correct files present and at least launch and appear to work in some manner, it'll be even worse.

6) Higher reliability for Gentoo. Many of the bugs will be reported by the Binary Gentoo users and devs, furthermore any compile time failures realized by the Binary Gentoo team will be reported to bugzilla, saving compile time failures for Gentoo users.

Except that, given your description, the developers are now overwhelmed with bug reports from inexperienced users trying to use a froms-ource distro shoe-horned into being a binary distro which has no developer or user community to help them, so they have no time to fix the bugs that even the original community of the from-source distro were reporting because they're too busy testing the latest binary version of their package for the Binary Gentoo distro which has no users.

According to your description, you've now got the same number of develoeprs essentially trying to maintain two completely different distros. And you expect reliability to improve?

7) The binary distro that will be made will be easier to debug since there's an underlying source distro on which it'll actually work. Changing USE will be a major help in debugging or updating the binaries to support more things to solve a bug/enhancement.

Compeltely ignoring the fact that developers of every existing binary distro also compile the original packages from source and have the exact same tools to help them debug issues on their distro that's actually designed from the ground up to be a binary distro. You mention changing use flags here, but this counters your earlier proposal of a standard make.conf, so how exactl does thta work?

This will save time configuring and compiling the kernel for source users... there'll be a generic binary kernel available always.

Completely ignoring the fact that the kernel for every Gentoo install is completely different, and genkernel. How does your "one standard gentoo-sources binary kernel" deal with Xen guests? KVM? Modules that interfere with each other? Modules that crash some peoples machines but not others?

9) The community wont be pouted. There will be different user groups of Binary Gentoo and Gentoo (even a different forum)... the Binary Gentoo users wont be that much of an expert so their bug reports will be inferior to the ones made by the source Gentoo users, so some Gentoo users will be 'qualified' by Gentoo foundation to review the bug reports and will be given right to close them if they are not valid. We can have a better validation system by using the experienced Gentoo community. This will save the tragedy of Ubuntu where there are no experienced users at all... as a result the community has turned to trash, you never get answers to real questions there + the packages are the most buggiest of any other distro...

I'm going to assume you mean "parted" here.

So you're saying the community won't be parted, because there's going to be two different communities? Say what?

We already have people who do bug triage - they're called Bug Wranglers. And they're already overwhelmed with the existing from-source distro community. Where are you going to get all these people suddenly interested in bug triage from?

We'll be saved from the tragedy of a community full of unknowledgeable users by the binary distros community of unkowledgeable users? Or by the from-source distros community of users who couldn't care less about a crippled binary distro with no user of developer community that is nothing like the distro they use - it just happens to use the same name and attempt to use the from-source distro to create a binary distro in a completely unsuitable manner?

10) The larger community will consists of experts and normal users so 2 the 2 can help each other... this will be a very healthy mix of users. The real questions will be migrated to Gentoo forums which the easy questions will be moved to the Binary Gentoo forums.

So now you're going to mix the support issues and communities of these two seperate distros with two seperate forums with completely different users who are completely uninstrerested in each others solutions because one group can quickly and easily modify just about anything and the other can't change anything? And where are you going to get these people who are interested in trying to run support for both distros at the same time?


Some finally, some words from a Gentoo developer:

11:55:10 <+a3li> oh him
11:55:14 <+a3li> I already wanted to reply
11:55:19 <+a3li> with one word.
11:55:20 <+a3li> NO.

[dE_logics]

It's not that simple. There can't be one make.conf for everyone - you can't take into account architectures (CHOST / CFLAGS), etc. You'd need a seperate BINHOST for each architecture you wish to support. You then have to produce all of those binaries in a timely manner, test them to make sure they work to at least some degree, host them and provide the bandwidth for them to be downloaded.

This is an issue with every distro... this one is also a distro, so these things will also be there; in fact doing it with as Gentoo backend is much easier.

If the community is expected to be large, so will the effort.

Initially we need to host only 2 projects... x86, x86_64 or actually to start off it will only be x86.

Then there's the issue of USE flags, which are a core part of Gentoo, in my opinion. As far as I can see, you're proposing that there's one set of USE flags that everyone uses and no one ever changes. This simply doesn't work.

To take an example, look at PHP. Some users will want the Oracle extensions, which require the commercially licensed Oracle packages, but not everyone will. How do you deal with this?

Another example: MySQL - Amarok (and other applications) use embedded MySQL, so some people will want that while others won't. Then you have people who use seperate machines for the server and client sides of MySQL (ie, some people will want minimal on some machines while not on others) - How do you deal with that?

Some people use Gentoo on a networked setup, with LDAP or Kerberos or some other network authentication method, while others use Gentoo on a single desktop / server and don't want networked authentication at all. Some people use a desktop environment and want all of the GTK / Qt / KDE / Gnome / etc frontends, while others use a desktop with only Qt/KDE or GTK/Gnome. Others use Gentoo on a headless or server environment and want no graphical frontends at all. How do you deal with that?

You cant use USE flags in a binary distro... BUT with this disto, since it's hybrid -

4) Hybrid disto -- Gentoo can be both a source and binary disto...

You will have an option to compile with your specific USE... this will be the only binary distro.

Oh, and one really really big thing you don't even mention in your whole plan: How do you deal with the concept of a rolling tree in a binary distro?

We have arch for an e.g. and even Sabayon...

That aren't enough people interested in this, within or outside of the Gentoo developers.

Ok, that's a strong reason...

Anyway, even if the project starts I wont use it, I'll stick to the sources.

If they were interested in working on a bianry distro, they'd be working on a binary distro already.

As I've emphasized always, this wont be another Debian distro, it has it's own advantages.

But only if you ignore:
- the package management system wasn't designed to be used as a binary distro package management system
- the concept of USE flags
- architectures
- CFLAGS / CXXFLAGS / LDFLAGS
- The ability to use different compilers

The package management system can be modified for the purpose... question is if there are a few devs who would like to modify for the purpose, will the existing devs allow (or will another project start to tell them that it WILL be a success e.g. - Exherbo

Why will we need different compilers?... ok, for a few packages -- exceptions.

Ok so skip the rest... so this concept will remain dormant until more devs join.

[krinn]

I don't think it's a viable project, it would be ok if you handle all cases gentoo can met, and provide package for those case, but it's impossible.

for every package you will need tons of binaries :
(let's say we are kind and pickup one example for x86 users with gcc-3.4.4 and glibc-2.11.0 that wish emerge the package name "fruit" that have (only) two use flag (named: "one" and "two")

this, just for that package, gives you those binaries:
fruit-x86-gcc3.4.4-glibc2.11.0
fruit-x86-gcc3.4.4-glibc2.11.0-one
fruit-x86-gcc3.4.4-glibc2.11.0-two
fruit-x86-gcc3.4.4-glibc2.11.0-one-two

This way you cover all cases a gentoo user might met for that "fruit" package.
But without even taking another arch, the gcc version, glibc version might expand that list to an amazing 50 binaries: just for one package !

[Mousee]

Then there's the issue of USE flags, which are a core part of Gentoo, in my opinion. As far as I can see, you're proposing that there's one set of USE flags that everyone uses and no one ever changes. This simply doesn't work.

To take an example, look at PHP. Some users will want the Oracle extensions, which require the commercially licensed Oracle packages, but not everyone will. How do you deal with this?

Another example: MySQL - Amarok (and other applications) use embedded MySQL, so some people will want that while others won't. Then you have people who use seperate machines for the server and client sides of MySQL (ie, some people will want minimal on some machines while not on others) - How do you deal with that?

Some people use Gentoo on a networked setup, with LDAP or Kerberos or some other network authentication method, while others use Gentoo on a single desktop / server and don't want networked authentication at all. Some people use a desktop environment and want all of the GTK / Qt / KDE / Gnome / etc frontends, while others use a desktop with only Qt/KDE or GTK/Gnome. Others use Gentoo on a headless or server environment and want no graphical frontends at all. How do you deal with that?

You cant use USE flags in a binary distro... BUT with this disto, since it's hybrid -

4) Hybrid disto -- Gentoo can be both a source and binary disto...

You will have an option to compile with your specific USE... this will be the only binary distro.

I'm sorry but this wouldn't work at all. The only possible way it *might* work, is if users weren't allowed to modify CFLAGS, CHOST, LDFLAGS, etc. Even still, some sets of USE flags simply won't work in combination with one another, or some USE flags that are required by one package will be an "anti-requirement" for others. For example, and I'm making this up entirely because I don't feel like digging for a real example, www-servers/apache requires the "php" USE flag set in order to work properly with dev-lang/php; however, net-proxy/squid requires that www-servers/apache NOT have the "php" USE flag set and rather the "fastcgi" USE flag set in order to work properly with the Apache webserver. How do you handle such USE flag discrepancies in a "Hybrid" distro? You'd have to have several binary packages of the same package/application built for all such situations and have some form of validation system in place so that, should a user choose to build a package from source and not use the binary equivalent, that package will fail to build until the user has solved the USE flag discrepancies that the validation system has flagged as an issue. That, in itself, is far too daunting of a task even for a "large" community. That's a lot of work for extremely little gain.

I could see starting a side-project for Gentoo that has pre-built binaries for commonly used and very large packages (ie. KDE, Gnome, or Open Office). You would essentially have a "build system" in place that cross-compiles for all of the various architectures, builds either every possible combination of USE flags with said packages, and then allows for a client to connect to it via portage and select the appropriate binary to distribute to the client based on settings in your make.conf (ie. CHOST, CFLAGS, etc). This would, again, be a lot of work and for little gain to those involved in maintaining the project. You'd require QC on the binary packages (people to test and verify nothing is wrong), a set of build systems in place to do all of the compilation work (every day I would guess), a set of distribution servers, and of course portage would have to be modified to support this additional functionality as well. So I can't say such a project would even be worth considering. Sabayon already does this well enough anyways, from what I understand.

Otherwise I believe AllenJB has made some solid points and I quite agree that these things are not as simple as you make them out to be. There is a lot of work involved in even putting together a team to head such a project. I'm afraid there aren't really that many people just "throwing" themselves at the Gentoo Project, offering tons and tons of help for little return, these days.

[AllenJB]

It's not that simple. There can't be one make.conf for everyone - you can't take into account architectures (CHOST / CFLAGS), etc. You'd need a seperate BINHOST for each architecture you wish to support. You then have to produce all of those binaries in a timely manner, test them to make sure they work to at least some degree, host them and provide the bandwidth for them to be downloaded.

This is an issue with every distro... this one is also a distro, so these things will also be there; in fact doing it with as Gentoo backend is much easier.

Er, I don't think you know the way Portage works nearly well enough to be saying this...

If the community is expected to be large, so will the effort.

Initially we need to host only 2 projects... x86, x86_64 or actually to start off it will only be x86.

The initial effort in creating the infrastructure and methods for a supportable binary distro based on Gentoo as massive. Go ask Sabayon. They've done it once, but not with Portage directly - they had to create additions to and modify Portage to get it to work.

Then there's the issue of USE flags, which are a core part of Gentoo, in my opinion. As far as I can see, you're proposing that there's one set of USE flags that everyone uses and no one ever changes. This simply doesn't work.

To take an example, look at PHP. Some users will want the Oracle extensions, which require the commercially licensed Oracle packages, but not everyone will. How do you deal with this?

Another example: MySQL - Amarok (and other applications) use embedded MySQL, so some people will want that while others won't. Then you have people who use seperate machines for the server and client sides of MySQL (ie, some people will want minimal on some machines while not on others) - How do you deal with that?

Some people use Gentoo on a networked setup, with LDAP or Kerberos or some other network authentication method, while others use Gentoo on a single desktop / server and don't want networked authentication at all. Some people use a desktop environment and want all of the GTK / Qt / KDE / Gnome / etc frontends, while others use a desktop with only Qt/KDE or GTK/Gnome. Others use Gentoo on a headless or server environment and want no graphical frontends at all. How do you deal with that?

You cant use USE flags in a binary distro... BUT with this disto, since it's hybrid -

4) Hybrid disto -- Gentoo can be both a source and binary disto...

You will have an option to compile with your specific USE... this will be the only binary distro.

You seem to totally misunderstand USE flags, what they are, how they work, and how most binary distros deal with these situations, and thus why it won't work with Gentoo's packaging setup. USE flags can affect not only the dependencies, but also the files installed AND more importantly the contents of those files. Additionally, while Gentoo's BINHOST system knows what USE flags a given binary package was created with (due to the metadata on it), it won't use that binary package if the use flags its told to use are different (because it can't - it doesn't make any sense). Additionally, it can only recognize one binary package per .ebuild in a BINHOST - it has no handling for alternative binary packages with alternative use flags.

The way binary distros deal with this is to have completely seperate packages - eg. mysql-embedded, mysql-minimal, mysql-library, etc. Because of the way Gentoo works, it hasn't needed to do this with its package tree, so the packaging setup becomes completely unsuitable for dealing with these binary packaging issues.

Oh, and one really really big thing you don't even mention in your whole plan: How do you deal with the concept of a rolling tree in a binary distro?

We have arch for an e.g. and even Sabayon...

Yes, but have you looked at how those deal with it in detail, and how Gentoo's Portage works in relation to that?

That aren't enough people interested in this, within or outside of the Gentoo developers.

Ok, that's a strong reason...

Anyway, even if the project starts I wont use it, I'll stick to the sources.

So even tho you're the one who's pushing this idea, you won't consider using it? Why push it then? What makes you think anyone who uses Gentoo thinks any different?

If they were interested in working on a bianry distro, they'd be working on a binary distro already.

As I've emphasized always, this wont be another Debian distro, it has it's own advantages.

What exactly do you mean by "another Debian distro"? How would Gentoo be different.

While we're on the subject of Debian, don't forget that because of the way Gentoo builds packages from source, on your machine, it has to worry a lot less about commercial packages in the tree. You're going to have to deal with that (take Mozilla's Firefox and Thunderbird as popular examples).

But only if you ignore:
- the package management system wasn't designed to be used as a binary distro package management system
- the concept of USE flags
- architectures
- CFLAGS / CXXFLAGS / LDFLAGS
- The ability to use different compilers

The package management system can be modified for the purpose... question is if there are a few devs who would like to modify for the purpose, will the existing devs allow (or will another project start to tell them that it WILL be a success e.g. - Exherbo

Why will we need different compilers?... ok, for a few packages -- exceptions.

Ok so skip the rest... so this concept will remain dormant until more devs join.

We need different compilers because some people like to choose to use ICC over GCC. And some architectures require it, iirc.

Why do you want to insist on trying to start a binary distro using a from-source distro? Using a package management system that's totally unsuited to it? Again I point to the millions of "not another Debian distro" binary distros and their package managers that already exist. You continually use Sabayon as an example, and you've also pointed to Arch which has been compared to Gentoo. Why not use Sabayon or Arch instead? Or use them as starting points?


Given the fact that you aren't even interested in using, never mind developing or even leading this project, it's basically a dead baby (compeletely ignoring all the reasons it simply wouldn't work anyway).

[dE_logics]

Given the fact that you aren't even interested in using, never mind developing or even leading this project, it's basically a dead baby (compeletely ignoring all the reasons it simply wouldn't work anyway).

If my business grow I will hire devs to do the job. Current Gentoo users might not see an advantage, but businessmen do. Finally majority are not Gentoo users... that's what Bill, Steve know and most Linux devs do not.


Anyway this project is clearly possible, portage is capable of merging. As of the USE of binaries, portage will always have the ability to re emerge a installed package from sources with different USE defined in package.use... as I've stated before. I'm aware that it may produce blocks and all.

I know the portage EAPI (atleast partially) and have considered them into the idea.

So my main concern is if someone tries to modify portage for the job, will the Gentoo community allow?

[Shining Arcanine]

I think that this is more feasible than other people think, although it would be to Gentoo what Ubuntu is to Debian. You would need to compile generic kernels, bloat packages with as many USE flags as makes sense, setup an overlay to host the kernels, make another overlay to host userland components and then implement some sort of forced reinstallation every time a core library changes and the changes require things to be rebuilt according to revdep-rebuild.

You would also need to implement a GUI installer and a wrapper for the system package manager. The forced reinstallation on binary compilation breakage is not going to be done by portage, because there is no facility in place to detect these changes when updates are done. While it is possible to implement such an infrastructure in Gentoo Linux, installing the binaries will make portage either think that the issue was resolved or make portage enter an infinite loop, neither situation being acceptable. This is because whether or not the binary packages are up to date is not an issue about which portage would care and any sort of out of sync issue would cause either of those scenarios depending on how the mechanism is designed.

To avoid having changes to portage break the wrapper, some changes to portage would likely be necessary to subjugate portage to the task of functioning as a back-end. With the blessing of portage's developer, portage could be split into a back-end part and a front-end part. The front-end part would be a traditional Gentoo front-end and would interact with the back-end through an extensible portage-specific API. Portage could be given a USE flag to state whether or not it is to install its front-end. The traditional front-end then could just be one of several front-ends, such as GUI and commandline front-ends that function as wrappers which implement the checks needed to ensure that binary compatibility issues are handled properly.

Such a distribution would likely need to support user compilation of kernels, so having two ovelays (kernel and userland) be separate from the kernel overlay would be necessary so that people could choose to manage their kernels themselves. Whether or not users are using their own kernels or the overlay-provided kernels would need to be considered by the wrapper for the package manager, so that it avoids reinstalling binaries for packages are recompiled by module-rebuild rebuild, which might occasionally cause problems for users that compile their own kernels, but that seems unavoidable to me. Most of them could likely be avoided by warning them about kernel updates and allowing them to do them separately. The wrapper for portage would likely handle the details.

A challenge that would need to be considered is how to configure the boot loader in a way flexible enough for the parent distribution's methodology to be used in the case of a custom kernel, but inflexible enough that kernel upgrades could be performed without breaking the system when the provided kernel is used. The wrapper would also need to be in its own bin-sys-tools overlay, an overlay for all of the userland components that make having the binary version possible.

Lastly, Gentoo's extensive architecture support and alts projects would need to be unsupported in the derivative distribution in the name of keeping the scale to something manageable.

I think that if these considerations are made it is quite feasible to derive a binary rolling distribution from Gentoo. The creation of such a distribution could cause the merger of the Arch Linux and Gentoo Linux communities. Another possibility would be getting other rolling distributions (i.e. Debian-derived rolling distributions like Linux Mint Debian Edition) to adopt the derivative distribution as their platform and extend it to get their own distributions out of it.

[dE_logics]

and then implement some sort of forced reinstallation every time a core library changes and the changes require things to be rebuilt according to revdep-rebuild.

No. The packages has to be re-build in case a dependency changes and revdep does suggest a rebuild. Sabayon does that in practice. Yeah that means lots of updates, but we have MBPS speed Internet.

[Shining Arcanine]

and then implement some sort of forced reinstallation every time a core library changes and the changes require things to be rebuilt according to revdep-rebuild.

No. The packages has to be re-build in case a dependency changes and revdep does suggest a rebuild. Sabayon does that in practice. Yeah that means lots of updates, but we have MBPS speed Internet.

Your statement assumes that I said that the packages are not recompiled anywhere, which is wrong because they must be recompiled by the distribution vendor. You could not have a binary rolling distribution without having the distribution vendor recompile packages. Using libpng 1.2 to 1.4 as an example, you also cannot have a binary rolling distribution if you do not reinstall packages by uninstalling the version compiled against libpng-1.2 prior to installing the version compiled against libpng-1.4 when you upgrade from libpng 1.2 to libpng 1.4.

For a Gentoo-derived binary rolling distribution that is 100% backward compatible with Gentoo portage, there is no room to change package versions, so you need to transparently update the binary packages in the background and implement logic into the package manager wrapper that will force a reinstallation. You might even need to change the remote host against which the portage tree is syncronized so that a lag in updates could be implemented allowing binary package updates to be built prior to portage's detection of updates.

[dE_logics]

Using libpng 1.2 to 1.4 as an example, you also cannot have a binary rolling distribution if you do not reinstall packages by uninstalling the version compiled against libpng-1.2 prior to installing the version compiled against libpng-1.4 when you upgrade from libpng 1.2 to libpng 1.4.

Yes, the packages depending on libpng will be recompiled by upstream and the new updated binaries will have to be provided to the user.


Ok, so package nomenclature is an issue here, but that can be solved IMO by modifying the binary format of portage. The metadata of the packages should also include the core libraries against which the package was compiled (actually this will be more of a dependency) and changes will be made as necessary.

For e.g. if an installed version of Gimp has been compiled with libpng-1.2 and after a package index sync, libpng upgraded to 1.4, a new version of Gimp will be downloaded from the mirrors who's metadata says, yeah, it has been compiled against the 1.4 version.

Now one issue arises -- the package might not get updated in the mirrors yet, so the user might just download the same version again.

There has to be a solution to this.

[Shining Arcanine]

Using libpng 1.2 to 1.4 as an example, you also cannot have a binary rolling distribution if you do not reinstall packages by uninstalling the version compiled against libpng-1.2 prior to installing the version compiled against libpng-1.4 when you upgrade from libpng 1.2 to libpng 1.4.

Yes, the packages depending on libpng will be recompiled by upstream and the new updated binaries will have to be provided to the user.


Ok, so package nomenclature is an issue here, but that can be solved IMO by modifying the binary format of portage. The metadata of the packages should also include the core libraries against which the package was compiled (actually this will be more of a dependency) and changes will be made as necessary.

For e.g. if an installed version of Gimp has been compiled with libpng-1.2 and after a package index sync, libpng upgraded to 1.4, a new version of Gimp will be downloaded from the mirrors who's metadata says, yeah, it has been compiled against the 1.4 version.

Now one issue arises -- the package might not get updated in the mirrors yet, so the user might just download the same version again.

There has to be a solution to this.

You can leave the binary format alone. Just maintain a separate database with md5sums for the latest packages, md5sums for the installed binary packages and a series of reverse-dependency trees for things that must be reinstalled alongside any packages that break libraries. It would be like inheritance in object oriented programming. There is probably a little more that would need to be done with that so that the entire thing does not explode when you try compiling things yourself, but the basic principle should remain the same.

Keep in mind that in this case, reinstalling means deleting the binaries that were compiled against the old version and installing the binaries that were compiled against the new version. Also, this depends on only allowing the most recent stable version of packages. If people decide that they want libpng 1.2 while the binary distribution depends on libpng 1.4, they should have no expectation that it will work or continue to work as the binary distribution is updated. There is no way of giving people much flexibility in terms of this, because producing binaries for every possible combination of software package versions is simply infeasible, even if keeping track of it all could work in theory.

[cach0rr0]

my two cents:

I'd have more interest in community-driven public binhosts. Maybe even something done through the overlays.

And to that end, a bit of doc on the wiki that explains how to use said binhosts.

I think KDE4 users especially would enjoy the time saved

It would still need to be a full time project keeping the binhosts maintained, but I think you'll gain more ground that direction than trying to formally fold anything into gentoo, or creating an entirely new 'distro'

[dE_logics]

So finally, this became a reality.

[eschwartz]

Yup.

And lots of interesting discussion from 14 years ago. Mainly interesting because it appears to have been pessimistic in unusual ways about how this could or would work -- in particular, the fact that portage already has a way to handle all sorts of rebuild scenarios e.g. dependencies can have SLOT bindings as well as configured USE flags, and portage will simply not consider a binhost package as a candidate if it was built for a different libpng SLOT. It transparently falls back to compiling from source.

Also, the binhost can have multiple copies of the same package compiled for different USE flags.

Of course, this took quite some time to get all polished. e.g. having multiple copies of the same package uses FEATURES=binpkg-multi-instance, which was added in 2015! Or migration from xpak to gpkg, which was added in 2022 and needed for security signatures. And that's really, probably, the reason why 2010 and 2024 look so different in terms of perspective on using binhosts. It took a long time for portage to enable all the features necessary to roll this out at scale, features which also allow people to run their own personal binhosts or even just save binaries of their own packages for quick cp'ing across local machines with greater peace of mind that those packages will do the right thing.

As a quick note: the current design means it's very easy to maintain the binhost, we just have a server with a few chrooted gentoo installs and they just emerge a lot of packages with different `eselect profile` options and a common /var/cache/binpkgs which is then uploaded to the mirrors. Most of the actual maintenance I actually end up doing on the binhost turns out to be solving package.use issues after stuff like the KDE 6 upgrade.

[C5ace]

"Calculate Linux" www.calculate-linux.org
is a Gentoo Binary Distro that uses 'cl-update ....' for binary packages and 'emerge ....' for source packages.
I use Calculate Linux on one of my families laptops.

[flysideways]

I'd rather see the documentation for local binary hosting and catalyst being written to the level that even luddites like me can find their way through with minimal other outside help.

Gentoo has been the toolset to create stuff my way, and I've gotten to keep the the parts that I have broken.

Recently I have been in the arm64 space with both Apple Silicon and Pi's. I will tell you, that when there are tools in Debian arm64, that I am bugging because they are not even keyworded in Gentoo, we are at an inversion of the Linux space from not too many years ago.

Even though I have strayed at times when I needed a tool and did not have the time to maintain it, I always came back.

It will be doom, if too many of the Gentoo developers think they are working on the next best distro, and not the premier toolset for creating your own distro.

[eschwartz]

I'd rather see the documentation for local binary hosting and catalyst being written to the level that even luddites like me can find their way through with minimal other outside help.

Have you looked at https://wiki.gentoo.org/wiki/Binary_package_guide

Gentoo has been the toolset to create stuff my way, and I've gotten to keep the the parts that I have broken.

Recently I have been in the arm64 space with both Apple Silicon and Pi's. I will tell you, that when there are tools in Debian arm64, that I am bugging because they are not even keyworded in Gentoo, we are at an inversion of the Linux space from not too many years ago.

Really? What's wrong with gentoo's ACCEPT_KEYWORDS support or package.accept_keywords?

You are welcome to submit a keywording bug if you've tested the tools and verified they work on that architecture. https://bugs.gentoo.org/ -> Product: Gentoo Linux, Compoent: Keywording

Even though I have strayed at times when I needed a tool and did not have the time to maintain it, I always came back.

It will be doom, if too many of the Gentoo developers think they are working on the next best distro, and not the premier toolset for creating your own distro.

I don't quite know what you mean binary packages are just one of the toolsets for creating your own distro, which gentoo.org then provides demos for.

[dE_logics]

Yup.

And lots of interesting discussion from 14 years ago. Mainly interesting because it appears to have been pessimistic in unusual ways about how this could or would work -- in particular, the fact that portage already has a way to handle all sorts of rebuild scenarios e.g. dependencies can have SLOT bindings as well as configured USE flags, and portage will simply not consider a binhost package as a candidate if it was built for a different libpng SLOT. It transparently falls back to compiling from source.

Also, the binhost can have multiple copies of the same package compiled for different USE flags.

Of course, this took quite some time to get all polished. e.g. having multiple copies of the same package uses FEATURES=binpkg-multi-instance, which was added in 2015! Or migration from xpak to gpkg, which was added in 2022 and needed for security signatures. And that's really, probably, the reason why 2010 and 2024 look so different in terms of perspective on using binhosts. It took a long time for portage to enable all the features necessary to roll this out at scale, features which also allow people to run their own personal binhosts or even just save binaries of their own packages for quick cp'ing across local machines with greater peace of mind that those packages will do the right thing.

As a quick note: the current design means it's very easy to maintain the binhost, we just have a server with a few chrooted gentoo installs and they just emerge a lot of packages with different `eselect profile` options and a common /var/cache/binpkgs which is then uploaded to the mirrors. Most of the actual maintenance I actually end up doing on the binhost turns out to be solving package.use issues after stuff like the KDE 6 upgrade.

But USE and slots are things which binary distributions cannot do. With Gentoo binhost, you get a binary based distribution which you can even do that. If you stick with binaries, you've to stick with them at least for the libraries for a complete binary support.

I remember filing that xpak feature request. This was needed for the project to be successful at least.

Everyone always saw Gentoo as the source of binary distributions. First it was chromeos and, now, finally Gentoo itself.