The logic of Gentoo's keywords versus the SDLC

[Ormaaj]

A given package can have three states:

• Considered unstable/testing by upstream.
  
• Considered stable by upstream, testing by Gentoo.
  
• Considered stable by both Gentoo and upstream.

There are three "transition functions" between package states:

• Large changes to or additions of upstream code.
  
• Changes to existing upstream code which fix bug, regressions, and security issues.
  
• Changes to the way Gentoo builds and installs upstream code.

A Gentoo system is made up of a set of packages which interact with one another. emerge -uDNav @world @system" transitions the whole system state. It's state as a whole is an aggregate of package states in combination with the way Gentoo builds and installs them. This means stability is multidimensional, with bugs either introduced or fixed by either upstream or downstream. Without user intervention, the system states are either all "~arch", or all "arch".

• With all ~arch, you get a mixture of upstream stable and upstream testing, with a high incidence of downstream bugs. New packages can contain any number of bugs either fixed or introduced by either upstream or downstream.
  
• With all "arch", you get all upstream stable and a lower incidence of downstream bugs, but without consideration for the above model, meaning it is possible to have installed packages with bugs fixed upstream in newer versions.

Following this model, the number of undiscovered security bugs and unfixed regressions in packages marked as stable by Gentoo should be approximately equal to those in "upstream stable" packages in ~arch except in cases where bugs are artifacts of the way Gentoo builds and installs software. So the overall effect of running "arch" doesn't translate to mean "stable" or "secure". It just means "old" and little more.

The linux kernel is a good example. With both feature and bugsfix releases, stability should be a variable of the tail element of the version tuple, while features correspond to the second-to-last member. This applies similarly to other packages like Firefox or Wine where most people probably want (or it is even advised) to always run the very latest stable version.

Gentoo is high maintainance for end-users because the categories of "~arch" and "arch" don't reflect this logic. While no defaults will satisfy everyones needs, and Gentoo is all about choice, the "stable and unstable branches" of Gentoo are at two extremes which diverge significantly from nearly everyone's needs. An ideal solution would be to have the right package metadata which indicates stability along all relavent dimensions plus an inference engine to determine what to install based upon a set of constraints - a bit like how portage handles package dependencies. One might for example want all packages in @world to run the latest upstream stable version without pulling in any alpha or beta versions. Not only does this potentially require a lot of either masking or unmasking, it requires that the users pay attention to what is going on upstream for all of the packages they have installed - regardless of whether you use Gentoo's whitelist (arch) or blacklist (~arch) package set as a starting point. Running a huge set of packages like KDE from the overlay is another example which requires the clumbsy method of symlinking a keywords list into the portage configs.

Masks and hardmasks help, but they don't really translate intuitively to the reality of the SDLC. Surely a better system can be invisioned that would reduce headaches for both users and developers. Perhaps something as simple as adding a few more categories in addition to merely marking as either ~ or not.

[depontius]

You've introduced a new acronym, "SDLC". What does it mean? Perhaps it's obvious to software developers, but I'm not one.

I got onto this thread because basically Gentoo flags are binary. When I saw the title, I was thinking of USE flags, now I see you're talking arch flags. Still, they're basically binary. Though there is some syntactic sugar to make things like VIDEO_CARDS look multivalued, look inside the ebuild and it appears to get broken up into a bunch of device-level binary flags. Obviously you've gone in a different direction than I had upon seeing the title, but still similar, in a way.

So imagine for a moment that portage accepts multivalue flags: With what you've suggested, "arch" could take on the 4 combinations of upstream and gentoo stable and unstable. For compatibility purposes, "arch" becomes stable/stable and "~arch" would need a little more definition work, which is really what you've suggested. As for USE flags, what we have now becomes 'USE="flag1 -flag2"' becomes 'USE="flag1=T flag2=f"', and things like VIDEO_CARDS become truly multivalued. I'm thinking that other functions which are currently split into multiple binary-value flags could become multi-valued, though at the moment I can't give an example. The idea would be to have plumbing which could run existing ebuilds and portage configuration, yet have extra power under the hood for a migration target, or special needs, sooner.
_________________
.sigs waste space and bandwidth

[Ormaaj]

Sorry, SDLC is a systems analysis term which has multiple (but related) meanings, either systems development life cycle or software development life cycle which are mainly used in a business setting, but I'm not sure of the best analogous term for the open-source/collaborative/distributed development process of which there are many styles.

If there were a need for finer grained control over useflags it would probably be something developers would be more concerned with than users. My only gripe with use flags is the difficulty in organizing them. Generally I have to do things like comparing the output of "eix -U" and "eix --installed-with-use" to determine whether I want to handle a specific useflag additively or subtractively with package.use or globally in make.conf, and it takes a lot of manual checking to cut down on redundant entries in these files. Occasionally flags become depreciated or not applicable to my set of installed packages which adds more configuration cruft. Keeping flags, keywords, and masks up-to-date is really a nusciance when maintaining multiple systems. I have a few computers I don't update very often and sometimes I just throw away their configs and start over based upon those of my main machine because it would be very hard to spot the differences and make changes where applicable to a specific system.

A bigger related issue is portage's lack of a package atom syntax for dealing with masks and keywords on a per-overlay basis. Paludis handles this better IIRC; I haven't used it for a few years. There are many other package-specific things portage could probably handle better such as multiple compilers and package-specific build flags.

The biggest issue of all though IMO is that the arch flags aren't very meaningful when it comes to stability. It takes a lot of ongoing modification from either ~arch or arch in order to get things where most people probably want them. The complexity comes from the difference between what constitutes the stability of an individual package, and the stability of a system as a whole (and the fact that Gentoo can only officially support the latter).

For example, I can tell just from looking at the version numbers that vanilla-sources-2.6.30.10 contains bugfix changes from 2.6.30.9, which is currently the newest Gentoo has marked as stable from the 2.6.30 series. The fact that 2.6.30.10 exists tells me that 2.6.30.9 contained bugs, and the changes made between these versions were likely to fix more issues than were introduced due to the lack of feature changes. This means the only reason someone would want 2.6.30.9 over 2.6.30.10 is if they care about Gentoo's redundant testing phase, and feel that the POTENTIAL downstream issues introduced from the version bump outweigh the CERTAIN bugs fixed in the upstream bump.

However, it gets worse, because ~arch not only pulls in these changes but also alpha/beta versions that nobody considers stable. Even after upstream releases their *.0 version after their own testing period, Gentoo has it's own testing period which can be for differing reasons depending upon bugfix or feature release and the maturity of ebuilds and downstream patches, which are all things which factor in to the estimated package stability. The total effect is that running a system with no keyword overrides is inappropriate for literally everyone.

Gentoo already has plenty of control. You can customize however you want. What needs to be done is to formally model how software development actually works and then create a system for organizing various things that corresponds with that model.

[AllenJB]

For some reason "reality of the software development lifecycle" jumped out at me while scanning this wall of text. After that I couldn't read any further because I was laughing too much.

You seem to have very little experience of development in:
a) Distros
b) Rolling distros like Gentoo
c) Open source
d) The real world (ie. outside of academia)

With regards to per-overlay syntax for package.*, I believe this is either already implemented in portage 2.2 or planned.

Testing isn't a blacklist. Everything (that's considered stable upstream) starts as testing, then gets whitelisted as stable. If there's a blacklist, it would be negative (eg. -x86) keywords and package.mask.

I don't believe you can compare, to take your example, kernel development to devleopment of the Gentoo package tree. The kernel is a single code base with everything connected. That's not at all true of the package tree - while there are dependencies and such, the links are weak and can easily not exist at all. The kernel is versioned, where as it's practically impossible to version the Gentoo package tree - you can snapshot it, but it has no real version in any terms.

[Ormaaj]

[devilheart]

[beandog]

[beandog]

[beandog]

One last thing I keep thinking to myself is, I imagine you'd make an interesting dev, and if you haven't considered applying to be one, to pleaes think about it.

(Don't let that be an excluded invitation though -- we still need devs for lots of stuff to help out, if anyone is interested)
_________________
If it ain't broke, tweak it. dvds | blurays | blog | wiki

[beandog]

[beandog]

[ccp]

My interpretation of his propose. A profile with English syntax

[beandog]

[Hupf]

[Ormaaj]

[ccp]

So far I fail to understand your definition of "Gentoo software management issue". You seems to key on "arch" versus "~arch"; where "arch" mean stable and "~arch" mean unstable. I think you surely understand stability is relative thing. when I said this thing is stable it not necessary mean the same to you. In Gentoo the "arch" branch only mean it is more "stabl-er" then the "~arch" branch. And hence it explain why there only two branches. If we try to make it three state then it will be even more confusing.

When it come to bug reporting you need to understand it is not Gentoo developer's responsibility for software bug, that belong to original developer/project. Gentoo develop can only responsible to its usability in relative to the current portage content. So it is same reason the Gentoo developer can mistakenly mark a software in to "arch" branch and it does not mean that the "Gentoo software management" have issue.

[John R. Graham]

[devilheart]

[timeBandit]