Discussion on user contributions to Gentoo: issues/proposals

[yngwin]

Split off from another topic (http://forums.gentoo.org/viewtopic-t-702174.html). To start with, I will just highlight a few quotes about the (im)possibilities of contributing to Gentoo.

I started an approach back in 2005 to give something back to the gentoo community, but nothing came out of that very try. Now I have a job, a wife, a child and a house and do not have as much spare time as I like to have. I am still interested in helping gentoo (I am used to linux && gentoo quite a bit ) but I don't want to beg for my help to be accepted.

the truth is that I know a lot of people that [would] help the project, myself included if it was not because the hard headed, top of the hill people that drive the project, that make it difficult for anyone to become a maintainer of a ebuild

cool then what can I do to be a kde ebuild maitainer for example? the last time I try to help people keep telling me to wait and post things to bugzilla.. but see I don't have time to wait.. is or I help or I don't..

as far as I understand it we cannot just "join and help out." You have to choose us. The best thing people like me can do is post bugs in Bugzilla and expect completely unpredictable results. Sometimes response is prompt (thank you Gentoo devs); issues get resolved and things get done. Other times bugs just pile on and get ignored. I have some time and skills, but I cannot seem to find a way to contribute effectively. In the end, I do not think Gentoo is able to make use of the potential "time and effort" that can be harnessed from its community.

Gentoo it seems has very closed policy when it comes to devs. Even when you want to help, you've got time and resources to do so, it's very hard to become a dev. Devs are trying to keep their numbers low apparently. It's quite funny, even if you visit fedora project website there are links everywhere how YOU can contribute. Ever seen it in gentoo? In gentoo, you can fix bugs in bugzilla, upload new ebuilds and and the end of the day _almost_ no one gives a shit really. Check out how many bugs are fixed by "users" and still marked as opened, because no one even looked at them...

I understand the use of sunrise is discouraged. I remember reading somewhere that its existence was not to be widely publicized in Gentoo media.
I have tried translating and I respect everyone who does it.

I have some time and really think I have some skills but I don't know where to start helping out. I am more the technical person, than a social engineer (like forum mod's; i know it is a lot of work and it needs to be done). I _really_ would like to get involved and take some responsibilities but, the barriers are quite high finding a hole where to fit in, I searched for about three years now. [...] I also think I think gentoo has a lot of potential which is not yet used.

To add to your claim that there are many people willing, wanting, waiting to help with gentoo, I ask how many overlays there are out there... about a billion. Those who run overlays are greatly desiring to help with gentoo, but haven't/can't get into the dev community. in fact, they've quit waiting. I know I could get involved with bugday http://www.gentoo.org/proj/en/bugday/index.xml, but I'm already doing that on bugs.gentoo.org with the few patches/ebuilds I can come up with.
I'm just saying I think there are many devs available, much better than I, doing overlays.

These are just from one thread over the last couple of days. I'm sure that if I searched the forums I could find dozens more in the same vein.

My conclusion is that there are clearly both perceived and real barriers for people to contribute to Gentoo, even though they are willing, and maybe even actively seeking ways to do so. It is therefore very important to do something about this. I want to start a discussion, among Gentoo users and staff alike, about how Gentoo can open up and tap into this rich reservoir of our community, how to make it easier for people to contribute, and how to publicize these possibilities widely.

I would like to use this thread as a starting point to make an inventory of the ways in which people can contribute to gentoo already, and the possible ways to do so that do not yet exist or aren't really being used. We need to come up with ideas to streamline things so that potential contributors can easily find out how and where to help, in order that Gentoo as a project may be improved, and in ways that will help both the devs and the user community.

I will try to keep a list of ideas and outcomes of this discussion, and will add links to this opening post when I take things to other places (mailing list, wiki, official documentation, etc).

So please lets brainstorm on:
1: What are the different ways in which people can currently contribute to Gentoo?
2: What could be other possibilities for people wanting to contribute, that are not currently used?
3: How and where could we best publicize these?

[Nick C]

I guess the first thing to do is figure out what herds/teams/projects are looking for and get that information up on the staffing needs page. After that maybe have someone/group as an 'anything else' contact for anyone who wants to contribute but either doesnt know where to go. This team would i guess, act as an initial point of contact for most new developers and triage the applications weeding out those who gentoo might not want (no experience etc.) with maybe tips on what to learn or where go for more learning materials. With those applications that look ok, the next step would be to find a developer in that herd/team who would be willing to mentor the applicant. Then recruitment goes as normal.

In my mind, from what ive seen, the problem is getting people wanting to be developers matched up with teams/herds looking for help and then starting the mentoring process. Streamline this and it may help. Of course it should still be possible for devs to headhunt and mentor people outside of this process.

[AllenJB]

Ways people can currently contribute to Gentoo:

• Documentation on Gentoo Wiki
• Updating official documentation by submitting patches to Bugzilla
• Fixing bugs filed by others and submitting the patches
• Filing bugs they find
• Testing the Gentoo Linux Installer (graphical installer on the LiveCD) and filing bugs
• Answering problems [url=irc://irc.freenode.net/#gentoo]on irc[/url]
• Answering problems in the forums
• Submitting new and updated packages to Bugzilla or Sunrise and other overlays
• Getting involved with discussions on the mailing lists (gentoo-dev, gentoo-project, project specific mls)
• Contributing articles to the newsletter
• Proxy maintainership - see http://dev.gentoo.org/~antarus/projects/proxy-maint/ (not sure how up-to-date this article is)

All of the above can be done without even being a developer, although obviously you'll rely on the relevant developers accepting your contributions and applying the changes in many cases.

Where to get help:

• #gentoo and #gentoo-dev-help irc channels - the latter is particularly good for help writing ebuilds
• project specific IRC channels
• Gentoo Dev Manual
• Official documentation
  • Gentoo Developer Handbook
• The mailing lists (gentoo-dev, gentoo-dev-help, gentoo-project, project specific mls)

Where to find information on becoming a developer / helping out:

• Gentoo Developer Handbook
• Staffing Needs

Further resources for ebuild development:

• Bash Beginners Guide
• Advanced Bash Scripting Guide

----
What could developers do more of:

• Keep the Staffing Needs page up-to-date.

----

Why do ebuilds for new packages sit in bugzilla forever?

Manpower. There simply isn't enough of it. Adding a package to portage isn't as simple as committing 1 user submitted ebuild - the developer who commits it is then expected to test it, maintain it, reply to bugs on it, bump it when (significant) new versions are released, fix security issues when they occur, etc. To maintain any sort of quality on the packages in the main repository requires a lot of work. At the end of the day, most people are giving up their free time to work on this distro, and that means that, whether users like it or not, they're going to scratch their itches first, not yours.

But RedHat/Suse/Mandrake fix my bugs within milliseconds!

Who's answering those bugs - those are all actual companies with PAID developers. Gentoo is a VOLUNTEER organisation and relies on people giving up their free time.

----
(Urgh - this forum doesn't seem to handle sub-lists well. And the syntax is horrible. Where's my wiki-syntax?)
(Mwahaha - I tamed the lists - got some horrible long lines without breaks in the code, but it looks nice for readers now!)

Changelog:
2008-08-20 - Add proxy maintainership article

[Nick C]

A useful irc channel is always #gentoo-dev-help for anyone wanting help writing ebuilds or other development stuff.

[yngwin]

Thanks, people, that's a good start, especially the lists of AllenJB. I would like to add a link to the Staffing Needs page: http://www.gentoo.org/proj/en/devrel/staffing-needs/

[AllenJB]

Thanks, people, that's a good start, especially the lists of AllenJB. I would like to add a link to the Staffing Needs page: http://www.gentoo.org/proj/en/devrel/staffing-needs/

Bah - mindreader! I was just adding that.

[node_one]

I posted this suggestion recently. I think it applies here. I have some additional ideas, but I want to formulate them more clearly before I post them.

[Nick C]

Hopefully a mod can sticky this thread as its quite useful already

[think4urs11]

Hopefully a mod can sticky this thread as its quite useful already

agreed and done

[node_one]

Here are some other, on-topic, suggestions. They primarily address "how to make it easier for people to contribute."

Create the reverse of Staffing Needs
Staffing Needs is a one-way publication that informs the user community about the needs of the developers. Unless users actively inquire about a position, the developers are completely unaware of potential "effort resources" that may be out there. Having a place where users can advertise available "effort" they are willing to contribute will make it possible for developers to approach helpful users directly. I am not sure what platform that can be employed here but both the Forum and Bugzilla seem appropriate for something like this.

Generate eclass documentation with doxygen or something similar
Users now can go to /usr/portage/eclass and read the eclasses, but that seems less convenient than looking at doxygen or similar output in a web browser. In addition, doxygen can generate dependency graphs from what I have seen. With better documentation new ebuild developers should be able to learn about the different eclasses quicker and use them more effectively.

Combine documents and create an "Ebuild/Package Development Handbook"
Combine
- Parts 1 - 3 of the "Guides" section in the "Gentoo Developer Handbook"
- The "Gentoo Development Guide"
- "How to fix autotools failures"
- "Automagic dependencies, what they are and how to fix them"
- eclass documentation as described above

All of these documents seem to belong together. They all describe how to write quality ebuilds.

Create an "Ebuild/Package Lifecycle Handbook"
Combine
- Parts 4 - 6 of the "Guides" section in the "Gentoo Developer Handbook"
- "Contributing Ebuilds"
- "Gentoo Overlays: Users' Guide"
- "Gentoo Overlays: Developers' Guide"
- Sunrise related documentation, if necessary

Also, create a comprehensive workflow diagram/graph that covers an ebuild's entire lifecycle; from when someone, either developer or user, decided to create a new ebuild, through its inclusion in the main portage tree, through its maintenance, to its potential removal from the tree, etc. I can create an example diagram/graph if it is unclear what I am talking about.

The "Ebuild/Package Lifecycle Handbook" should document a unified process for ebuild management that covers both Gentoo developers and other contributors. I think this will address a lot of issues, remove the perceived barriers and clearly define the real ones.

Please comment on these.

[cctsurf]

Now in the thread from which this split, there was much emphasis on using overlays... Which I am doing, however, I find that this makes Gentoo much more difficult to maintain. There is overlap in ebuilds, different overlays have different channels by which they receive bugs/questions concerning their ebuilds--or none at all... How is this helping make Gentoo better?
I love the leanness which I have with Gentoo, I love the ability to have the OS customized to my desires and computers, and I have at times attempted to use other distributions, but I can't get away from the feeling of bloat when I have used them and I have always come back. Sometimes we just have to wait a little bit longer, but the wait is well worth it.
Thanks devs, I hope we can get things speeded up, but I you've got the best thing going already!
James

[node_one]

I do not mean to criticize your editorial decisions, but I think the title of this thread does not accurately describe it. A Sticky thread with this title might appear to be informative instead of a discussion thread and users might be inclined to skip it.

Something like "User contributions - new proposals and discussion of current issues" might be more effective. Just my opinion of course.

[fangorn]

I for my part do not get why everybody seems to have to build an own overlay. I did not find one overlay with all the needs I have. Ergo I have to copy ebuilds from different overlays and fix the dependencies myself. OK, this is doable for me, but why do I have to do it? And will others also be able to do it? This is even less convenient than building the sources by hand with configure, make and make install.

If all the effort that goes in dozens (sometimes I think hundreds) of overlays was directed to the main portage tree I think nobody would talk about an unrecent tree. If people stopped using ~arch as productive systems it could be a testbed again. So nobody would rant if an ebuild does not compile with some strange CFLAGS, ...

I am not an insider and was quite unactive for a while. I did not follow internal discussions. So maybe I missed something important. So please excuse me if I say something obvious/stupid.

[_puck_]

I for my part do not get why everybody seems to have to build an own overlay. ...

+1, nothing more to say.

[krackpot]

Staffing Needs is a one-way publication... Having a place where users can advertise available "effort" they are willing to contribute will make it possible for developers to approach helpful users directly. ...both the Forum and Bugzilla seem appropriate for something like this.

Every major company (Boeing, GSK, URS, Comcast), and many smaller ones, have the exact same problem when looking for paid employees. Their answer is a personalized, searchable resume builder that HR and management can zap through to match applicants to positions.

Our organization has the added benefit where once the dev(s) have found the skills they are looking for they can further investigate the "applicant" by checking out their collective Bugzilla and forum posts.

I don't know how big a project this would actually be, but I wouldn't be surprised if we had a few experienced developers in this arena who were looking for a way to contribute to Gentoo.

[yngwin]

I do not mean to criticize your editorial decisions, but I think the title of this thread does not accurately describe it. A Sticky thread with this title might appear to be informative instead of a discussion thread and users might be inclined to skip it.

Something like "User contributions - new proposals and discussion of current issues" might be more effective. Just my opinion of course.

A valid point. I was still looking for a better title, and hopefully the current one is clearer.

[node_one]

I don't know how big a project this would actually be, but I wouldn't be surprised if we had a few experienced developers in this arena who were looking for a way to contribute to Gentoo.

Ok. So I am assuming there are now at least two people who think that "the reverse of Staffing Needs" or "monster.gentoo.org" is something worth doing.

I for my part do not get why everybody seems to have to build an own overlay.

I think that is a complicated issue that needs to be addressed separately. There are many aspects that I do not seem to completely grasp myself.
________________________________________

Well, I decided to spend some time trying to make a workflow diagram of an ebuild's lifecycle, since I proposed it earlier. Wow, that was a lot harder than I expected. I was trying to see if there is any place where "contributors" (not developers) can get into the development process.

Right now, it appears to me, people like me use Bugzilla as a somewhat ineffective way to contribute. We either make new ebuilds or fix some and post fixes on there. The limited pool of developers then tries to isolate "the good stuff" they can assume maintainer responsibilities for and adopt it. Sometimes ebuilds just sit there and the users who contributed them are upset that their time seems to have been wasted.

Sometimes it is suggested that contributors submit their ebuild to Sunrise. Some do that, they go through the process of getting it there. Others do not. They do not see the added benefit of having an ebuild in Sunrise in addition to having it on Bugzilla. There may be other reasons, but that is another separate topic.

Anyway, I guess I will keep trying to understand the current processes and where people like me can "plug-in." Maybe we should not. Maybe we should be using our time to contribute to Gentoo in other ways instead, and leave the development to the developers. I do not know. I am just trying to figure it all out.

I apologize if I am negative anywhere above. These are my observations and I am trying to inform, not necessarily judge.
________________________________________

Something else I also noticed recently. There does not appear to be a defined way to rename topics as solved in the support forums. I occasionally visit to see what is going on and if I can answer some questions. I have seen [Solved] [solved] and [SOLVED] all used. Sometimes they are not used at all even if the question has been answered completely to the user's satisfaction.

I was guilty of this myself (I should check my own posts and see if I need to "close" any of them). Actually, at that time I did not think to do it and I was not exactly sure how to do it. I think there should be a sticky or an announcement post in the Support forums telling users how and when to do this.

There are a couple of other suggestions around this issue, but those might involve dealing with Forum code. Something I do not have the skills for and it seems to me I am in the minority when it comes to doing anything more with the forums.

EDIT: Found this topic related to this.

[cctsurf]

I for my part do not get why everybody seems to have to build an own overlay.

I think that is a complicated issue that needs to be addressed separately. There are many aspects that I do not seem to completely grasp myself.

While I agree that this is a complex issue, it is one that NEEDS to be dealt with, and if you read the thread that this is a fork from, it was a central question of that thread. Are Overlays the right way of going about extending Gentoo? It was my understanding that this was the thread in which to ask and to answer that question. I realize the first posts were about how can we get involved, well overlays are the method that MOST PEOPLE have used to get involved because they are much simpler than any other method.

[node_one]

cctsurf, I agree with almost everything you said. The other thread has about 5 to 1 readership for a reason. All I was stating is that maybe we should fork "the overlay issue" too, so it can be discussed separately, without interruption by other issues. I would fork this issue myself if I knew how to approach it correctly, asking the right questions, etc. Right now my misconceptions could be misinterpreted and needlessly offend/upset people etc. We can talk about overlays here too. It does not matter to me at all.

EDIT: I missed some of your previous posts (maybe we should fork ). A discussion already exists here. Retracting.

... there was much emphasis on using overlays... Which I am doing, however, I find that this makes Gentoo much more difficult to maintain. There is overlap in ebuilds, different overlays have different channels by which they receive bugs/questions concerning their ebuilds--or none at all... How is this helping make Gentoo better?

I agree, however I do not see how this situation can be improved. There are at least the users' and the developers' perspectives here. (Not trying to divide people.)

[yngwin]

Overlays aren't going away. Actually they can be very useful. But indeed the way things work could be improved. I think it could help to have a document with guidelines for how to setup and maintain a compatible overlay. Also portage needs to implement some features to make it easier to work with overlays (such as the ability to (un)mask packages by overlay, as well as prioritize one repository over another, see bug 185377).

[fangorn]

A portage that could handle overlays better would surely help. But it does not solve the problem.

As I unterstood, overlays were designed to be for software of the development state "if it cancels your bank account and eats your puppy, don't blame me". This is (or at least was) not designed to be used by masses of people on a regular basis. In the meantime if you want that feature use this overlay, if you want another feature use another overlay, ...

Isn't that what originally ~arch was designed for?

So what is keeping all the (good) work that is done to maintain all these overlays from being contributed to the mainline portage tree? I do not wish to kick someones butt, I just want to understand.

Are there differences in what software is ready to be used by everybody? I know the difference between supporting an ebuild in an overlay which everybody has the free deciscion to use or not to use and an official supported package. And I do not ask for official releases of svn-ebuilds or something heavily work intensive like that. But official upstream releases should be doable (the work is done in the overlays anyway).

Are there some differences in understanding how a good ebuild has to look like?

Are there other personal barriers?

[cctsurf]

A portage that could handle overlays better would surely help. But it does not solve the problem.

It's my understanding that there are "political problems" in the KDE team over just such a project -- paludis... I wonder that if there are needs for portage--and I would say that some overlay understanding by portage would be a need--then they ought to be incorporated into portage, not creating a political problem, or that if there is a better alternative, we should all be moved to it...

As I unterstood, overlays were designed to be for software of the development state "if it cancels your bank account and eats your puppy, don't blame me". This is (or at least was) not designed to be used by masses of people on a regular basis. In the meantime if you want that feature use this overlay, if you want another feature use another overlay, ...

Isn't that what originally ~arch was designed for?

This was my understanding as well, I remember the days when ~arch broke my system regularly and I had all sorts of masks, but at the same time, arch was up to date, now it's been more than a year since my last ~arch breakage...

So what is keeping all the (good) work that is done to maintain all these overlays from being contributed to the mainline portage tree? I do not wish to kick someones butt, I just want to understand.

I guess I'm not interested in kicking some butt, but I am interested in more than understanding. If there is a way in which we can get gentoo moving forward better, I would love to see it!

Are there differences in what software is ready to be used by everybody? I know the difference between supporting an ebuild in an overlay which everybody has the free deciscion to use or not to use and an official supported package. And I do not ask for official releases of svn-ebuilds or something heavily work intensive like that. But official upstream releases should be doable (the work is done in the overlays anyway).

Are there some differences in understanding how a good ebuild has to look like?

Are there other personal barriers?

+1 for me too
there are svn-ebuilds in the standard portage tree, lots of them, ffmpeg, mythtv just off of the top of my head, and they work GREAT! ok, so most of them...

But official upstream releases should be doable (the work is done in the overlays anyway).

+another 10 for me...

[yngwin]

As I unterstood, overlays were designed to be for software of the development state "if it cancels your bank account and eats your puppy, don't blame me".

Not necessarily. There are overlays that indeed do concentrate on experimental stuff, like "live" svn/cvs/git ebuilds. But other overlays can have special interest stable stuff that is just not maintained in portage (like pro-audio).

So what is keeping all the (good) work that is done to maintain all these overlays from being contributed to the mainline portage tree?

If your talking about the non-experimental (or not-too-experimental) stuff, then the issue is often that the overlay maintainer is not a Gentoo developer and none of the current Gentoo devs is willing to maintain the package, or simply doesn't have the time to do that. There are a couple of solutions to this issue, and I hope we can improve and/or add to these here.

First of all, there is Sunrise overlay, specifically geared towards user contributed ebuilds, with Gentoo devs reviewing them. This is a very good starting point for people wanting to contribute, to learn the finer points of writing ebuilds and maintaining packages. People active in Sunrise often go on to become full Gentoo developers.

Then there is also the possibility of proxy maintainers. That means finding a gentoo developer willing to commit your ebuilds, but you keeping up with maintenance and bugs. This should be more widely advertised.

[node_one]

yngwin, what are the benefits to official Gentoo developers to maintain public overlays, either their own or project overlays? What do public overlays provide you with that the main portage tree cannot? If you can mask packages, or leave them un-keyworded, why not have them in the main tree?

[AllenJB]

yngwin, what are the benefits to official Gentoo developers to maintain public overlays, either their own or project overlays? What do public overlays provide you with that the main portage tree cannot? If you can mask packages, or leave them un-keyworded, why not have them in the main tree?

A number of projects use overlays to allow non-developers to contribute to packages. In many cases the contributed packages are then vetted before being moved to the main tree. This also allows potential developers to be "trained up" while making useful contributions. Ultimately this sort of activity leads to the recruitment of more developers.

Other overlays are used to keep rarely used packages available. Packages that the developers don't want to dedicate as much time and effort into keeping track of versions to, but still want to be available to the community for those who do want the package.

The overlays for experimental packages have already been mentioned. These allow developers to develop new eclasses to replace existing ones or develop other new methodologies without breaking the main tree.

----

The problem I forsee with a "monster.g.o" style site is that developers will end up sifting through a lot of noise to get the good results. There will be a lot of people who say nothing more than "I want to be a developer". There's also the potential for a lot of spam on such a site and it would have to be moderated carefully. In the end, while the idea isn't totally bad, I think the signal-to-noise ratio would mean that most developers won't use it.