Idea: Gentoo Automated Error Reporting/Resolution Client

[ttuttle]

Here's an idea I'd like to propose. I'm sorry if it's already been suggested--I didn't see anything in the forums or the GLEP list.

I'd like to create a feature in emerge that, when it encounters a build error, gives the user the choice of submitting the error to a server (possibly hosted by Gentoo, or by someone else). When they submit the error, it will collect information about their system (i.e., installed packages [maybe limited to the deps of that package and their deps], C/CXX/LDFLAGS, USE flags, FEATURES, etc...) and the error, and submit it to the server. Then Gentoo developers and/or users will be able to use the site to figure out what is causing the problem. If they figure it out, they can submit a solution, and then when users encounter the problem again, the system will display the solution and offer to execute it for them. (Executable solutions could require the PGP signatures of n gentoo developers, for n > 1 to prevent developers from accidentally or intentionally messing things up.)

I know this seems a little like Microsoft's error reporting system, but the same process goes on manually in the forums and #gentoo all the time. I just think it would be great if emerge could discover an error and either contribute data towards fixing it or automatically discover a solution.

Comments and constructive, well-thought-out criticisms are requested. Nasty criticisms should go to /dev/null. Oh, and please vote in the poll so I can get an idea of what people think, even if you don't want to post comments.

Thanks for your time.

[dobysirius]

I think it's a great idea. Countless times there's some extremely common problem and it's all over the forums and the topic in IRC but people still ask and ask (just to name a few: pam-login/shadow, kde-env/kdebase).

One thing that I find strange even now, though it is not entirely related, is things that depend on certain USE flags of other things. The package fails to build. It tells you what to do, but what if you're not there?

Back to the topic, it would be a great idea. It could provide the emerge --info and maybe some other things. Of course, it HAS to be optional, and configurable (should it fix automatically, should it ask, should it not?). It wouldn't be too useful if it's off by default, but it has to be. Maybe if it's off and there's an error, Portage could mention about how to enable this to fix your problem.

It has its problems, of course. How do you know what caused the problem? Some errors can arise from multiple causes. The server has to send back commands to check for other possibilities. And it still won't be perfect. As a general idea, though, it's great IMO.

[reillyeon]

I can't agree more. While it should probably be turned off by default I just wish that when something failed to build I could check somewhere to see if it had happened to someone else. An automated reporting system would definitely help that.

I'm not so sure about adding automatic fix scripts. If there's a problem with an ebuild it should just be fixed and the system should suggest that you sync. If it's something like "Recompile this package to fix a missing header." then that can be a message to the user. Distributing more code just means more code to maintain.

[thpani]

Hi,

I voted 'won't be useful', that doesn't fit 100%, so I'll explain my thoughts here:

1. Whatever you collect, it will result in an immensly huge amount of data
2. I think the traditional way of asking in Gentoo's forum/irc/..., and after that if it really turns out to be a bug - posting it in bugzilla is much easier. There are lots of duplicates in bugzilla *now*, which have to be sorted out by gentoo staff, and I guess nobody would have enough time to look at all those error reports. Someone would have to look at any them because
3. I don't understand or can think of a way to match a newly submitted report to an existing one to find the solution. The only one I can think of is someone going through every report and matching it to solution number xxx. Therefor the described traditional approach is more targeted.

So, you could understand my answer not as 'won't be useful', but as 'won't be possible'.

[Naib]

I vote NO
simply because it will fill the auto-buzilla with pointless entries swamping the legit (sounds good in theory)

take my example from today: Updating world so I get GNOME-2.16 and gtk+ keeps failing time after time after time
turns out cairo keeps getting downgraded by pycairo not being ~ARCH and cairo being ~ARCH

round and around until I decide to do a -t output and see it was pycairo causing the probs

[ttuttle]

I think it's a great idea. Countless times there's some extremely common problem and it's all over the forums and the topic in IRC but people still ask and ask (just to name a few: pam-login/shadow, kde-env/kdebase).

I can't agree more. While it should probably be turned off by default I just wish that when something failed to build I could check somewhere to see if it had happened to someone else. An automated reporting system would definitely help that.

2. I think the traditional way of asking in Gentoo's forum/irc/..., and after that if it really turns out to be a bug - posting it in bugzilla is much easier. There are lots of duplicates in bugzilla *now*, which have to be sorted out by gentoo staff, and I guess nobody would have enough time to look at all those error reports. Someone would have to look at any them because

I think it would be useful if the system could at least try to find a fix for the user. Searching the forum is great, but wouldn't you rather emerge spit out a message saying "91.7% of people building package foo/bar with the CFLAG `-funroll-all-loops' had this problem." or "82.5% of people building package foo/bar who had package baz/quux installed had this problem."? (Of course, if the package maintainer has a more useful hint about how to fix the build, they could put that in instead.)

One thing that I find strange even now, though it is not entirely related, is things that depend on certain USE flags of other things. The package fails to build. It tells you what to do, but what if you're not there?

Apparently that is being worked on. marienz told me that this going to be fixed eventually.

Back to the topic, it would be a great idea. It could provide the emerge --info and maybe some other things. Of course, it HAS to be optional, and configurable (should it fix automatically, should it ask, should it not?). It wouldn't be too useful if it's off by default, but it has to be. Maybe if it's off and there's an error, Portage could mention about how to enable this to fix your problem.

I'm not so sure about adding automatic fix scripts. If there's a problem with an ebuild it should just be fixed and the system should suggest that you sync. If it's something like "Recompile this package to fix a missing header." then that can be a message to the user. Distributing more code just means more code to maintain.

Yeah... I've been wondering whether automatic fixing is such a great idea. Perhaps it should just print a message about a known fix if one exists.

It has its problems, of course. How do you know what caused the problem? Some errors can arise from multiple causes. The server has to send back commands to check for other possibilities. And it still won't be perfect. As a general idea, though, it's great IMO.

3. I don't understand or can think of a way to match a newly submitted report to an existing one to find the solution. The only one I can think of is someone going through every report and matching it to solution number xxx. Therefor the described traditional approach is more targeted.

The idea would be that the server aggregates the error reports and highlights similarities in the reports it receives, so it is apparent to whoever's reviewing the reports that a certain package fails when another package is/isn't installed, or a certain CFLAG is present. To parse error messages, it would have a set of regexps for picking out errors in the output from gcc, configure, etc..., and it will aggregate reports with the same (or perhaps similar) message(s).

1. Whatever you collect, it will result in an immensly huge amount of data

Yes, I know. It will aggregate it, so it won't keep a copy of everything, but simply the percentage of systems with a certain flag set. Optionally it could keep compressed copies of the full error reports for a short time (i.e. a week or a month). But longer than that and they're less useful, as things may have been fixed.

So, you could understand my answer not as 'won't be useful', but as 'won't be possible'.

I respectfully disagree. ;-) While the auto-fixing might not be feasible, I think aggregating error reports and helping developers identify their causes would be useful.

[ttuttle]

I vote NO
simply because it will fill the auto-buzilla with pointless entries swamping the legit (sounds good in theory)

take my example from today: Updating world so I get GNOME-2.16 and gtk+ keeps failing time after time after time
turns out cairo keeps getting downgraded by pycairo not being ~ARCH and cairo being ~ARCH

round and around until I decide to do a -t output and see it was pycairo causing the probs

Developers would have an option to filter their error reports by things like that. For example, it would have an option to hide reports from stable packages where the user has unmasked unstable versions of the deps, or things like that. But in your case there is a bug in the ebuild. Portage should be upgrading cairo to the newer version. Each package should build successfully by itself, since gtk+ should pull in the new cairo and pycairo should pull in the old. But whichever one you didn't build last won't work.

[thpani]

So, you could understand my answer not as 'won't be useful', but as 'won't be possible'.

I respectfully disagree. While the auto-fixing might not be feasible, I think aggregating error reports and helping developers identify their causes would be useful.

Aggregating error reports and helping developers identify their causes certainly *IS* useful, but as I said it won't be possible.
What you are designing is not done by playing a around with rexexps. You're talking about a huge datamining application, and I just wonder who will implement it / pay for it.

[96140]

automatically discover a solution

Bad idea. The fluid nature of Gentoo means that configuration steps are constantly changing. Take a look at baselayout -- a fix in one version to solve a problem likely won't be possible in the next version, since variables and/or the "way to do it" has changed. There's no way to realistically keep ahead of configuration changes, so any sort of automatically-discovered fix is impossible.

Also, too much information to collect. Very much an invasion of privacy. Plus, it's not practical to have all these excess processes spawned if, say, you're running a server or some sort of headless machine. It's liable to be a security risk and/or just plain inconvenient.

Also, who's going to build and maintain this datamining (as thpani put it) application? Who's going to act as the support behind each problem? It's generally common knowledge that the devs are stretched pretty far. The Gentoo community, especially these forums, are usually the first place users come when looking for solutions. The users have to support each other, after all. You'd need a non-community, paid support organization to effectively manage a huge troubleshooting backend like the one you're asking for.

[ttuttle]

automatically discover a solution

Bad idea. The fluid nature of Gentoo means that configuration steps are constantly changing. Take a look at baselayout -- a fix in one version to solve a problem likely won't be possible in the next version, since variables and/or the "way to do it" has changed. There's no way to realistically keep ahead of configuration changes, so any sort of automatically-discovered fix is impossible.

First, as to your specific example... it will separate error reports by version.

I'm not saying it will automatically fix problems, and I'm now hesitant to suggest it could even automatically offer fixes.

What I'm thinking about is a system that will collect error reports and present them in aggregate form, so devs can, instead of trolling the forums, IRC channels, and bugzilla, simply get a list of what errors are popping up in their packages with what frequency, and what (if any) common factors relate the systems with the errors. So they could notice if everyone getting an error is on a certain arch, or if everyone is

Also, too much information to collect. Very much an invasion of privacy.

Of course it won't be mandatory, and most likely it will be manually activated when the user wants.

Plus, it's not practical to have all these excess processes spawned if, say, you're running a server or some sort of headless machine. It's liable to be a security risk and/or just plain inconvenient.

Um, I think you misunderstand what I'm doing. This only catches build errors. When it's enabled, it transparently captures the output of each emerge, and if the emerge ends with a build error, it collects information (the emerge output, config.log, make.conf, etc...) and submits it to a server. The server will optionally reply with some sort of message, perhaps a note from the maintainer of the package about how to fix the error, if they've figured it out yet.

Also, who's going to build and maintain this datamining (as thpani put it) application?

I could do it. Everyone seems to think it's this insanely complex datamining system. It doesn't solve errors. It just presents information about them to devs, and allows the devs to enter a fix to be shown to users. The idea is that, instead of having to fix a bug on bugzilla, answer a few messages on the mailing list, answer some threads on the forum, and explain the fix to many people on IRC, the dev will simply enter a message into the system, and when someone else encounters the error, it shows them the message.

Who's going to act as the support behind each problem?

Devs of each package, or perhaps someone can post solutions found using bugzilla, the forums, or IRC.

It's generally common knowledge that the devs are stretched pretty far. The Gentoo community, especially these forums, are usually the first place users come when looking for solutions. The users have to support each other, after all. You'd need a non-community, paid support organization to effectively manage a huge troubleshooting backend like the one you're asking for.

The idea of this is that it saves the devs and the users the trouble of using the forums for sharing solutions to build problems. Instead of searching through pages of threads about a package to see if anyone had the exact same error (since, let's face it, search doesn't always work that well), a user will just let this program check the site. If there's a solution, they see it. And if there isn't, the dev will be able to count the number of users having the problem, see their system configuration, look at the logfiles, and more, without having to track down each user and nag them.

My intent is to save time by collecting useful data in one place and providing a more targeted way to get it to the people who need it.

[96140]

Bugzilla is where errors are reported . . . and also on IRC. The reason why most devs don't use the forums, much less troll it, is that they're too busy. Bugzilla, IRC, and to some extent the mailing lists are where the problems are found and fixed.

[Headrush]

Although it sounds like a decent idea I think the very nature of Gentoo makes interpreting the data hard.
Systems are too varied and no real mechanism to know what was on each system.
Although you can look at emerge --info to guess at what these are, there is no guarantee that any packages was compiled with those settings.
(ex. Some packages I had to modify CFLAGS just for that package.)

This is something all other binary OSes don't have to worry about. You know exactly how and with what settings every packages was compiled with.

Not trying to offend anyone, but seems lately a lot of threads by end users suggesting ways of helping the developers, but not the other way around which is probably better. Developers suggesting things they would like to help them. (if any needed)
A hidden message in these posts?

[djdunn]

its one of theose things that looks good on paper but you end up spending 10X the amount of time trying to impliment it and keep it working, There is just too many variables to keep track of, not to mention, yes, most problems are caused by the same thing, but its bad for the people new to linux especially gentoo to be hand fed the answer, there are a lot of tools and sources that have been around and proven to work. Use them.

Automatically feeding people the answers only destroys the community that built linux. linux survives because the community learns from each other and helps each other.

[djay]

I understand that people are just lazy and they don't want to make an effort.

And yet, there are lots of requests for automatic tools that will do users job.

Come on, people, stop making Fedora from Gentoo. Gentoo has its strenghts and weaknesses, but its not ultimate solution for all problems.

Some "problems" of Gentoo are inherent and are there by design. I don't hear people say "Gentoo is great, but man, compiles are killing me!!!" because this is how Gentoo works. Deal with it.

I just had to stick this here.

And,

Not trying to offend anyone, but seems lately a lot of threads by end users suggesting ways of helping the developers, but not the other way around which is probably better. Developers suggesting things they would like to help them. (if any needed)
A hidden message in these posts?

Well, I don't understand the concept of developer asking users something or suggesting ideas to help ....whom? Users? don't know.

I do know though, that if out of 100 users' requests/suggestions 1 were implemented (its 1 freaking percent), we would be probably in better shape than we are now. And still, thats what is great in our community - ideas get noticed at least by other users.

[Headrush]

And,

Not trying to offend anyone, but seems lately a lot of threads by end users suggesting ways of helping the developers, but not the other way around which is probably better. Developers suggesting things they would like to help them. (if any needed)
A hidden message in these posts?

Well, I don't understand the concept of developer asking users something or suggesting ideas to help ....whom? Users? don't know.

It's simple. There is an assumption by end users that the developers are unaware of problems and that we need to tell them so they can fix it.
Ask any developer if this is the real problem. The point is we should be asking developers what they would like from us that would help them best.
My guess is it isn't this kind of setup.

I like the idea, but I think it is more work than you think to implement and ultimately all you would get from this system is a " these packages seem to be broken for X number of people, so you should look here."

I do know though, that if out of 100 users' requests/suggestions 1 were implemented (its 1 freaking percent), we would be probably in better shape than we are now.

I'm sure more than you think is implemented. Debugging is time consuming and developer resources are limited. If you look back at older Gentoo you will see changes and improvements have been substantial. I think as Linux as a whole matures more, developers will be able to concentrate more on the small remaining issues. (Look at changes lately, gcc, xorg, KDE, hal, dbus, etc)

[djay]

It's simple. There is an assumption by end users that the developers are unaware of problems and that we need to tell them so they can fix it. Ask any developer if this is the real problem.

No its not. The assumption is correct. Of course there are problems that developers do not know about. That's the nature of things, otherwise, if developers new all the problems in the the world, we wouldn't have any bugs, wouldn't we?

The point is we should be asking developers what they would like from us that would help them best. My guess is it isn't this kind of setup.

Well, isn't this what Bug reporting guidelines are there for? And this would not be enough, don't worry, developers will ask for more info they need to solve specific problem . Its not really feasible to give you generic list of what would developer need to solve your problem. Each problem may require its own info.

In addition, you may throw on developers lots of information they don't need, thus complicating the process of finding what the problem is.

I like the idea, but I think it is more work than you think to implement and ultimately all you would get from this system is a " these packages seem to be broken for X number of people, so you should look here."

That is your best case? And if this is all it can do, whats the point? not useful enough.

If problem has an "official" solution (we are talking about an official tool, right?) then it will be integrated into next ebuild version and simple --sync will solve it. If a problem doesn't have an official solution yet, all you'll get is a message that there's no solution. I still don't see the point in such tool.

[Genone]

As I (and others) said on IRC, it's going to be an overly complex project and probably won't be very useful.
The main problem is that the classification engine would have to be very smart to get a useful classification, likely some kind of AI (a simple pattern match won't work). If it isn't extremely accurate (>95% at least) you'll need too much manual work to make it useful. And even then you'll get the occasional mismatch, so it's going to need at least some manual work. Oh, and of course you'd have to integrate it into our existing system (bugs.gentoo.org).
And the automated solution execution/suggestion is a big no no anyway (unless you are 100% accurate, which is impossible).
Short version: rather insane effort required compared to questionable use and high risk of being ignored by the dev community (if it requires too much of their time on a regular base).
What seems to be more useful is to try to streamline bug reports better (like adding a new field to bugzilla to note the cat/pkg-version) so automated searches could become useful, then you might even think about an automated bug reporting tool (but for that you need a reliable dupe search function, otherwise it's just going to be a pain).

[Headrush]

djay, I think you are missing the points I am trying to make.

Your looking at it from a specific bug fixing approach and I'm looking at it from a developer view.
bugzilla is not what I mean by giving the developers what they want. developers don't need more bug reports, they need more useful debugging info like people running debug versions of software that produce useful information. (debugging symbols, etc. )

Just stating CFLAGS, software installed, etc, can be helpful, but isn't that efficient for diagnosing many problems.
(Not saying some problems can't be solved with this)

** I'm not saying this idea can't work for some issues **

Bottom line, I've been on these forums long enough to not trust the info from noobies machines. Its nothing personal but I know due to the complexity and differences with Gentoo and the mistakes they can make nothing is constant and should not be assumed so.

[djay]

djay, I think you are missing the points I am trying to make.

Your looking at it from a specific bug fixing approach and I'm looking at it from a developer view.
bugzilla is not what I mean by giving the developers what they want. developers don't need more bug reports, they need more useful debugging info like people running debug versions of software that produce useful information. (debugging symbols, etc. )

Just stating CFLAGS, software installed, etc, can be helpful, but isn't that efficient for diagnosing many problems.
(Not saying some problems can't be solved with this)

** I'm not saying this idea can't work for some issues **

Bottom line, I've been on these forums long enough to not trust the info from noobies machines. Its nothing personal but I know due to the complexity and differences with Gentoo and the mistakes they can make nothing is constant and should not be assumed so.

You are right, I don't really understand your point.

Let me try this way: we agree that there are many ways and places that developers can get info from and users can send it to. But you talking about the fact, that hand-made info by users about their problem is not trustworthy? And that this process should be automated to eliminate possible errors?

[Headrush]

But you talking about the fact, that hand-made info by users about their problem is not trustworthy? And that this process should be automated to eliminate possible errors?

I am saying that automated system information, (emerge --info), can not be trusted either.
(People say they haven't done something, but they have. eg messed with other CFLAGS, gcc flags, etc)

There is no guarantee that those setting were even used for compiling any installed packages and I don't see any way of finding this information. Because Gentoo is a system a large percentage of people tinker with, your findings can be skewered.

There is just too many variable type things with Gentoo to make this effective.

[djay]

But you talking about the fact, that hand-made info by users about their problem is not trustworthy? And that this process should be automated to eliminate possible errors?

I am saying that automated system information, (emerge --info), can not be trusted either.
(People say they haven't done something, but they have. eg messed with other CFLAGS, gcc flags, etc)

There is no guarantee that those setting were even used for compiling any installed packages and I don't see any way of finding this information. Because Gentoo is a system a large percentage of people tinker with, your findings can be skewered.

There is just too many variable type things with Gentoo to make this effective.

Weird, it seems that we do agree on everything and still arguing. I totally agree with what you're saying. This time I have nothing to add.

[Headrush]

I was never arguing.

My point, whether conveyed well or not, was that from my development and Gentoo experience, I think implementing this would be a lot of work and the results may not be worth the effort to either developer or end user.
(I think the community model works better for several reasons and is what makes Gentoo what it is.)

[djay]

I was never arguing.

(I think the community model works better for several reasons and is what makes Gentoo what it is.)

Thumbs up.
Gentoo rules.

[Corona688]

On an operating system that allows you to do brutally stupid things, I'm not sure auto error notification is such a good idea. Emerge has threatened to cut off my fingers for doing stupid things on occasion. Imagine if the devs were notified every time I did something stuid... They'd want to REALLY cut off my fingers.

Also consider a system that connects to the internet with dialup. Should it dial in and report an error long-distance to yog knows who whenever something goes pear shaped? And imagine when the devs start getting error reports like "F|\33 v!@gr@ L@@@@@@K"

etc, etc, ad infinitum. A barrage of random errors will just make the developers stop listening to error reports. When's the last time an MS or Mozilla auto error report ever changed anything? Half the time it's not even an error in their programs.

[NeddySeagoon]

ThinkingInBinary,

I'm voting no - many users don't understand why Gentoo exists so let me try to simplify it a great deal.
Gentoo is a volunteer distro, all the devs give thier time for free and by an large they do what they want to do when they develop something. If that happers to suit users, thats fine too but develpers essentially do it for themselves.

This means two things. If you don't like the way is going, all the code is open, so fix it yourself. Post bugs with patches, pretty soon you will be a Gentoo develper too. You will find it much easier to change Gentoo from the inside than as a user asking for something from the outside. It also means users 'wish lists' are largely ignored unless they are shared by a developer who is already interested.

I'm with the other developers that have posted in this thrread. Its too much effort for too little return. There are other ways to automate or semi-automate help that work much better and are much lower maintainance.