Idea: Gentoo Automated Error Reporting/Resolution Client

Message

`VL · Post by **`VL** » Wed Oct 25, 2006 7:40 pm

I think that some sort of such a system would be usefull for developers to downgrade rate of user`s requests in case of common problems on compilation of some package.

What we have now? If there`s a _common_ problem, people start to come to irc,post in forums, fill bugzilla`s entries. Flood starts.
Dupicate bugzilla entries appear, which developers have to deal with. Multiple topics on forums, crowds on IRC channels, all asking the same:

"I updated xxx to version yyy and now zzz fails". I have USE=a b c -d -g.

Sometimes decision is simple and found fast, sometimes it`s to say "we know about this problem and work on it. Please,be patient.

So, this is the process we can easily(i beleive =)automate.

Here is how i imagine work of such tool:

User emerges something, it fails. Currently emerge simpy reports about an error and quit. But this is not what we want.
Now we want portage to ask us:
Do you want to check problems database for problems on this package?
If we tell it 'yes', portage collects some _simple_ info about our system: arch, portage version,package version,use flags for this package (possibly deps/it`s use) and
sends it to the server.

On the server side, report is analysed by quite simple algorithm: it searches on given keys for assigned problem. For example we know that this problem happens
to package xxx of version yy when flags a b enabled on arch i386, so if client info matches, we send him some decision ( provided by developers: do update, rollback or wait);
If problem not found, we tell: oops, you`re lucky to have new problem, please send report to bugzilla OR:

we can add this information to database and mark it as 'new'. But in this case we need some program which will regulary scan reports (by package ) and search for common criteria. So,
it will be possible to automatically detect common criterias in case of many reports.
(i.e. if we have 200 reports for package xxx, which all have some use flag and version same, there`s a high possibility of this combination being a problem)

ttuttle · Post by **ttuttle** » Wed Oct 25, 2006 11:39 pm

`VL wrote:I think that some sort of such a system would be usefull for developers to downgrade rate of user`s requests in case of common problems on compilation of some package.

What we have now? If there`s a _common_ problem, people start to come to irc,post in forums, fill bugzilla`s entries. Flood starts.
Dupicate bugzilla entries appear, which developers have to deal with. Multiple topics on forums, crowds on IRC channels, all asking the same:

"I updated xxx to version yyy and now zzz fails". I have USE=a b c -d -g.

Sometimes decision is simple and found fast, sometimes it`s to say "we know about this problem and work on it. Please,be patient.

So, this is the process we can easily(i beleive =)automate.

Here is how i imagine work of such tool:

User emerges something, it fails. Currently emerge simpy reports about an error and quit. But this is not what we want.
Now we want portage to ask us:
Do you want to check problems database for problems on this package?
If we tell it 'yes', portage collects some _simple_ info about our system: arch, portage version,package version,use flags for this package (possibly deps/it`s use) and
sends it to the server.

On the server side, report is analysed by quite simple algorithm: it searches on given keys for assigned problem. For example we know that this problem happens
to package xxx of version yy when flags a b enabled on arch i386, so if client info matches, we send him some decision ( provided by developers: do update, rollback or wait);
If problem not found, we tell: oops, you`re lucky to have new problem, please send report to bugzilla OR:

we can add this information to database and mark it as 'new'. But in this case we need some program which will regulary scan reports (by package ) and search for common criteria. So,
it will be possible to automatically detect common criterias in case of many reports.
(i.e. if we have 200 reports for package xxx, which all have some use flag and version same, there`s a high possibility of this combination being a problem)

You have hit the nail on the head. This is exactly my idea! Basically, it's a way to collect statistics and help developers see patterns among error reports. So they won't be left chasing dupes on bugzilla, they'll be seeing a nice aggregated display of what problems people are having.

bernieb · Post by **bernieb** » Thu Oct 26, 2006 1:18 am

I'm somewhat skeptical of all this, mostly because emerge should not be interactive, and I can't see how this could be possible if this needs to prompt the user about whether to send or not.

However, I think there are a few ideas that would be quite useful--

It would be nice to provide an option to have portage automatically resume and skipfirst upon a failure. Basically, if one package fails, to keep trying all other packages and report a summary of what succeeded and what failed afterward.

Also, it would be great to have portage dump all output surrounding an error to a file for later reporting. I can't count how often I had to re-emerge a package, knowing it will still fail, just because the error managed to exceed the terminal buffer.

dobysirius · Post by **dobysirius** » Thu Oct 26, 2006 10:52 am

bernieb wrote:I'm somewhat skeptical of all this, mostly because emerge should not be interactive, and I can't see how this could be possible if this needs to prompt the user about whether to send or not.

However, I think there are a few ideas that would be quite useful--

It would be nice to provide an option to have portage automatically resume and skipfirst upon a failure. Basically, if one package fails, to keep trying all other packages and report a summary of what succeeded and what failed afterward.

Also, it would be great to have portage dump all output surrounding an error to a file for later reporting. I can't count how often I had to re-emerge a package, knowing it will still fail, just because the error managed to exceed the terminal buffer.

About the interactiveness, it could be configurable. You should choose whether it should try to auto-fix, or ask, or just die and maybe tell you what you can do.

About the auto skipfirst, something like this should be enough (well, it won't show a summary, but that's not hard to do):

Code: Select all

emerge -uDN world || until emerge --resume --skipfirst; do true; done

About the error dump to a file, put this in your make.conf:

Code: Select all

PORT_LOGDIR=/var/log/portage

The former is a bit of a "hack", of course, I agree it would be nice if there was a portage FEATURE or command-line option to make it do that.

pjv · Post by **pjv** » Thu Oct 26, 2006 1:50 pm

Seems useful enough. Give it a go. The portage devs are behind on implementing features that most users already use though. It will also give the debuggers a lot of forced work: can they handle it!?

Reminds me of gnome's bug buddy too, which is a great idea. However I've never understood how bug buddy actually sends things (it uses procmail or something like that, but why not my evolution-configured POP email address??? procmail doesn't seem intended for normal users) so I've never been able to use it.

ping-uino · Post by **ping-uino** » Thu Oct 26, 2006 8:04 pm

pjv wrote: Reminds me of gnome's bug buddy too, which is a great idea. However I've never understood how bug buddy actually sends things (it uses procmail or something like that, but why not my evolution-configured POP email address??? procmail doesn't seem intended for normal users) so I've never been able to use it.

from gnome 2.16 wrote: New Bug-buddy backend

GNOME's bug reporting tool, Bug-buddy, now uses an XML-RPC protocol and no longer requires that the user have sendmail installed. As a result of this change, projects using Bug-buddy must have the correct information in their application's .desktop file.

In my humble opinion it should be usefull for collection information, at the same way of mozilla agent or bug buddy.
Focus this kind of utility on users may be an alternative solution to bugzilla or forums; but require a lot of work of developers.
Maybe a new figure could handle the collection, not a developer and not a user... a sort of power user.

And, of course not installed by default (or a flag to avoid installation)... I believe this kind of tool is usefull only for desktop pc.

pjv · Post by **pjv** » Thu Oct 26, 2006 8:16 pm

In my humble opinion it should be usefull for collection information, at the same way of mozilla agent or bug buddy.
Focus this kind of utility on users may be an alternative solution to bugzilla or forums; but require a lot of work of developers.
Maybe a new figure could handle the collection, not a developer and not a user... a sort of power user.

And, of course not installed by default (or a flag to avoid installation)... I believe this kind of tool is usefull only for desktop pc.

Ok thx, I should look into bug buddy again now it doesn't need sendmail anymore.

`VL · Post by **`VL** » Thu Oct 26, 2006 8:36 pm

'm somewhat skeptical of all this, mostly because emerge should not be interactive, and I can't see how this could be possible if this needs to prompt the user about whether to send or not.

This is the easyiest of all tasks. User should just set one option in config to tell portage what he wants:
a) automatically search for bugs on failed package
b) be prompted to search
c) never be prompted

possibly command-line option should have more priorty than config one.

However, I think there are a few ideas that would be quite useful--
It would be nice to provide an option to have portage automatically resume and skipfirst upon a failure. Basically, if one package fails, to keep trying all other packages and report a summary of what succeeded and what failed afterward.

The idea is not about successfull emerge of failed package. The idea is to collect data about failed once and inform developers faster than bugzilla ( because adding new bug into bugzilla it`s not a trivial task - you have to think =), you have to search for simular bugs, you have to give info about your pc, you should check your conifg, you should not be lazy....) and inform back users if this problem is already investiaged and has a decision. Also such tool gives us some usefull statistics to analyze.

Corona688 · Post by **Corona688** » Thu Oct 26, 2006 10:27 pm

dobysirius wrote:About the interactiveness, it could be configurable.

Already is. See the --ask flag.

nkmcc · Post by **nkmcc** » Fri Oct 27, 2006 4:54 am

`VL wrote: The idea is to collect data about failed once and inform developers faster than bugzilla ( because adding new bug into bugzilla it`s not a trivial task - you have to think =), you have to search for simular bugs, you have to give info about your pc, you should check your conifg, you should not be lazy....) and inform back users if this problem is already investiaged and has a decision. Also such tool gives us some usefull statistics to analyze.

Just thought I'd point out that thinking is a good thing. The more users think, check their config, learn about their pc, do research etc.... the better Gentoo will become. Trying to automate that "thinking" process is not only a large amount of work for developers, as has already been pointed out, it also robs the community of "thinking" users, which would undoubtedly decrease the quality of Gentoo as a whole.

so even if a system like this would work, and wasn't much work for the developers, *please* don't do something that takes the responsibility of finding a solution for problems off of us as users. The forums are a pretty good example of how we as users can actually contribute to Gentoo by sharing our troubles and solutions. Trying to automate that would be a shame.

my 2 cents,
nkmcc

`VL · Post by **`VL** » Fri Oct 27, 2006 9:01 pm

nkmcc wrote: Just thought I'd point out that thinking is a good thing. The more users think, check their config, learn about their pc, do research etc.... the better Gentoo will become.

Well.. you gave us premise and a conclusion. But in reality conclusion can be opposite. Imagine: the more users f**k with their system, more they hate it and blame it on forums.
I would never do presumptions about people. They are ALL different.Some people just want to use software, some people want to improve it. Both groups have it`s right exist.

Trying to automate that "thinking" process is not only a large amount of work for developers, as has already been pointed out, it also robs the community of "thinking" users, which would undoubtedly decrease the quality of Gentoo as a whole.

Thinking can be split into 2 categories: usefull and useless. If someone broke compiler, trying to understand why i can`t compile my favourite music player is not usefull. It`s just waste of time. Of course, compiler developer needs help in testing.All developers need help. Always. And one man is unable to help them all. Man has his own goals in this life, and they are not to help all developers every time =). Creating barriers for users is the best way to drive them out.So instead of community of 'thinking' users we will have nothing.

so even if a system like this would work, and wasn't much work for the developers, *please* don't do something that takes the responsibility of finding a solution for problems off of us as users. The forums are a pretty good example of how we as users can actually contribute to Gentoo by sharing our troubles and solutions. Trying to automate that would be a shame.

So, what our goals for gentoo? Create community of 'thinking' users or create an ideal OS ? I had to visit this page: http://www.gentoo.org/main/en/philosophy.xml

Every user has work they need to do. The goal of Gentoo is to design tools and systems that allow a user to do that work as pleasantly and efficiently as possible, as they see fit. Our tools should be a joy to use, and should help the user to appreciate the richness of the Linux and free software community, and the flexibility of free software. This is only possible when the tool is designed to reflect and transmit the will of the user, and leave the possibilities open as to the final form of the raw materials (the source code.) If the tool forces the user to do things a particular way, then the tool is working against, rather than for, the user. We have all experienced situations where tools seem to be imposing their respective wills on us. This is backwards, and contrary to the Gentoo philosophy.

Put another way, the Gentoo philosophy is to create better tools. When a tool is doing its job perfectly, you might not even be very aware of its presence, because it does not interfere and make its presence known, nor does it force you to interact with it when you don't want it to. The tool serves the user rather than the user serving the tool.

The goal of Gentoo is to strive to create near-ideal tools. Tools that can accommodate the needs of many different users all with divergent goals. Don't you love it when you find a tool that does exactly what you want to do? Doesn't it feel great? Our mission is to give that sensation to as many people as possible.

Daniel Robbins
Previous Chief Architect

friedmud · Post by **friedmud** » Sun Oct 29, 2006 11:54 pm

I have thought of a similar system before... but for slightly different reasons...

I think it would be great of Portage could report back to a server the packages it has just installed... and any errors it encountered. My reasoning for reporting _everything_ is that it would be useful for knowing when ebuilds are stable enough to un-keyword them.

If _everyone_ is un-keywording a specific package and the installation of that ebuild is going smoothly then that should be an indication that that specific ebuild has stabalized and should be unmasked.

On the flip side if the package fails to install this will be reported and recorded... and is an indicator that more work needs to be done on that ebuild.

I could see just adding a couple of fields to packages.gentoo.org... showing how many times the ebuild has been merged and the success rate (Maybe break this out per architecture). This would be useful to both users and developers. As a user if I go to packages.gentoo.org and look for the new firefox release and see that it is keyword masked but has been installed successfully by thousands of users on my same architecture then that gives me some confidence in trying it myself.

To get back on topic a bit... I think that a system like what the OP proposed is just too much.... TOO much info will be coming in and it won't be able to be managed. Instead I think something like what I am proposing is a lot more manageable.... aggregate stats are the best you can hope to do. With all the hundreds of thousands of Gentoo installations out there... if they were all sending back bug reports every single time an emerge failed... there would be _no_ way to get anything useful out of the data. You can say that a computer will try to find similarities and whatnot... but as a Computer Scientist I know that this is a lot harder to achieve than you think.

Friedmud

artbody · Post by **artbody** » Sat Nov 04, 2006 10:34 pm

Yes, but turn it off by default.

If you want to do such a work - do it
As a Kernelfeature ? Ok it's a long way home.

So first way I agree with the developers

May be such a thing has to be a feature offered by the developer of his package.
a framework to use such a feature seams to be a good idea
so let's think about a framework

who wants use this framework

Most developers didn't code for one special disribution

nobody includes this feature

On the other way if all developers know about the framework... you'll have done a great work

Perhaps I use it as Tester

djay · Post by **djay** » Wed Nov 08, 2006 4:23 pm

friedmud wrote:I have thought of a similar system before... but for slightly different reasons...

I think it would be great of Portage could report back to a server the packages it has just installed... and any errors it encountered. My reasoning for reporting _everything_ is that it would be useful for knowing when ebuilds are stable enough to un-keyword them.

If _everyone_ is un-keywording a specific package and the installation of that ebuild is going smoothly then that should be an indication that that specific ebuild has stabalized and should be unmasked.

No, wrong. Un-keywording a package means setting it to some wacky "stable" state (it needs to be defined yet what exactly "stable" is). But stable package doesn't mean it "emerges" ok, it should work more or less stable.

friedmud wrote: I could see just adding a couple of fields to packages.gentoo.org... showing how many times the ebuild has been merged and the success rate (Maybe break this out per architecture). This would be useful to both users and developers. As a user if I go to packages.gentoo.org and look for the new firefox release and see that it is keyword masked but has been installed successfully by thousands of users on my same architecture then that gives me some confidence in trying it myself.

Now this is really interesting idea.

friedmud wrote: I think that a system like what the OP proposed is just too much.... TOO much info will be coming in and it won't be able to be managed. [....] With all the hundreds of thousands of Gentoo installations out there... if they were all sending back bug reports every single time an emerge failed... there would be no way to get anything useful out of the data. You can say that a computer will try to find similarities and whatnot... but as a Computer Scientist I know that this is a lot harder to achieve than you think.

Indeed. True in every word.

Besides, can anyone tell me that 2 ebuilds failures are really 2 different ones? even if sets of USE flags are different? The error might be the same for both and caused by something installed on the system and not by specific USE flags used at the moment .

Such automatic process would generate much more duplicates than a manual one.

friedmud · Post by **friedmud** » Wed Nov 08, 2006 4:38 pm

No, wrong. Un-keywording a package means setting it to some wacky "stable" state (it needs to be defined yet what exactly "stable" is). But stable package doesn't mean it "emerges" ok, it should work more or less stable.

You are right that being un-keyworded has more to do with how it runs than rather or not it installs... but I contend the two things are not entirely seperate....

How do you quantify _when_ something is stable? I contend that you look at the number of people that have successfully installed the package compared to the number of bug reports you have for said package.

For instance, if only 5 people have successfully installed a package... and you also have 5 bug reports for that package.... then it's obviously not stable. On the other hand... if tens of thousands of people have installed a package and the bug reports are few or minor then I believe you could say that package has "stabilized" and is ready for general use.

Without knowing how many people have used the package... it's impossible to tell from looking through bugzilla how stable the package is. A ratio of users/bugs would essentially be an indicator of stability. Something package maintainers could use in judging when a package is fit for general use. Sure, it shouldn't be the only criterion... but it would definitely help!

Friedmud

djay · Post by **djay** » Wed Nov 08, 2006 9:17 pm

friedmud wrote:
No, wrong. Un-keywording a package means setting it to some wacky "stable" state (it needs to be defined yet what exactly "stable" is). But stable package doesn't mean it "emerges" ok, it should work more or less stable.
You are right that being un-keyworded has more to do with how it runs than rather or not it installs... but I contend the two things are not entirely seperate....

How do you quantify _when_ something is stable? I contend that you look at the number of people that have successfully installed the package compared to the number of bug reports you have for said package.

For instance, if only 5 people have successfully installed a package... and you also have 5 bug reports for that package.... then it's obviously not stable. On the other hand... if tens of thousands of people have installed a package and the bug reports are few or minor then I believe you could say that package has "stabilized" and is ready for general use.

Without knowing how many people have used the package... it's impossible to tell from looking through bugzilla how stable the package is. A ratio of users/bugs would essentially be an indicator of stability. Something package maintainers could use in judging when a package is fit for general use. Sure, it shouldn't be the only criterion... but it would definitely help!

Friedmud,
in a miracle gentoo gist, your prayers are answered.

Apparently, there's an application being developed, which's purpose it to collect different data from your system.

So hopefully we will see something sooner than later.

friedmud · Post by **friedmud** » Wed Nov 08, 2006 10:07 pm

Sounds great!

Do you have any details? Do they need help?

Thanks!

Friedmud

ccx · Post by **ccx** » Fri Nov 10, 2006 1:04 am

The feature of suggesting similar bugs seems enough to be interesting, it might save lot of googling / bugzilla searching.
If users themselves could enter comments and/or solutions it would be great, as we can't force devs to manage one more bugzilla (and a larger one).
The point is we can auto-search the bugs for duplicates if we have some unified-style information about the user's system.
It could be done right now by using some script to extract system info after incomplete merge, make quazilion google searches, compare them ...
Maybe scanning and parsing gentoo bugzilla could be also done, as atleast emerge --info is usually there.

Actually it would be maybe more search engine than a bugzilla

Of course building self-comparing bugzilla with this huge database would require lot of resources and effort, for something that is probably small gain.

Genone · Post by **Genone** » Fri Nov 10, 2006 7:43 am

friedmud wrote:Do you have any details? Do they need help?

Not at this time.

jcat · Post by **jcat** » Mon Nov 13, 2006 12:43 pm

This sounds good to me, even if it's just used to collect stats about portage problems. Gets my vote!

Cheers,
jcat

Dralnu · Post by **Dralnu** » Sat Nov 18, 2006 9:26 pm

Good idea, but I want the option to turn it off, and limit the info given.

To help with the massive amount of error reports, maybe see about filtiering it by CFLAGS (if they are insane, then send a reply saying error not sent: Try sane CFLAGS), and then what does get through, check their checksums (whatever that one thing is that I've forgoteen the name of that tells you if a download worked or failed? You know the one I think) to help reduce duplicate entries, and simply increase the # reported (say in the heading: Portage 3.0 break - reported: 8053498534596 times), and then on the machinehaving troubles, keep a list of the errors sent, and require a cool-down time of X minutes/hours/days before resubmitting.

As for finding the solution, send it back via email (they should have a LiveCD handy, anyways), and help reduce the post in the forums some.

BTW: I voted to have it turned off by default. It may be a good idea to turn it on by default if you can accuratly tell if the user is new or not, but since that is at best difficult, keep it off until the user decides to send it in. Tying it in with PORTAGE_ELOG would probably help, as well, just to send a dup of the report to the sysadmin so they know what is going on.

jmorganson · Post by **jmorganson** » Wed Dec 20, 2006 11:00 pm

Any more info on the status on this idea or the development of it?

gdoubleu · Post by **gdoubleu** » Sat Feb 17, 2007 2:06 am

jmorganson wrote:Any more info on the status on this idea or the development of it?

??

This seems like a great idea to me. It doesn't have to be as complex as the OP suggests though. Start off small and build from there. I would think that even a simple number of successes and failures could be helpful.

jmorganson · Post by **jmorganson** » Sat Feb 17, 2007 5:07 pm

I thought so too. I have been working on developing this. If there is as much interest as is shown on this thread then there might be other user and developer interest in it also. I found too that it in not just an issue limited to Gentoo, but all of Linux. The first step that I am working on in building this system is a daemon that collects errors from the system and sends them to an online server. That part is relatively simple. The next thing that this will need though is a server that will take these reports 24-7. If there are still people listening to this thread I would be greatful for any assistance on this aspect.

ccx · Post by **ccx** » Mon Feb 19, 2007 11:00 pm

I may provide my gentoo box that is on 100Mbps academic connection.
I program in Python,C,C++,C# and Java a bit, but ATM I'm too busy to help coding.

Idea: Gentoo Automated Error Reporting/Resolution Client

Do you think an error reporting system is a good idea?