new search stopwords list

[Albert_Alligator]

Hi guys and gals,

I'm not here to add fuel to the fire, but I believe that it would be prudent to have the list of omitted words on your quick search page. This will at least save searchers some time and frustration wondering why half or all of their search terms were omitted.

Gentoo can't scale vertically because the expense is much to great...the secret is to scale horizontally. GASLAMP servers or (Grid Application Server on Linux Apache MySQL and PHP) can answer your search problems. All you need to do is balance your search load over several very low cost systems. You don't need a dual or quad processor system to handle all the queries. You need around 5 to 10 300 dollar systems put together as a GASLAMP. The computers in this system do not have to be located in the same facility, hence you can reduce the expense by asking partners or already cooperating Universities to pitch in.

Before you ask, I can't help just yet (however, in this year, my budget could very well quadruple which would give me much more discretionary spending cash and open the door to such a project....just not yet.)

I do know that there are plenty of people out there with the experience and know-how that would probably help if someone in charge here with Gentoo would just ask. Open source it!!!

Just some random thoughts.

Al

[[Lx]-=Mystify=-]

hm get's me to the problem that i can't search for 'config.guess' while i have a problem with that file during compilation...


sorry, but this function caused me more troubles than i see benefits in it...

[arnolde]

http://www.google.de/search?q=config.gu ... gentoo.org

I wish they would add next to the quicksearch a googlequicksearch field... ("seerch this site with google") cant be that complicated...

[[Lx]-=Mystify=-]

@arnolde:

yeah, very good idea...

wish i had it myself

[arnolde]

wish i had it myself

So do I

[ZamZing]

I moved to Gentoo a couple of months ago and I must say that the most frustrating thing for me as a noob was searching through the forums for answers. Gobs of precious time was consumed parsing through hundreds of irrelevant posts because I could not search for an exact phrase. I thought I needed special codes to search but seeking help on that by searching for phrases similar to "Forum search" was of no help, and there was no obvious indication anywhere that the system was purposely crippled.

I hope the Gentoo folks will remedy this situation soon. Its hard enough for new users to learn the intricacies of Gentoo without the additional frustration of ineffective forum search tools.

Ian... that beta still on the way? I wholeheartedly agree with an earlier post that a Google sitesearch button is an excellent fix until a useable permanent solution if found.

C'mon guys, help us make better use of the awesome store of information in these forums!

...ZamZing

[Earthwings]

Yes, it's still being worked on, but it takes time because a) it has to be tested thoroughly and b) recreating indices / converting stuff (which is part of it) alone takes days.

[PeGa!]

I accidentally saw that "screen" word is blocked (listed in the stoplist). I suggest for this word to be released, because i had a problem with screen command, and really it caused a great pain to solve my problem without searching that word.

Thanks in advance,

Paul.

[DarkMind]

very bad the filter, i search for "package.provided" and filters this and search for "provided" only... this is NOT util for me

[NaiL]

there are some words usefull for search.. like "default".

i think that

We've analyzed the most commonly occurring words on the forums

is not a valid mechanism.

[tomk]

there are some words usefull for search.. like "default".

i think that

We've analyzed the most commonly occurring words on the forums

is not a valid mechanism.

When the new stopword list was added we had to remove the most common words as the size of the search index was a massive hit on search performance, to the point that you couldn't search. When we add the new search engine we'll be having another look at which words will be in the stopwords list.

[tedj]

Can you at least tell us what it would take to have a unfiltered search? Something along the lines of "if someone donates $500 for more memory" or whatnot would be nice.

[devsk]

I hope someone realises that this is counter-productive. What's happening because of this filtering is that no relevant search results are being returned to the users. Hence number of duplicate threads with the same keywords are getting increased everyday, hence putting more load on the database. In this case it has become counter-productive. I would rather have all the results returned in a slow fashion to me than to find out that I posted and someone flamed me for not searching hard enough.

[Chaosite]

Can you at least tell us what it would take to have a unfiltered search? Something along the lines of "if someone donates $500 for more memory" or whatnot would be nice.

I think "If someone donates a good search algorithm that can be easily integrated with phpbb"... =/

[yaneurabeya]

What database method do you guys use-assuming that you run MySQL-MyISAM or Innodb?

[tomk]

What database method do you guys use-assuming that you run MySQL-MyISAM or Innodb?

Currently we're using InnoDB, although the new search engine will have some MyISAM tables.

[renrutal]

Are 2-3 years old posts that aren't in gentoo-wiki any useful for anyone searching for recent problems?

Ate least try to make the 1-year option in the search the default one.

[GentooMik]

Their is an issue where typing in exact error messages do not return any good results. It is understandable that not enough resources. What is not understandable is how to fix this? That is having relevant results and same time minimize the load. So far it is getting harder and harder to find answers without duplicate posting. Personally, I would try to avoid double posting. But if using the search engine does not bring relevant results it has to be posted again. This may have been rehashed and sorry for adding more. However, this is not really whining just re-bringing
reemphasizing" attention for the admins. That I think their is relevancy issues with more stop words being added. Although looking at that list some words are amateurish and OK. But sometimes simple words just add up. Hope others agree with me that the relevance problems exist and not my imagination.

P.S: Using googles site search for the forums is not always updated and not intusive.

See ya around in the gentoo land.

[manther]

I think "compile" and "error(s)" should be taken out, after all this is gentoo not SuSE

The reason these words are on the list is that they are too commonly appearing to actually be of use in identifying a particular thread. There are so many posts with the words 'compile' or 'error' that it's not a useful descriptor. If I were trying to describe myself to you so you could pick me out of a crowd at an amusement park I would want to avoid a description such as "medium height with blue jeans, sneakers and a tshirt" because it wouldn't really tell you anything that would set me apart from virtually everyone else. This is essentially the kind of description you get when searching for the words 'compile' and 'error'.

I'm sure the guys at google would have something else to say about that. Leaving words out like 'compile' just makes other words useless. If I want to know how to 'compile kernel 2.6.18' I don't want to see every entry in the database having to do with the kernel, without filtering out the ones that don't even mention the word 'compile'. You're just turning it around, so instead of searching for a guy in 'blue jeans', you're just searching for the guy in a 'red polo shirt', of which there could be just as many, only most of them are wearing khaki's or something. It's still just as hard for the user to search through.... There has got to be a better solution to your problem.

How about working on a solution that uses certain key words/phrases from a users new post and automatically searches for them, bringing up a set of possible matches for what the user is looking for on the preview page. Then if the user sees something useful, they won't hit the submit button, and won't post the duplicate.

[DaysSinceTheDoor]

I would just like to say I have used gentoo for since 2003 when I was in college. I have moved on to a nice job doing web application development. It took me about four months to convince my boss that Gentoo was the perfect platform to run our systems on. Now it is about three months later, and we are running into Gentoo specific problems. Searching these forums for error messages turns up jack because half of the words get dropped from the search. Now I can understand the whole search index DB getting huge problem as I have set up site crawlers in the past. However in this case I would say get a better server and get a better search algorithm. I know that there are ones out there that do not index on words like "a" , "but" , "the" , etc but when it gets back the search for the terms not on the black list it searches through the search results for the dropped words and moves those to the top. The lack of good result for errors is basically causing these forums two major problems. First it is causing duplicate posts which inturn leads to monitors having to move posts and point out that issues have been discused and possibly solved elsewhere wasting their time and resources. And second it is removing Gentoo's crediblity as a good plateform for business. My boss is seriously thinking of switching the servers over to SuSE because of the lack of good support. My boss (like i hope many others who use them) is usually very good about supporting open source projects with donations of both time and money but only when he has faith in a project. Luckily someone here pointed out that google lets you do site:forums.gentoo.org and I am now getting good search results for our errors. I seriously suggest rethinking the search on site and finding a better option, or your problems will likely get worse.

[Element Dave]

I am in utter disbelief at the decision to exclude terms from the search index based on their frequency. Only the USER himself can determine relevancy of the results returned; administrators cannot. A far better solution to the problem would have been to disable searching altogether until it can be resolved without limiting functionality. The current method only reinforces the stereotype that Gentoo users are "penny wise and pound foolish": search engine speed may have improved, but it takes much longer to find the desired information.

As others have pointed out, the current implementation is also deceptive to its users because they are not informed that their search terms have been ignored. At the very least, it should be made clear on the search form that various words are ignored, and a link to the list should be provided.

[marijuana]

i dunno why, probably this things to make unsearchable some keywords does it's part, but.. the search engine of this forums.. really SUCKS.
i don't wanna flame.. but all we know that this is true.
is this directly riconducible to this keywords?
please.. do a second or a third or an n-th thougt.. in this forums there are very important information.. but often unreachable. i can use keywords site:forums.gentoo.org on google but more updated posts may be missing.
is important that if all we spend our time posting asking and answering questions our work help more people as possible.
if the problem isn't just this.. please.. do something. it's frustrating the message: no result found while the forum is full of answers.
-> the account is virgin.. but i'm a long time gentoo user and i always found forum search unproductive.

[devsk]

there are two problems with the current approach:

1. the relevancy of a stopped word in the context of the other search words is doomed this way. Its plain wrong to exclude any search word just because they occur too many times.
2. the duplicate posts are adding to the server load and index sizes, not reducing them.

Someone needs to restore sanity and bring the usefulness of these forums back.

[Satori80]

the relevancy of a stopped word in the context of the other search words is doomed this way. Its plain wrong to exclude any search word just because they occur too many times.

That's pretty much what I said two years ago. It hasn't gotten any better since then.

I never thought I'd hear myself say this but; God, I wish these forums were slow! If that's the downside of bringing back the functionality, so be it. At least this would be useful again.

Admins: I understand you are looking for an alternative, and I really do appreciate the significant amount of work that entails. But this stop word list is completely unreasonable even as stopgap measure.

And I still don't see a warning near any search input area that one's search isn't likely to produce anything viable. When a user sees the words "Search for all terms" they really do expect the engine to "search for all terms." Strangely enough. Common guys. It's been more than long enough for someone to at least issue a prominent warning for the poor user.

[Raenk]

I just wanna add my voice to those petitions. Search engine works terrible. Hence duplicated threads or no solutions found whatsoever.

Lots of information are being hidden and wasted. Gentoo community information has been always top notch, but now is getting more and more unreachable.