Gentoo forum search sucks

[madpenguin8]

Anyone else think that the gentoo forum search sucks. It comes up with just about nothing relevant to my search. It seems like really popular posts that have absolutely nothing to do with what I am searching for always turn up close to the top. Writing up a dupe sucks, but what can a guy do if search turns up garbage.

[rac]

Some tips: make sure your search terms are all 3+ letters long. Join all your terms with 'and' or check "search for all terms", otherwise the default is "or", which is probably not what you want. Use many anded search terms, and widen the search later. Try to choose obscure words if you can: unusual words that appear in an exact error message are usually very good. Things like "bug", "portage", "broken" are bad.

Do you have any recommendations for improving the search engine while still being able to upgrade phpBB in a timely fashion?


EDIT: To consolidate some search threads, I'm including the following quote from another thread. -- pjp

Here's how the phpBB search function works. Each post is split into words. First, some characters are replaced. There are three classes of characters here, those that get replaced by spaces, those that get elided, and those that get left alone. Next, whitespace is used to delineate words. All words of less than 3 or more than 20 characters are dropped. Then an entry is made in the dictionary table for every word that is not in the dictionary, so that it can be referenced by number. An entry is made in a colossal table for each and every word in each and every post. That's what gets searched against.

To get back to Reformist's two examples, gnome2 is a word. 'gnome 2' is two words, one of which is impossible to match because it is one character long. "1.1.0" is three words, each of which are impossible to match, because they are one character long. One modification that it might be feasible to make would be to change the status of '.'. If it were left alone, version numbers would become searchable. However, words at the end of sentences, followed by periods, would become unsearchable, because a separate entry would be made including the period. If it were elided, the end-of-sentence problem would go away, but then you would have to search for "abiword and 110", and "2.1" would become "21" and fall under the three-character limit.

[PARENA]

Here's an example of a (what I think) good search. Let's say you want to know about using the slit in Openbox:

openbox AND slit

(almost) guaranteed to come up with your answer. Unless it's not on the forum of course.

[idl]

would be cool if we could do the old "kde3.1 emerge failure" note the "" that makes sure it matches the whole string. like on google

[rac]

would be cool if we could do the old "kde3.1 emerge failure" note the "" that makes sure it matches the whole string. like on google

That, while a nice feature, is completely impossible with the current way the search databases work, because the search match tables have fields for word number and post id only. There is no sense of what words occur next to one another.

[Matje]

would be cool if we could do the old "kde3.1 emerge failure" note the "" that makes sure it matches the whole string. like on google

That, while a nice feature, is completely impossible with the current way the search databases work, because the search match tables have fields for word number and post id only. There is no sense of what words occur next to one another.

It _is_ possible, by making php look into the actual post texts. Would be a query like:
SELECT post_id FROM phpbb_posts_text WHERE post_text LIKE '%$searchstring%'
But this'll butcher the mysql I think with these number of posts on the forum

[xlyz]

Join all your terms with 'and' or check "search for all terms", otherwise the default is "or", which is probably not what you want.

please make "AND" default. "OR" is seldom used

[gsfgf]

Join all your terms with 'and' or check "search for all terms", otherwise the default is "or", which is probably not what you want.

please make "AND" default. "OR" is seldom used

at least for the quick search.

[pilla]

Moving to Gentoo Forum Feedback.

[Matje]

Join all your terms with 'and' or check "search for all terms", otherwise the default is "or", which is probably not what you want.

please make "AND" default. "OR" is seldom used

at least for the quick search.

Indeed, I'd like that too, would be a great improvement.

[David_Escott]

I have the compelling urge to say the following:

THIS IS A DUPLICATE THREAD.
PLEASE SEARCH

http://forums.gentoo.org/viewtopic.php?t=30782
was the closest I could get, searching was a pain because well it sucks (sorry that was a duplicate thought)
But I do remember seeing one thread where rac may have explained some of the difficulties in trying to make search work a little better, if only I could find it again :(

Ahhh found it
http://forums.gentoo.org/viewtopic.php?p=167526#167526 down the page rac explains some of the difficulties

[rac]

Thanks for those links, David_Escott. As a journey of a thousand miles begins with a single step, I have changed the default setting of the checkbox in the search screen to "Search for all terms". Hopefully this will improve things in some small measure. Note that this change only affects the purple "gentoo" theme - people still using subSilver will not be affected.

[xlyz]

quick search is still "search for any words" as default

are you going to change it as well?

TIA

[rac]

Apparently phpBB was caching the template file, so my changes to quicksearch weren't taking hold. It should be fixed now.

[gsfgf]

Rac i noticed in yopur other post that version #s aren't searchable. If you convert 2.2.2 to 222 and make search strip periods as well so if you search for kde 3.1 search will treat it as kde 31, that would solve that issue. That may be harder than it looks, though.

[rac]

If you convert 2.2.2 to 222 and make search strip periods as well so if you search for kde 3.1 search will treat it as kde 31, that would solve that issue. That may be harder than it looks, though.

It is harder than it looks, for a couple of reasons. Either a period causes a word break or it doesn't. Now what would be best is if it caused a word break only if it wasn't a version number, but that could be a challenging regex. Maybe we could steal it from Portage. I think if we're going to go this far, we might as well get it right and have "version numbers" go into the index.

I just had an idea on how this might be implemented. Details later.

[phong]

Not that tough of a regex... For matching (in a generic way) words of 3+ characters and version numbers, I might use the following for matching words to put into the index (offhand, could be more robust):
(\b\w{3,}\b|\b\d(\.\d){1,2}\b)

What's the current regexp?

[Lion]

I think there is still something basically wrong with search.
I always try to include as many relevant search terms in my query, but still I do not get posts that I know to be available.

Simple example: Search for the word 'world'.
'No topics or posts met your search criteria'.
I know this is not true, because the world file is mentioned in many posts.
Search for 'world AND file'.
Thousands of posts, many of which do NO contain the word 'world'.

So, my question is: what am I doing wrong?

[rac]

So, my question is: what am I doing wrong?

Being unlucky. To help keep the size of the search tables down, there is a "stopword list" in phpBB's search function. Words on the stopword list are not indexed because they are too common. Unfortunately for your example, world is on the stopword list, so nothing shows up, and then when you search for "world and file", you are really only searching for 'file'.

[pjp]

Maybe emerge, portage, gentoo, gnome and alot of others should be added to the list

[xlyz]

So, my question is: what am I doing wrong?

Being unlucky. To help keep the size of the search tables down, there is a "stopword list" in phpBB's search function.

what are the words included in the list?

[rac]

what are the words included in the list?

http://forums.gentoo.org/language/lang_ ... pwords.txt

[dufeu]

what are the words included in the list?

http://forums.gentoo.org/language/lang_ ... pwords.txt

I assume that gentoo bugzilla has a similar stopword list? It certainly would explain some of my difficulties in using search there.

Do you have an inkling (and could you share the location) of the search stopword list there?

[rac]

I assume that gentoo bugzilla has a similar stopword list?

Bugzilla's completely different software. I don't know off the top of my head whether there's a stopword list. If I get some time I may look into it further.

[edoloughlin]

So, my question is: what am I doing wrong?

Being unlucky. To help keep the size of the search tables down, there is a "stopword list" in phpBB's search function. Words on the stopword list are not indexed because they are too common. Unfortunately for your example, world is on the stopword list, so nothing shows up, and then when you search for "world and file", you are really only searching for 'file'.

Perhaps a lesson could be learned from Google. Stopwords are identified if they are included in a search, viz:

"the" is a very common word and was not included in your search