Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How about: The Gentoo antispam initiative?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Off the Wall
View previous topic :: View next topic  
Author Message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Wed May 28, 2003 11:04 pm    Post subject: How about: The Gentoo antispam initiative? Reply with quote

Most anti-spam programs will fail. This is obvious since they use statistics and they will never reach a probability of one to stop all spam and let all legit email through.

There is one thing that works though and chances are you're not too familiar with it since it is currently made for MS Outlook only. It is Cloudmarks Spamnet. I joined when it was beta, now it is final and they've started asking for money. But more on this later.

It works like this: 1.000.000 people install the program. You receive a spam. The program generates an unique key to that email (avoiding personal wirds like names) and compares this key with an online database. If enough people have received this email and clicked on the "spam" button, it is labeled as spam and you can choose to have it deleted ot moved to another folder or whatever.

I don't remember when I installed it but it reports that it has blocked 1885 spams for me personally and I've reported perhaps 20 of them myself. Now, when they have reached what propbably is "critical mass", they start charging for the service and the moment my beta stops working I'll stop using the service. Not that it's not worth paying for, but because I'm emmigrating to linux and perhaps there is a better solution.

Be warned: I'm no programmer so I have little to no idea how difficult this would be to implement.

Wouldn't it be great PR for gentoo (or linux) if someone made a similar program for Gentoo? Is it possible to implement this on a lower level than what Cloudmark is doing making it independent on mail client?
In the beginning, just to test this out, the database could be accessed through the gentoo mirror servers. You will create quite a lot of hits but the bandwidth requirements are low since keys are generated and I will assume that in principle, 1 or 2 kbs should be more than enough to uniquely separate emails.

If this proves to be a success, some smart heads can try to make a distributed database...spreading the load over thousands of users, each contributing with what little they have. You don't need to be online all the time, you don't need an ûber-fast internet connection, just willing to help the community.

You see, the beauty with this program is that you will never ever mark a legit mail as spam because you need a certain amount of positive feedback before it is labeled as spam.

We've all read reports on how spam is costing businesses this and this much money each year in lost time, well, what would happen to gentoo or linux if we could say "Use Gentoo and get Spam-free!"

The reason I'm using gentoo as an example is because I think the portage system is ideal for this kind of program and this kind of system can only survive in a non-commercial community.

Am I making myself clear? Your thoughts on this?

Cheers,
Odegard
Back to top
View user's profile Send private message
nonhuman
Apprentice
Apprentice


Joined: 30 Sep 2002
Posts: 236
Location: Washington DC

PostPosted: Wed May 28, 2003 11:24 pm    Post subject: Reply with quote

I think that sounds like a really cool idea. Next fall and winter I'll be working on my "Integrative Exercise" for my Computer Science major (like writing a thesis, but code :D). My group's project is to write a spam filter. We have no model or plan for how it will work yet, but this one sounds really cool, and I'll definitely propose it (though I don't know how feasible it will be for our purposes). It will have to be closed (for obvious reasons ) for the first 26 weeks or so of development until we finish the version that we submit for grading, but after that, with the approval of the rest of my group, I was planning on opening it up. And letting the rest of the world give it a go and see what comes out of it. It will definitely run on Linux, as that's the primary platform for my school's CS department, and I, at least, would definitely approve of our project turning into the official Gentoo or Linux spam filter. If nothing else, I'll certainly have plenty of experience to lend to such an effort if people really want to go forwards with it.
_________________
"Ambition is a poor excuse for not having sense enough to be lazy." -Edgar Bergen
Back to top
View user's profile Send private message
dolbz
Tux's lil' helper
Tux's lil' helper


Joined: 29 Mar 2003
Posts: 138
Location: Nuneaton,UK

PostPosted: Wed May 28, 2003 11:25 pm    Post subject: Reply with quote

This sounds a great idea to me. can I just clarify a point in my understanding though?

When you say that if a mailing has a high enough rating then it is moved to another folder or whatever does that mean locally or on the pop server? If it is local then the spam will still have to be downloaded which sometimes takes a while to do. but at least not reading it will help.

I think this is something that developers should look at seriously because it could benefit thousands of internet users. Also if the majority of the interface is kept on the web then clients could be made for all platforms and mail programs and ultimately has the power to eliminate spam (I know it wont happen but still dreaming is nice). Any developers out there interested. I wish I'd really tried to learn C properly I would have been able to help lol oh well best to leave it to the proffesionals.

Dolbz
Back to top
View user's profile Send private message
Vancouverite
Apprentice
Apprentice


Joined: 28 Sep 2002
Posts: 162
Location: Vancouver, Canada

PostPosted: Thu May 29, 2003 12:01 am    Post subject: Reply with quote

This is an interesting idea. I am expecting that future email clients will have integrated bayesian features and feel this is a good solution combined with whatever anti spam is running on the mail server.
_________________
Screenshot
Back to top
View user's profile Send private message
Krookednek
Tux's lil' helper
Tux's lil' helper


Joined: 20 Feb 2003
Posts: 78

PostPosted: Thu May 29, 2003 2:26 am    Post subject: Reply with quote

I think it sounds cool too, sadly I dont have sufficient knowledge to implement the idea, but I support anyone who does.
_________________
Linux n00b, beware!
Back to top
View user's profile Send private message
aja
l33t
l33t


Joined: 26 Aug 2002
Posts: 705
Location: Edmonton, Canada

PostPosted: Thu May 29, 2003 3:39 am    Post subject: Reply with quote

Ummmm.....It's been done.

Vipul's Razor: http://razor.sourceforge.net/

Works great when combined with SpamAssasin ( http://spamassassin.rediris.es/index.html )
and procmail
Back to top
View user's profile Send private message
Naan Yaar
Bodhisattva
Bodhisattva


Joined: 27 Jun 2002
Posts: 1549

PostPosted: Thu May 29, 2003 4:04 am    Post subject: Reply with quote

Furthermore, SpamNet is essentially a Razor2 equivalent for Windows. Also, there are ebuilds for both spamassassin and razor; the former automatically detects and uses the latter. The success rate is very good.
aja wrote:
Ummmm.....It's been done.

Vipul's Razor: http://razor.sourceforge.net/
...
Back to top
View user's profile Send private message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Thu May 29, 2003 10:14 am    Post subject: Reply with quote

Ah great it already exists! I will certainly try it out as soon as I get the chance.

Have anyone tried it? Does it work? Then WHY haven't I heard about this!?!?!?!? (Becuase I'm stupid, but thats beside the question) This bit of software could very well be THE argument to run linux in the future for big businesses...
Back to top
View user's profile Send private message
Naan Yaar
Bodhisattva
Bodhisattva


Joined: 27 Jun 2002
Posts: 1549

PostPosted: Thu May 29, 2003 3:18 pm    Post subject: Reply with quote

Spamassassin + razor2 work quite well in my experience. I have had few (~2-5%) false negatives, i.e., spam classified incorrectly as non-spam, and only one false positive over a few months of usage. The false positive was due to spamassassin and not razor.

There are about 16 other threads in the forum discussing spamassassin and razor, BTW.
odegard wrote:

Have anyone tried it? Does it work? Then WHY haven't I heard about this!?!?!?!? (Becuase I'm stupid, but thats beside the question) This bit of software could very well be THE argument to run linux in the future for big businesses...
Back to top
View user's profile Send private message
schmuck
n00b
n00b


Joined: 21 May 2002
Posts: 31
Location: sthlm, sweden

PostPosted: Thu May 29, 2003 4:15 pm    Post subject: Reply with quote

odegard wrote:
Have anyone tried it? Does it work? Then WHY haven't I heard about this!?!?!?!? (Becuase I'm stupid, but thats beside the question) This bit of software could very well be THE argument to run linux in the future for big businesses...


Spamassassin rules! I've been using it for six moths now with zero false negatives!
I have it setup via procmail = nice and easy.

regards /jens
_________________
rules do not apply to me
Back to top
View user's profile Send private message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Thu May 29, 2003 10:58 pm    Post subject: Reply with quote

Thanks for the enlightment. The point with spamassassin is that you can never be sure you're not having false negatives while with Razor you can sleep well at night...
Back to top
View user's profile Send private message
bluz
n00b
n00b


Joined: 18 Aug 2002
Posts: 61
Location: Canada

PostPosted: Mon Jun 02, 2003 12:49 am    Post subject: Spamassasin Reply with quote

I'm using Spamaassin w/Evolution (cmd line, no server support) and I seem to be getting quite a few false positives. It actually catches nearly every SPAM that comes in, but also seems to be wrong about 1 in 5 times.

Is this just tweaking that I need to do? I admit, I haven't had a chance to go through the doc's in real depth but I was hoping I didn't need to! ;)

BlUz
Back to top
View user's profile Send private message
bulletman
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2002
Posts: 126

PostPosted: Mon Jun 02, 2003 1:24 am    Post subject: Reply with quote

In your .spamassassin/user_prefs file, you can add domains/persons that you want to mark as ham. Add a line like:

whitelist_from *@something.com

From the file comments:

# Whitelist and blacklist addresses are now file-glob-style patterns, so
# "friend@somewhere.com", "*@isp.com", or "*.domain.net" will all work.

-- Stephen
_________________
Stephen

If your desktop gets out of control easily,
you probably have too much stuff on it that
doesn't need to be there.
Donna Smallin, "Unclutter Your Home"
Back to top
View user's profile Send private message
jlg
Guru
Guru


Joined: 31 May 2002
Posts: 360
Location: Montreal, CANADA

PostPosted: Mon Jun 02, 2003 2:52 am    Post subject: Reply with quote

The easiest and most efficient way to vanish spam is to have mail servers only accept email from email addresses which actually exist. Spam is coming 99.9999999999999999999% of the time from non-routable email addresses...

Personaly I have solve the problem a long time ago.. I made a python script that connects to any pop server and deletes any email that is coming from addresses which do not contain an email address or keyword from my list.
Back to top
View user's profile Send private message
AkiAki007
Apprentice
Apprentice


Joined: 07 Feb 2003
Posts: 150
Location: New York, NY

PostPosted: Tue Jun 03, 2003 5:52 pm    Post subject: Response Required Reply with quote

Well, I had this idea (apparently I wasn't the only one since I saw a website with this soon after I mentioned my idea to someone) is that the person sending me email (if not on my address book) will be sent a "Response Required" email. All they have to do is hit send again and I will get the email and they will be added to my Address Book. Of course I can do this myself, by going through my "Pending Approval" stack - which is also the junk mail stack.

Anyway, this is rather easy to do. You maintain an Address Book within the program (say Evolution or Mozilla Mail, or even in sendmail), and your program (plugin) will fire up every time you get a mail and then compare the sender to what you already have in your address book. If the user exists, then you get the mail right away, and if the user does not exist, then the email is put into some database or another mail folder, and the sender is notified via email that they have to hit reply to the email they've just received. Once they do this, the plugin/script will go into the "junk" or "pending" folder and simply move the mail to your Inbox and also at the same time add that user to your Address Book.

Simple, 100% effective. I haven't had the time to do this yet, but the script and program for this is rather easy to do. Most of the database and folder stuff is already installed on the user machine and all you have to do is write a simple script that will do the grunt work and stuff like that. It would be super easy to do if done via MySQL or another database (Perl flatfile or whatever) so lookups would be really fast. This would make a nice plugin for both MozillaMail and for Evolution.
Back to top
View user's profile Send private message
jlg
Guru
Guru


Joined: 31 May 2002
Posts: 360
Location: Montreal, CANADA

PostPosted: Tue Jun 03, 2003 6:35 pm    Post subject: Reply with quote

eh :? AkiAki007 no your not the only one with that idea.. my program does that also... its a simple python scripts that I wrote 2 years ago.. oh! and you don't need a database for such a simple program.. flat files are more than enough.. unless you have 100 000 records in your address book LOL!!!

PS: do a search in google or freshmeat and I'm sure you'll find plenty of others that do just the same...
Back to top
View user's profile Send private message
punkki
n00b
n00b


Joined: 04 Jun 2003
Posts: 1
Location: Finland

PostPosted: Wed Jun 04, 2003 8:19 am    Post subject: Re: Response Required Reply with quote

AkiAki007 wrote:
the person sending me email (if not on my address book) will be sent a "Response Required" email. All they have to do is hit send again and I will get the email and they will be added to my Address Book. Of course I can do this myself, by going through my "Pending Approval" stack - which is also the junk mail stack.


Although this might be useful for some, there are serious drawbacks that crops up if widely used:
- you need to remember whitelist mailing lists beforehand (many moderators just delete the subscribers that use confirmations like this and forget to whitelist)
- if the sender uses similar system there is real danger that he never sees your "response required" message
- many (me included) are not willing to jump thru hoops just to send you e-mail (for example an answer to your important question in the Usenet news)
- if you don't rate limit the response required messages you let your system to be abused for mail bombing (small message to your script expands to larger messages destined to the victim, the victim might have blocked the attacker already, but not your responder)

There has been lots of discussion about this kind of schemes in various on-line discussion forums, for example in spam-l mailing list, asrg mailing list at ietf, anti-spam at yahoo to name a few.

There are commercial services that implement this as well.
Back to top
View user's profile Send private message
punkki
n00b
n00b


Joined: 04 Jun 2003
Posts: 1
Location: Finland

PostPosted: Wed Jun 04, 2003 8:26 am    Post subject: Reply with quote

aja wrote:
Vipul's Razor: http://razor.sourceforge.net/


Another one is DCC (http://www.rhyolite.com/anti-spam/dcc/ ), which works with SpamAssassin as well.

esa
Back to top
View user's profile Send private message
Naan Yaar
Bodhisattva
Bodhisattva


Joined: 27 Jun 2002
Posts: 1549

PostPosted: Wed Jun 04, 2003 12:16 pm    Post subject: Re: Response Required Reply with quote

Already available here. See also Slashdot stories regarding Earthlink and Mailblocks featuring similar schemes.
AkiAki007 wrote:
Well, I had this idea (apparently I wasn't the only one since I saw a website with this soon after I mentioned my idea to someone) is that the person sending me email (if not on my address book) will be sent a "Response Required" email. All they have to do is hit send again and I will get the email and they will be added to my Address Book...
Back to top
View user's profile Send private message
quag7
Apprentice
Apprentice


Joined: 12 Aug 2002
Posts: 288
Location: Marana, Arizona - USA

PostPosted: Wed Jun 04, 2003 1:51 pm    Post subject: Reply with quote

...And I've been using Bogofilter, a Bayesian spam filter, set up to work with Evolution. For me, this has worked surprisingly well. I used to maintain a list of keyword-based filters, which was a hassle and kind of primitive, but in just a week of using Bogofilter, I'm getting far better filtration.

I use an external SMTP/POP server that someone else runs so it was more practical for me to filter right at the client.

Some quickie information for how to do this is here; and it's especially easy since Bogofilter is in portage.

http://www.ime.usp.br/~rsilva/bogo-and-evo/

I have a kind of messy spam situation here, getting > 150 spams a day (it's an old e-mail address), and on average after a week of "teaching" Bogofilter I get about 5-7 unfiltered spams in my inbox per day, plus zero false positives so far. I just filter off things for mailing lists, and I have a very small whitelist first, after which Bogofilter deals with the rest. This way I don't have to rely on a whitelist as some of the most interesting mail I get is from strangers.

This is a simple solution for people who grab their mail from servers they don't admin. Takes all of ten minutes to set up too, and you can use the URL above even if you don't use Evolution to get an idea of how to run it with other mail clients.
_________________
http://www.dataswamp.net
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Off the Wall All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum