I use iptables with the 'recent' module to catch brute force attempts
I don't have access to my system now, but it's something similar
# $IPTABLES -A INPUT -p tcp -i eth0 --dport 22 -m state --state NEW -m recent --name sshprobe --set -j ACCEPT
# $IPTABLES -A INPUT -p tcp -i eth0 --dport 22 -m state --state NEW -m recent --name sshprobe --update --seconds 60 --hitcount 3 -j LOGDROP # LOGDROP is a chain that first logs the attacker and then drop him.
This is limited to sropping someone from retyring ssh brute force, but it can be generalized. A google with 'iptables recent' showed me this link:
http://www.stearns.org/doc/adaptive-fir ... rrent.html
Which shows an example of an attacker of a mail server being blocked for several minutes after 3 attempts.
Hope this helps