Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Efficiently use a live blacklist feed in iptables (w/ ipset)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Sat Feb 05, 2011 8:11 am    Post subject: Efficiently use a live blacklist feed in iptables (w/ ipset) Reply with quote

[Edit: Nov. 2011: Updated the 3 main scripts to ipset-6 syntax]
[Edit: Nov. 2012: Update to incorporate code improvements suggested by Truc]

Background on ipset (if you're familiar with it, skip this):

I'm not affiliated with the netfilter team. I'm writing this simply because there's not much instructive documentation (e.g. tutorials, how-tos) available pertaining to the use of ipsets, so I'm sharing what I've learned by playing with it, hopefully it will save people some time.

ipset is an extremely useful plugin to iptables, particularly if you want to have a firewall rule that matches against a large set of addresses and/or ports, or if you want to dynamically change the addresses and/or ports that a rule matches against.

It lets you create huge lists of ip addresses and/or ports (with tens of thousands of entries or more) which are stored in a tiny piece of ram with extreme efficiency. In your iptables rules, you can then simply refer to the lists by name, and the entire list is checked with remarkable speed and in a single netfilter rule. Also, you can change the contents of the list while the firewall is running.

iptables is actually just the user interface to netfilter. When you write an iptables rule that lists multiple addresses or ports to be blocked (other than "from-to" ranges or netmasks), each item in the list is actually translated into its own netfilter rule, each of which must be processed. Unnecessary processing of this type can really slow down your traffic, since every new connection (or in some cases, every packet) has to be processed through these rules.

Although it possible to add and remove iptables rules while the firewall is running, it is not very efficient to do so. This is how most firewalls (i.e. iptables "front-ends") add and remove entries to their "dynamic blacklist". Blacklist a new address? That's a new rule that must be processed for every connection. You can't "edit" an iptables rule in a running firewall to change the addresses or ports that it matches. Ipset is, in my opinion, by far the best way to run blacklists, whether static, periodically updated from the Internet, or dynamically managed by your firewall or intrusion detection rules.

Also, while iptables provides for "address ranges" and port ranges (in a from-to, netmask, or CIDR format), it cannot efficiently handle long lists of non-contiguous addresses. Try writing iptables rules to block 3,000 specific addresses that have nothing in common. With ipset, this is trivial. It's also efficient, because you can check a packet against the ipset in a single netfilter action (as opposed to 3,000) much, much faster.

Ipset is made by the same netfilter team that makes iptables. It is actually a collection of additional netfilter-related kernel modules (additional "matches" and "targets"). It is part of the Xtables add-on package. In Gentoo, you can get it simiply by emerging ipset (after your "make modules_install") and enabling the appropriate kernel modules.

There are different types of ipsets for storing different types of information (e.g., for random ip addresses, for addresses that are from the same netblock, for random blocks of addresses, for same-sized blocks of addresses, for random ports, etc.).

You can group multiple ipsets of different types into a single "setlist" (an ipset of type list:set) which is still treated as a single ipset by iptables (therefore, you only need one rule to match packets against multiple ipsets). You can bind ipsets together (e.g. a list of addresses and a list of ports). The man page explains it all.

Here is some basic information, and I will provide a couple of practical examples of managing a basic ipset for use in a firewall.
http://ipset.netfilter.org/ipset.man.html#index

An example of updating an ipset in an automated fashion, from an internet source:

There are many ways to use ipsets. This script is just an example. This script is intended to periodically (hourly) update an ipset used as a "blacklist" in a firewall (while the firewall is running and actively processing). The list in this case is a list of "Class C"-sized networks (i.e. CIDR /24 blocks of addresses), published hourly by DShield.org, listing the top blocks of networks from which port-scanning activity has been coming in the last 3 days. It's a tiny list as ipsets go, but it serves the purpose of this example (ipsets can contain many thousands of entries).

That's all incidental. What's important is that it's a list of /24 netblocks, and we want to blacklist them. There is a type of ipset called an "iphash" (hash:ip) that is very efficient and handling lists of same-sized networks (i.e., the ipset efficiently contains a list of networks that have the same netmask, in this case /24), so we'll use that hash:ip type of ipset. We'll use wget to retrieve the block list only if it's been updated. Then, if we've got a new list, we'll load it up to the firewall. Since we can instantaneously swap the contents of one ipset for another, that's how we'll update the live firewall -- we'll parse each address out of the downloaded block list and add it to a temporary ipset, then swap the contents of the temporary ipset into the live ipset in the running firewall, instantly updating the firewall's blacklist en masse.

In later posts, there are other examples that are variations on the theme, demonstrating the use of different types of ipsets and other related concepts. This, however, is good place to start.

This script runs hourly by cron on my system, about five minutes after DShield publishes its hourly update (presently, between HH+15m - HH:20m). For logging purposes it uses "logger", which you can install, or you can substitute what you want or modify the logging lines to simply echo to your log file.

Code:

#! /bin/bash

# /usr/local/sbin/block
# BoneKracker
# Rev. 11 October 2012
# Tested with ipset 6.13

# Purpose: Load DShield.org Recommended Block List into an ipset in a running
# firewall.  That list contains the networks from which the most malicious
# traffic is being reported by DShield participants.

# Notes: Call this from crontab. Feed updated every 15 minutes.
# netmask=24: dshield's list is all class C networks
# hashsize=64: default is 1024 but 64 is more than needed here


target="http://feeds.dshield.org/block.txt"
ipset_params="hash:ip --netmask 24 --hashsize 64"

filename=$(basename ${target})
firewall_ipset=${filename%.*}           # ipset will be filename minus ext
data_dir="/var/tmp/${firewall_ipset}"   # data directory will be same
data_file="${data_dir}/${filename}"

# if data directory does not exist, create it
mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format
# stderr redirected in case file is not present
get_timestamp() {
    date -r $1 +%m/%d' '%R
}

# file modification time on server is preserved during wget download
[ -w $data_file ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have
wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then
    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
    exit 1
fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"
    ipset create ${temp_ipset} ${ipset_params}

    networks=$(sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' ${data_file})

    for net in $networks; do
        ipset add ${temp_ipset} ${net}
    done

    # if ipset does not exist, create it
    ipset create -exist ${firewall_ipset} ${ipset_params}

    # swap the temp ipset for the live one
    ipset swap ${temp_ipset} ${firewall_ipset}
    ipset destroy ${temp_ipset}

    # log the file modification time for use in minimizing lag in cron schedule
    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."

fi


I run it using cron (I use vixie-cron, so I put this in /etc/cron.d):
Code:
# /etc/cron.d/update_block
# BoneKracker
# 31 March 2011

# Global variables
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
MAILTO=root
HOME=/

# Every 15 minutes, poll for update to dshield
# block list, and update firewall blacklist.

# To check timing: grep "block updated" /var/log/crond
# Last I checked it was being published at:
# H+:08                                 
# H+:23
# H+:38
# H+:53

09 *  * * * root    /usr/local/sbin/block
24 *  * * * root    /usr/local/sbin/block
39 *  * * * root    /usr/local/sbin/block
54 *  * * * root    /usr/local/sbin/block


Last edited by Bones McCracker on Fri Oct 12, 2012 3:21 am; edited 19 times in total
Back to top
View user's profile Send private message
ssteinberg
Apprentice
Apprentice


Joined: 09 Jul 2010
Posts: 206
Location: Israel

PostPosted: Sat Feb 05, 2011 9:41 am    Post subject: Reply with quote

This is actually really interesting, well done. I'd be interested in more feeds.
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Sat Feb 05, 2011 6:26 pm    Post subject: Reply with quote

Here's another. This one is used to accurately match (block or allow) traffic from or to entire countries.

This script takes a single command-line argument: the 2-letter IANA country code (the top-level domain, such as "us" or "cn"). Based on that, it downloads a list of all the networks of various sizes that are registered to that country code with the regional registrar. Last I checked, these lists are published twice daily by the ipdeny.com project. These are larger lists than the previous example.

Like the list in the previous example, this is a list of networks. However, unlike the previous example, they are networks which vary in size (i.e., have various netmasks). For that reason, we will use the nethash type of ipset (formally now called hash:net, although referring to it the old way still works).

Selecting the appropriate hashsize: In the example above, where the list is always a specific size and very small, I specifically set the hashsize to something smaller than the default. We will not do that here, because the default hashsize (1024) is appropriate for all but the smallest of the country codes, and the hashsize is just a starting point anyway (as an ipset gets larger, the hashsize is adjusted upward dynamically). Setting the hashsize manually only makes sense when you know the approximate size of the ipset in advance, and the point of doing so is to: (a) save a few KiB of ram in the rare case where you are using a list you know is smaller than the default; or (b) to save a bit of rehashing (which occurs as the ipset grows) in the cases where you know the list is larger than the default. It is also possible to optimize this dynamic resizing process in various ways using the "--probes" and "--resize" options, but the defaults are fine, and we don't need to go into that now.

This script also demonstrates the use of of the setlist type of ipset. All of the country-specific ipsets are added to a single, combined setlist, so we can refer to the whole group as a single ipset (and matching against it is a single efficient netfilter action). In the script you'll see where the required setlist is created (if the setlist does not already exist) and where the ipsets are added to the setlist (if they are not already a member of the setlist).

Other than taking a command-line paramater, using a different type of ipset that is of more of a normal size, and employing a hierarchy of ipsets (i.e., using a setlist), this script is pretty much the same as the one above. I actually run this script twice a day from cron, about an hour after ipdeny.com updates the lists. I run it multiple times, fetching the lists for several countries and creating a corresponding ipset for each (these are just example countries here; no offense intended to anyone):
Code:
# /etc/cron.d/update_ipdeny
# BoneKracker
# 4 November 2011

# Twice per day, poll for update to ipdeny
# block lists, and update firewall blacklist.

# To check timing: grep "IPSet: .. updated" /var/log/crond.log
# Last I checked by running hourly, they were being published as follows:
# 05:05 - 05:07
# 15:05 - 15:07


countries="cn ru ir ng"

09  5 * * * root    for c in $countries; do /usr/local/sbin/ipdeny $c; done
09 15 * * * root    for c in $countries; do /usr/local/sbin/ipdeny $c; done


Those four ipsets could be used directly in iptables (for example, by using "+cn" anywhere an ip address would normally go), but as noted above, I have combined them into a single "setlist". That way, I can simply refer to all the networks in all the countries as "+ipdeny" (there are over 7,000 networks in the example I give here). To check if a packet matches any of those, Netfilter needs only execute a single action: one call to check the setlist, which responds with a match or no-match almost instantaneously.

In fact, using ipset in general is trivial, with the exception of initially gaining an understanding the different set types and what they are for (that takes a thorough reading of the man page).

So here is that second script:

Code:

#! /bin/bash

# /usr/local/sbin/ipdeny
# BoneKracker
# Rev. 11 October 2012
# Tested with ipset 6.13

# Purpose: Load ip networks registered in a country into an ipset and load that
# ipset into a setlist containing several such ipsets, while this setlist is
# being used in a running firewall.
#
# Notes: Call this from crontab. Feed updated about 05:07 and 15:07 daily.
#
# Usage: 'ipdeny <TLD>' (where TLD is top-level national domain, such as "us")


[ -n "$1" ] && firewall_ipset="$1" || exit 1
ipset_params="hash:net"
filename="${firewall_ipset}.zone"       # on server, files are "us.zone" etc.
target="http://www.ipdeny.com/ipblocks/data/countries/${filename}"
data_dir="/var/tmp/ipdeny"
data_file="${data_dir}/${filename}"

# if data directory does not exist, create it
mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format
get_timestamp() {
    date -r $1 +%m/%d' '%R
}

# file modification time on server is preserved during wget download
[ -w ${data_file} ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have
wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then
    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
    exit 1
fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"
    ipset create ${temp_ipset} ${ipset_params}

    while read network; do
        ipset add ${temp_ipset} ${network}
    done < ${data_file}

    # if ipset does not exist, create it
    ipset create ${firewall_ipset} ${ipset_params} 2>/dev/null

    # swap the temp ipset for the live one
    ipset swap ${temp_ipset} ${firewall_ipset}
    ipset destroy ${temp_ipset}

    # if the setlist does not exit, create it
    ipset create -exist ipdeny list:set

    # if the ipset is not already in the setlist, add it
    ipset add -exist ipdeny ${firewall_ipset}

    # log the file modification time for use in minimizing lag in cron schedule
    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."
fi


Last edited by Bones McCracker on Fri Oct 12, 2012 3:59 am; edited 12 times in total
Back to top
View user's profile Send private message
ssteinberg
Apprentice
Apprentice


Joined: 09 Jul 2010
Posts: 206
Location: Israel

PostPosted: Mon Feb 07, 2011 2:34 pm    Post subject: Reply with quote

Thanks. I was aware of filtering based on country. Doesn't appeal to me personally as much as a dynamic block list, it is just too general, but people should find it useful. Well done with the scripts.
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Mon Feb 07, 2011 11:42 pm    Post subject: Reply with quote

ssteinberg wrote:
Thanks. I was aware of filtering based on country. Doesn't appeal to me personally as much as a dynamic block list, it is just too general, but people should find it useful. Well done with the scripts.

You are right. It serves to demonstrate the basics, though.

Beyond that, it's also easy to dynamically blacklist attackers, etc., by writing iptables rules that use the "SET" target (as opposed to the "set" match). The basic iptables options are:
--match-set (compare a packet to an ipset)
--add-set (add address to an ipset)
--del-set (remove address from ipset)

Gentoo provides a rudimentary initscript that saves ipsets upon shutdown and restores them on startup. Some firewall tools, such as Shorewall, have built-in facilities for loading and saving ipsets when the firewall is started or stopped. You can use those, or manage them yourself (just make sure they are loaded up before iptables).

This inistscript is specific to the init system used by Gentoo Linux, and other Linux distributions provide their own startup scripts. The latest version (as of this edit) accommodates setlist type ipsets (which much be destroyed before the sets they contain can be destroyed). If you are installing ipset manually, you can use this as a model.

Code:
#!/sbin/runscript

extra_commands="save"

IPSET_SAVE=${IPSET_SAVE:-/var/lib/ipset/rules-save}

depend() {
    before iptables ip6tables
    use logger
}

checkconfig() {
    if [ ! -f "${IPSET_SAVE}" ] ; then
        eerror "Not starting ${SVCNAME}. First create some rules then run:"
        eerror "/etc/init.d/${SVCNAME} save"
        return 1
    fi
    return 0
}

start() {
    checkconfig || return 1
    ebegin "Loading ipset session"
    ipset restore < "${IPSET_SAVE}"
    eend $?
}

stop() {
    # check if there are any references to current sets

    if ! ipset list | gawk '
        ($1 == "References:") { refcnt += $2 }
        ($1 == "Type:" && $2 == "list:set") { set = 1 }
        (scan) { if ($0 != "") setcnt++; else { scan = 0; set = 0 } }
        (set && $1 == "Members:") {scan = 1}
        END { if ((refcnt - setcnt) > 0) exit 1 }
    '; then
        eerror "ipset is in use, can't stop"
        return 1
    fi

    if [ "${SAVE_ON_STOP}" = "yes" ] ; then
        save || return 1
    fi

    ebegin "Removing kernel IP sets"
    ipset flush
    ipset destroy
    eend $?
}

save() {
    ebegin "Saving ipset session"
    touch "${IPSET_SAVE}"
    chmod 0600 "${IPSET_SAVE}"
    ipset save > "${IPSET_SAVE}"
    eend $?
}


Last edited by Bones McCracker on Sat Jan 07, 2012 1:22 am; edited 3 times in total
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Thu Feb 10, 2011 7:51 am    Post subject: Reply with quote

Here is another that's more practically useful. This script creates an ipset that can be used to block all bogons (not just rfc1918 private IP addresses, but every network block that is unassignable or has not yet been assigned by the regional authorities). These addresses are often used by botnets, spammers, and so on. It creates a fairly large ipset of a complex type, so it takes tens of seconds to initially load up the temporary ipset (the swapout operation is still virtually instantaneous, as are queries).

Code:

#! /bin/bash

# /usr/local/sbin/fullbogons-ipv4
# BoneKracker
# Rev. 11 October 2012
# Tested with ipset 6.13

# Purpose: Periodically update an ipset used in a running firewall to block
# bogons. Bogons are addresses that nobody should be using on the public
# Internet because they are either private, not to be assigned, or have
# not yet been assigned.
#
# Notes: Call this from crontab. Feed updated every 4 hours.


target="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"
ipset_params="hash:net"

filename=$(basename ${target})
firewall_ipset=${filename%.*}           # ipset will be filename minus ext
data_dir="/var/tmp/${firewall_ipset}"   # data directory will be same
data_file="${data_dir}/${filename}"

# if data directory does not exist, create it
mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format
get_timestamp() {
    date -r $1 +%m/%d' '%R
}

# file modification time on server is preserved during wget download
[ -w ${data_file} ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have
wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then
    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
    exit 1
fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"
    ipset create ${temp_ipset} ${ipset_params}

    #sed -i '/^#/d' ${data_file}            # strip comments
    sed -ri '/^[#< \t]|^$/d' ${data_file}   # occasionally the file has been xhtml

    while read network; do
        ipset add ${temp_ipset} ${network}
    done < ${data_file}

    # if ipset does not exist, create it
    ipset create -exist ${firewall_ipset} ${ipset_params}

    # swap the temp ipset for the live one
    ipset swap ${temp_ipset} ${firewall_ipset}
    ipset destroy ${temp_ipset}

    # log the file modification time for use in minimizing lag in cron schedule
    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."

fi

This is the crontab I use:
Code:
# /etc/cron.d/update_bogons
# BoneKracker
# 16 May 2011

# Every four hours, poll for update to ipv4-fullbogons
# block list, and update firewall blacklist.

# Last I checked by running hourly, it was being published as follows:
# 00:48 - 00:50
# 04:48 - 04:50
# 08:48 - 08:50
# 12:48 - 12:50
# 16:48 - 16:50
# 20:48 - 20:50
#
# To check timing:
# zgrep "fullbogons-ipv4 updated" /var/log/old_logs/cron*

# Global variables
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
MAILTO=root
HOME=/

52 0  * * * root    fullbogons-ipv4
52 4  * * * root    fullbogons-ipv4
52 8  * * * root    fullbogons-ipv4
52 12 * * * root   fullbogons-ipv4
52 16 * * * root   fullbogons-ipv4
52 20 * * * root   fullbogons-ipv4

_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.


Last edited by Bones McCracker on Fri Oct 12, 2012 3:28 am; edited 12 times in total
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Thu Feb 10, 2011 8:21 am    Post subject: Reply with quote

If anyone is wondering how I came up with the ${ipset_params}, the answer is trial-and-error. In the meantime, I'll share the limited understanding I have gained. I should also point out that I've been working exclusively with hash-type ipsets and the iptreemap, so my optimization insights do not extend to other types.

I would be appreciative of any tips regarding this by a more knowledgeable person, on how to identify optimal values for of these and similar ipset parameters.

When you build an ipset that is one of the "hash" types, do an 'ipset -L' and look at it's size. The default size is 1024 (which I assume to be bytes). If your list is small and you guess that a smaller hash could store it, or if the hashsize has been dynamically "grown" to be substantially larger (e.g. well into the multiple megabyte range), then you may want to try to optimize it (i.e., cause the process of loading the ipset to create a more efficient hash).

This is not necessary, since ipset will dynamically grow and rehash an ipset as entries are added (I haven't tested to see if it will dynamically shrink them as entries are removed, but I doubt it). When all is said and done, pursuing this optimization process might cut the size of an ipset in half, and since they typically stay resident in RAM, this can free up some tens of megabytes of RAM. Whether that's worth your time is up to you.

There may be a better way to do this, but I basically run the process of loading the ipset multiple times, with varying parameters, trying to get a smaller hash size. Living by the rule "default is good", and noting that there is typically a point of diminishing returns on varying from them, one can bracket one's way into a reasonable value without too much effort. Basically the process would be:

Code:
# ipset -X <your_ipset>
# ipset -N <your_ipset> <ipset_type> --hashsize <hash_size> --resize <resize_percent> --probes <probes>

See the man page for an explanation of the parameters. I have found that all three can have an impact of the resulting size of the hash.

Then you'd run your loop that loads up the ipset. It may be useful to "time" this, to test if varying parameters makes loading up the ipset take more or less time. For example:
Code:
time while [ $((--i)) -ge 0 ]; do /sbin/ipset --add temporary_ipset ${networks[i]}; done

Then look at the resulting ipset's hash size:
Code:
# ipset -L <your_ipset> | head
Back to top
View user's profile Send private message
mimosinnet
Guru
Guru


Joined: 10 Aug 2006
Posts: 525
Location: Barcelona, Spain

PostPosted: Sun Apr 22, 2012 9:53 pm    Post subject: Reply with quote

I find these scripts really useful. I am learning some perl and I have made a perl version of them. I started with perl on Christmas, as a hobby, so do not expect too much of this version. In the the exercise, I have learned some perl, some bash, and really enjoyed deciphering the regex expressions, still a mystery to me!

Thanks for this great scripts and the introduction to ipsets!
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Sun Apr 22, 2012 11:32 pm    Post subject: Reply with quote

Glad you found them useful, and thanks for the credit on your page and in your scripts. I'd be interested to know how they compare in terms of performance.
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
mimosinnet
Guru
Guru


Joined: 10 Aug 2006
Posts: 525
Location: Barcelona, Spain

PostPosted: Tue Apr 24, 2012 5:40 pm    Post subject: Reply with quote

BoneKracker wrote:
I'd be interested to know how they compare in terms of performance.


Thanks for the suggestion! I have created a test.sh script that calls the three bash scripts:
Code:
# cat test.sh
#!/bin/bash
./Ipset_Dshield.sh
echo "Dshiedl"
./Ipset_bogons.sh
echo "Bogons"
./Ipset_Regions.sh cn
./Ipset_Regions.sh vn
echo "Regions"
/etc/init.d/ipset save


and executed both the test.sh and Ipset.pl scripts with the command 'time'. These are the results:

Quote:
./test.sh 0,65s user 0,87s system 16% cpu 9,023 total
./test.sh 0,76s user 0,93s system 17% cpu 9,876 total
./test.sh 0,71s user 0,90s system 17% cpu 9,295 total
./test.sh 0,62s user 0,88s system 16% cpu 8,900 total
./test.sh 0,70s user 0,97s system 16% cpu 9,826 total
./test.sh 0,70s user 0,91s system 17% cpu 9,414 total
./test.sh 0,67s user 0,87s system 16% cpu 9,122 total
./test.sh 0,63s user 0,98s system 17% cpu 9,320 total

perl Ipset.pl 0,71s user 4,85s system 37% cpu 14,729 total
perl Ipset.pl 0,58s user 3,83s system 31% cpu 14,147 total
perl Ipset.pl 0,54s user 3,65s system 30% cpu 13,902 total
perl Ipset.pl 0,97s user 5,75s system 43% cpu 15,618 total
perl Ipset.pl 1,09s user 6,01s system 45% cpu 15,623 total
perl Ipset.pl 0,84s user 5,11s system 40% cpu 14,851 total
perl Ipset.pl 0,88s user 4,85s system 39% cpu 14,536 total
perl Ipset.pl 0,92s user 5,40s system 42% cpu 14,919 total


The bash script tends to be faster and uses less resources than the perl script. ( Bash 1, Perl 0! ;-) )

This could also be because I am practicing Object Oriented Perl, and loading Moose library for this.

Cheers!

Update: I have rewritten the script without using Moose. These are the results:
Code:

perl Set_ip.pl  0,40s user 2,20s system 15% cpu 16,770 total
perl Set_ip.pl  0,50s user 2,62s system 12% cpu 24,782 total
perl Set_ip.pl  0,51s user 2,44s system  8% cpu 36,539 total
perl Set_ip.pl  0,43s user 2,07s system  5% cpu 44,494 total
perl Set_ip.pl  0,35s user 2,00s system 11% cpu 20,693 total
perl Set_ip.pl  0,49s user 2,30s system  8% cpu 31,933 total
perl Set_ip.pl  0,51s user 2,19s system  4% cpu 59,530 total
perl Set_ip.pl  0,48s user 2,43s system  8% cpu 33,630 total


Moose is the guilty one! ;-)
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved.
Thank the community answering other people's post, specially those unanswered.


Last edited by mimosinnet on Wed Apr 25, 2012 10:20 am; edited 1 time in total
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Tue Apr 24, 2012 8:02 pm    Post subject: Reply with quote

Interesting.
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
truc
Advocate
Advocate


Joined: 25 Jul 2005
Posts: 3199

PostPosted: Fri May 25, 2012 8:00 am    Post subject: Re: Efficiently use a live blacklist feed in iptables (w/ ip Reply with quote

BoneKracker wrote:
(snip...)


Intersting lecture thanks, I've been meaning to try ipset for a while now, guess it's the time:)

Anyway, a few comments on your script though.


Code:
# if data directory does not exist, create it
/bin/mkdir -m 0750 ${data_dir} 2>/dev/null


You should probably use
Code:
/bin/mkdir -m 0750 -p "${data_dir}"
instead(note the -p and the absence of stderr redirection)

Code:
# function to get modification time of the file in log-friendly format
get_timestamp() {
    timestamp=$(/bin/date -r ${data_file} +%m/%d' '%R 2>/dev/null)
}
# file modification time on server is preserved during wget download
get_timestamp
old_timestamp=${timestamp}

This is not really important here, but it is counter intuitive that get_timestamp actually defines the timestamp variable.

You could probably use something like this instead(and again, why hidding stderr? it's there for a reason, it helps debugging when something goes wrong)
Code:
get_timestamp() {
   /bin/date -r "$1" +'%m/%d %R'
}
# file modification time on server is preserved during wget download
old_timestamp=$(get_timestamp "$data_file")


Last comment, in the code below, there is no need to use bash arrays (and if you don't, your script is no longer pure bash but pure sh which is somewhat better since you share it)
Code:

   networks=( $(/bin/sed -rn 's/(^([0-9]{1,3}.){3}[0-9]{1,3}).*$/\1/p' ${data_file}) )

   i=${#networks[*]}
   while [ $((--i)) -ge 0 ]; do
      /usr/sbin/ipset add ${temp_ipset} ${networks[i]}
   done


Code:

   networks=$(/bin/sed -rn 's/(^([0-9]{1,3}.){3}[0-9]{1,3}).*$/\1/p' "${data_file}")

   for net in $networks; do
      /usr/sbin/ipset add ${temp_ipset} ${net}
   done


Two more things about it:
=> your regex is wrong you really want to match a . (dot) with \. and not any character with .
Code:
networks=$(/bin/sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' "${data_file}")

And here is how I would say it, I find it easier to understand, but that's personnal opinion;)
Code:
networks=$(/bin/sed -rn '/^([0-9]{1,3}\.){3}[0-9]{1,3})/ { s/[^0-9.].*//; p }' "${data_file}")

=> And, in the code above you don't really need to use the variable networks:
Code:

   for net in $(/bin/sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' "${data_file}"); do
      /usr/sbin/ipset add ${temp_ipset} ${net}
   done


or even
Code:

   /bin/sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' "${data_file}" | while read net; do
      /usr/sbin/ipset add ${temp_ipset} ${net}
   done

_________________
The End of the Internet!
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Thu Oct 11, 2012 5:01 am    Post subject: Reply with quote

Truc, it took me a long time to get back to this, but thank you for the excellent corrections and suggestions. I am incorporating all of them (with one exception, and one in modified form, as below).

Exception: I got rid of the BASH array as you suggested (don't know what I was thinking there), but I left the $networks variable in place because I think it's more apparent what's going on that way.

Modified: The stderr redirection in the get_timestamp function was there to allow processing to continue if the file is not present (which is a normal condition on first run and when the user might purge the data file from /var/tmp). I took it out per your suggestion, so I made the assignment of "old_timestamp" (which calls the function) conditional on the presence of the data file.

Thank you.
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
elmar283
Apprentice
Apprentice


Joined: 06 Dec 2004
Posts: 195
Location: Netherlands

PostPosted: Mon Oct 15, 2012 7:02 pm    Post subject: Reply with quote

Thank you for the script. When I tried it I got the next error:
Code:

elmarotter@masterserver /etc/cron.d $ sudo /usr/local/sbin/block
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument
ipset v6.8: Kernel error received: Invalid argument


I might have forgotten some kernel function. My iptables works fine, so that can't be it.
This is my .config: http://eotter1979.xs4all.nl/bestanden/forums.gentoo.org/config
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2246

PostPosted: Mon Oct 15, 2012 8:16 pm    Post subject: Reply with quote

elmar283 wrote:
I might have forgotten some kernel function. My iptables works fine, so that can't be it.
This is my .config: http://eotter1979.xs4all.nl/bestanden/forums.gentoo.org/config

elmar283 ... you need ipset enabled and your .config shows: CONFIG_IP_SET is not set

best ... khay
Back to top
View user's profile Send private message
elmar283
Apprentice
Apprentice


Joined: 06 Dec 2004
Posts: 195
Location: Netherlands

PostPosted: Tue Oct 16, 2012 7:56 pm    Post subject: Reply with quote

Thank you. It works fine now! :)
Back to top
View user's profile Send private message
Rain91
n00b
n00b


Joined: 03 Aug 2004
Posts: 14

PostPosted: Sat Nov 03, 2012 5:59 pm    Post subject: Thanks for this! Reply with quote

Could you post the updated script?
Back to top
View user's profile Send private message
elmar283
Apprentice
Apprentice


Joined: 06 Dec 2004
Posts: 195
Location: Netherlands

PostPosted: Sat Nov 03, 2012 6:04 pm    Post subject: Reply with quote

I changed the script so I could put it in my cron width 'crontab -e'.
This are my working scripts:
Code:

elmarotter@masterserver ~ $ cat /usr/local/sbin/block

#! /bin/bash
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
MAILTO=root
HOME=/

# /usr/local/sbin/block
# BoneKracker
# Rev. 11 October 2012
# Tested with ipset 6.13

# Purpose: Load DShield.org Recommended Block List into an ipset in a running
# firewall.  That list contains the networks from which the most malicious
# traffic is being reported by DShield participants.

# Notes: Call this from crontab. Feed updated every 15 minutes.
# netmask=24: dshield's list is all class C networks
# hashsize=64: default is 1024 but 64 is more than needed here


target="http://feeds.dshield.org/block.txt"
ipset_params="hash:ip --netmask 24 --hashsize 64"

filename=$(basename ${target})
firewall_ipset=${filename%.*}           # ipset will be filename minus ext
data_dir="/var/tmp/${firewall_ipset}"   # data directory will be same
data_file="${data_dir}/${filename}"

# if data directory does not exist, create it
mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format
# stderr redirected in case file is not present
get_timestamp() {
    date -r $1 +%m/%d' '%R
}

# file modification time on server is preserved during wget download
[ -w $data_file ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have
wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then
    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
    exit 1
fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"
    ipset create ${temp_ipset} ${ipset_params}

    networks=$(sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' ${data_file})

    for net in $networks; do
        ipset add ${temp_ipset} ${net}
    done

    # if ipset does not exist, create it
    ipset create -exist ${firewall_ipset} ${ipset_params}

    # swap the temp ipset for the live one
    ipset swap ${temp_ipset} ${firewall_ipset}
    ipset destroy ${temp_ipset}

    # log the file modification time for use in minimizing lag in cron schedule
    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."

fi


part of crontab:
Code:


09 *  * * *  /usr/local/sbin/block
24 *  * * *  /usr/local/sbin/block
39 *  * * *  /usr/local/sbin/block
54 *  * * *  /usr/local/sbin/block
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Sat Nov 03, 2012 7:10 pm    Post subject: Re: Thanks for this! Reply with quote

Rain91 wrote:
Could you post the updated script?

It wasn't the script that was messed up; it was his kernel configuration.

Also, maybe he's using a different version of cron, or a different method of employing it, but I don't believe most people should need to change the scripts in order to use them with cron. As I see it, those variables belong in the crontab, not in the scripts.
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
upengan78
l33t
l33t


Joined: 27 Jun 2007
Posts: 676
Location: IL

PostPosted: Wed Dec 12, 2012 3:49 pm    Post subject: Reply with quote

hello,

Can I use this http://www.wizcrafts.net/exploited-servers-iptables-blocklist.html in addition to http://feeds.dshield.org/block.txt ? Thanks for the steps by the way, I got it working on my box but I just felt the list on block.txt isn't very comprehensive.

ipset list
Code:
Name: block
Type: hash:ip
Header: family inet hashsize 64 maxelem 65536 netmask 24
Size in memory: 1368
References: 0
Members:
82.221.99.0
150.164.168.0
61.147.112.0
125.64.12.0
70.88.227.0
186.202.164.0
91.143.199.0
120.86.151.0
61.236.64.0
222.211.95.0
61.150.76.0
222.173.120.0
199.192.241.0
223.78.153.0
216.144.247.0
182.213.176.0
122.154.101.0
74.63.224.0
114.84.107.0
199.241.186.0
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Thu Dec 13, 2012 2:26 am    Post subject: Reply with quote

Sure. That looks fine to me. That's what I intended, that these examples would help people create and maintain their own, and in general make use of ipsets.
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
upengan78
l33t
l33t


Joined: 27 Jun 2007
Posts: 676
Location: IL

PostPosted: Thu May 08, 2014 3:08 pm    Post subject: Reply with quote

Sorry to post into this old thread. I noticed this morning target="http://feeds.dshield.org/block.txt" is not working for me since this morning. It's redirected to something. I am using target="http://www.dshield.org/block.txt" . Hope this helps some people.
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1567
Location: U.S.A.

PostPosted: Fri May 09, 2014 12:40 am    Post subject: Reply with quote

Thank you. I suggest changing it to:
"https://www.dshield.org/block.txt"

(Note https.)
Back to top
View user's profile Send private message
OverrideZ
n00b
n00b


Joined: 27 May 2014
Posts: 1

PostPosted: Tue May 27, 2014 4:07 pm    Post subject: Reply with quote

hi,

i have made a python version, except that do not check timestamp, just update new entry and delete old one. i am not a expert dev. thanks for block list source :D

Code:

#!/bin/env python3.3
"""
---------------------------------------------------------------
 OverrideZ
 Get remote bock list and add to ipset
 ipset block list updater v1.0
---------------------------------------------------------------

 Before use this script, check you created ipset and iptables entry like above

 ipset create banned_ipv4_net hash:net family inet
 ipset create banned_ipv6_net hash:net family inet6

 iptables -I INPUT 1 -i eth0 -m set --match-set banned_ipv4_net src -j DROP
 ip6tables -I INPUT 1 -i eth0 -m set --match-set banned_ipv6_net src -j DROP

---------------------------------------------------------------
"""

IPSET_PATH = "/usr/sbin/ipset"
IPV4_URL = ["http://dshield.org/block.txt", "http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"]
IPV6_URL = ["http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt"]

#---------------------------------------------------------------
import re, subprocess
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError

#---------------------------------------------------------------
class Ipset:
    """
        Manage ipset entry Read/Add/Delete
    """
    #---------------------------------------------------------------
    def __init__(self, inet):
        self.setname = "banned_%s_net" % inet       # ipset chain name
        self.inet = inet                            # inet mode
        self.ripset = re.compile(r"^\d")            # ipset regexp
        self.currentstor = set()                    # ipset stor

    #---------------------------------------------------------------
    def process(self, netlist):
        """
            Process the blocklist data downloaded
        """
        self.read()

        deleted = self.currentstor.difference(netlist)
        added = netlist.difference(self.currentstor)
        same = netlist.intersection(self.currentstor)

        for ip in deleted:
            self.del_ip(ip)

        for ip in added:
            self.add_ip(ip)

        print("%s net | Add : %s | Dup : %s | Del : %s" % (self.inet, len(added), len(same), len(deleted)))

    #---------------------------------------------------------------
    def read(self):
        """
            read and parse current ipset list content
        """
        cmd = [IPSET_PATH, "list", self.setname]
        result = subprocess.check_output(cmd)
        data = result.decode("utf-8")

        for item in data.split("\n"):
            if self.ripset.match(item):
                self.currentstor.add(item.strip())

    #---------------------------------------------------------------
    def add_ip(self, ip):
        """
            add ip to ipset
        """
        cmd = [IPSET_PATH, "add", "-q", "-!", self.setname, str(ip)]
        subprocess.call(cmd)

    #---------------------------------------------------------------
    def del_ip(self, ip):
        """
            del ip to ipset
        """
        cmd = [IPSET_PATH, "del", "-q", "-!", self.setname, str(ip)]
        subprocess.call(cmd)

#---------------------------------------------------------------
class Updater:
    """
        Download and Parse files
    """
    #---------------------------------------------------------------
    def __init__(self, url, mode):
        self.urls = url                                                     # download url
        self.oip = Ipset(mode)                                              # ipset object
        self.rcymru = re.compile(r"^#")                                     # cymru bogon regexp
        self.rdshield = re.compile(r"(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$")    # dshield regexp
        self.currentstor = set()                                            # downloaded ip stor

    #---------------------------------------------------------------
    def download(self, url):
        """
            Download Files and launch parser
        """
        try:
            req = Request(url)
            data = urlopen(req)
            code = data.getcode()

            if code == 200:
                urlsplit = re.split("/", url)
                filename = urlsplit[len(urlsplit)-1]

                if filename == "block.txt":
                    self.parse_dshield_txt(data.read())
                elif filename == "fullbogons-ipv4.txt":
                    self.parse_cymru_txt(data.read())
                elif filename == "fullbogons-ipv6.txt":
                    self.parse_cymru_txt(data.read())

        except HTTPError as error:
            print("HTTP Error: %s %s" % (error.code, url))
        except URLError as error:
            print("URL Error: %s %s" % (error.reason, url))

    #---------------------------------------------------------------
    def parse_dshield_txt(self, data):
        """
            Parse Dshield block list
        """
        dec = data.decode("utf-8")

        for line in dec.split("\n"):
            if self.rdshield.match(line):
                detail = line.strip().split("\t")
                self.currentstor.add("%s/%s" % (detail[0], detail[2]))     

    #---------------------------------------------------------------
    def parse_cymru_txt(self, data):
        """
            Parse cymru full bogon
        """
        dec = data.decode("utf-8")

        for line in dec.split("\n"):
            if not self.rcymru.match(line):
                self.currentstor.add(line.strip())

    #---------------------------------------------------------------       
    def run(self):
        """
            main run func
        """
        for url in self.urls:
            self.download(url)

        if len(self.currentstor) != 0:
            self.oip.process(self.currentstor)
        else:
            print("Download fail")

#---------------------------------------------------------------
if __name__ == "__main__":
    Updater(IPV4_URL, "ipv4").run()
    Updater(IPV6_URL, "ipv6").run()
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum