Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[solved] Blocking ad-sites with pdnsd doesn't work overall
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Fri Oct 10, 2014 11:13 am    Post subject: [solved] Blocking ad-sites with pdnsd doesn't work overall Reply with quote

Hi there,

I'm using pdnsd to block the ads of a lot of sites. I'm using this list.

Works quite good. But there are site, which are still tainted by annoying ads, e.g. webfail.com. The ad-site criteo.com is linked in the ads. Criteo is also blacklisted in my pdnsd.conf.

ping criteo.com:
ping: unknown host criteo.com


Nevertheless:
  • the ad is shown.
  • the linked content leads me to the destination site. (In my case teufel.com)


How can I get rid of this?


Last edited by musv on Tue Oct 14, 2014 9:20 am; edited 1 time in total
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Fri Oct 10, 2014 11:53 am    Post subject: Reply with quote

Is there any other reason why you are using pdns?

If all you want is to block a bunch of servers the easy and clean way for me is to use /etc/hosts, there are lots around, or you can create your own one, it's straightforward. Just google for "hosts blacklist" and pick a reputable site, in any case, examine it before dumping it into your /etc/hosts

Note that either method will only block DNS resolution. If a given site has hardcoded IPs into it, the only thing that can help you is an iptables rule.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Fri Oct 10, 2014 12:21 pm    Post subject: Reply with quote

i92guboj wrote:
Is there any other reason why you are using pdns?

If all you want is to block a bunch of servers the easy and clean way for me is to use /etc/hosts, there are lots around, or you can create your own one, it's straightforward. Just google for "hosts blacklist" and pick a reputable site, in any case, examine it before dumping it into your /etc/hosts

Note that either method will only block DNS resolution. If a given site has hardcoded IPs into it, the only thing that can help you is an iptables rule.


I did the same but i never updated this list. I even manually add some sites when they get too annoying like facebook for example.

We definitely need an ebuild with an up to date host list easy to pull in on any gentoo installation than. I wonder if there is such list already?

And browser plugins help too
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Fri Oct 10, 2014 12:33 pm    Post subject: Reply with quote

An ebuild for that would be nice, though we would also need a dedicated tool to do the merge. I don't know if just etc-update would be fine for complex hosts files. I just use a cron job to pick the file and merge it with my /etc/hosts.mine file, that's trivial to do and works nicely ;)

Browser plugins... well, they add a unnecessary level of complexity and the problem with them is that they suck lots of ram. If you use /etc/hosts you get all this for free without any extra resource consumption.

edit: typos
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Fri Oct 10, 2014 12:56 pm    Post subject: Reply with quote

i92guboj wrote:
An ebuild for that would be nice, though we would also need a dedicated tool to do the merge. I don't know if just etc-update would be fine for complex hosts files. I just use a cron job to pick the file and merge it with my /etc/hosts.mine file, that's trivial to do and works nicely ;)

Browser plugins... well, they add a unnecessary level of complexity and the problem with them is that they suck lots of ram. If you use /etc/hosts you get all this for free without any extra resource consumption.

edit: typos


Is there a way to just make an easy ebuild which just downloads the source text files and add them together to a large list?

I think it does not matter if there are any duplilcate entries

And there should be an option to have a custom list which is also added.

I agree these browser plugins suck badly and the browser is not hte right spot for this at all.
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Fri Oct 10, 2014 2:19 pm    Post subject: Reply with quote

i92guboj wrote:
Is there any other reason why you are using pdns?

Yes, there are several reasons.

First of all, Pdnsd was proposed quite long time ago in the German subforum. I've tried it, and it worked. It was also a good solution, because I've found a prefilled list of Ad-servers.

And last but not least, I've installed Pdnsd on my Nas as DNS-cache. So I don't need to update every machine in my network and have a DNS cache at the same time.

But back to the topic:
Any idea, how criteo.com bypasses the blacklisting?
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Fri Oct 10, 2014 2:27 pm    Post subject: Reply with quote

Well, being a dns proxy, and given that all the entries in that file are domain names and not IPs, I'll insist that maybe the are using hardcoded IPs to bypass this kind of add blocker.

My suggestion: find the ip for criteo.com (ping should do, if not use whois or traceroute or whatever you prefer) then block incoming and outgoing connections to that ip. Failing that you'll have to chech the site source code and see what is going on.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Mon Oct 13, 2014 11:04 am    Post subject: Reply with quote

Tried to block the IP:

Code:
iptables -A INPUT -s 178.250.0.87 -j DROP


The rule seems to be working:
Code:
ping 178.250.8.87
PING 178.250.8.87 (178.250.8.87) 56(84) bytes of data.
From 77.75.248.97 icmp_seq=1 Time to live exceeded


But I still see the ad on the webpage. And if I click on that thing, I still get to the destination side (www.teufel.com).

Not nice. :evil:
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Mon Oct 13, 2014 3:25 pm    Post subject: Reply with quote

According to whois teufel.com IP is 67.23.28.153
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Tue Oct 14, 2014 3:17 am    Post subject: Reply with quote

i92guboj wrote:
According to whois teufel.com IP is 67.23.28.153


That's right. But teufel.com or teufel.de is only the destination, to where it will be referred to.

The ad is included by an iframe on the webpage. The link there:
Code:
<a id="logo" class="last" href="http://cas.fr.eu.criteo.com/delivery/ck.php?ckmode=9&amp;cb=f94d6094f8&amp;did=f94d6094f8&amp;nlsid=60122&amp;c1=onbg-DAny2qV4gcQWIEEgg56HREz8nLPwBtqU9ScJF0NewYjX9kKNo4NEYmJX6KvHdu1N6hilxQH4JxKZN_b54jLATYhHBBSaQakBpT5opFR8J6GhZ72yzxWptNL_3Mh3XLan47tspMcH05sj6Y4 WltmkVgED8IE16NBS1NFbth6gTvyzBJwxiWY3Xq4l3rD0kACW5lvTWiieyIoA77NmgvfBrSqNtP4UGeC_IHTSFMzLtCqr36pDv9UpytsJFFu729gksgRtgfNImaeYTObsaRRDbVntP91HN6SSH
PzP4rGDBYpscQCiWEj4UsEPRDU&amp;maxdest=http%3a%2f%2fwww.teufel.de%2fzero-heroes.html%3fpartner_id%3drem.criteo.all.logo.1441082" target="_blank" "=""></a>


That means, it links to cas.fr.eu.criteo.com, which is definitely blocked by my pdnsd and iptables. The destination url teufel.de is only specified as a parameter. But nevertheless the link works. :evil:


Last edited by musv on Wed Oct 15, 2014 9:10 am; edited 1 time in total
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Tue Oct 14, 2014 6:27 am    Post subject: Reply with quote

Well, something is going on, since the ip I see from here for that domain is still different from the one you used in the iptables rule, I see 178.250.0.101.

You could try blocking the whole 178.250.0.0 IP range and see if that effectively cuts down the spam.

By the way, I took a quick tour in that web site and I can't see any add, can you share a concrete URL or instructions on how to navigate to find it? I have a hosts-based block list and several lines for criteo on it, but I examined the source for the web site and I don't see "criteo" anywhere on it, and a hosts-based block can't definitely mangle the source code for any web site.

This are the sites I am locking, in case you are interested:

Code:

0.0.0.0 sslwidget.criteo.com
0.0.0.0 cas.criteo.com
0.0.0.0 dis.criteo.com
0.0.0.0 dis.eu.criteo.com
0.0.0.0 dis.ny.us.criteo.com
0.0.0.0 dis.sv.us.criteo.com
0.0.0.0 dis.us.criteo.com
0.0.0.0 ld2.criteo.com
0.0.0.0 rta.criteo.com
0.0.0.0 rtax.criteo.com
0.0.0.0 sapatoru.widget.criteo.com
0.0.0.0 static.criteo.net
0.0.0.0 static.eu.criteo.net
0.0.0.0 widget.criteo.com
0.0.0.0 www.criteo.com

_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Tue Oct 14, 2014 8:59 am    Post subject: Reply with quote

Muchas gracias. It works.

After putting your your list into my pdnsd, the ads are gone. I guess the problem were the different domains. I tought wrongly, that e.g. sslwidget.criteo.com would be blocked, if the domain criteo.com is blocked. That's not the case.

Btw. your solution, to put the ad-domains into a host file seems a better solution to me than the mine one. Did you manage it to include a separate host file with the blocked hosts into the main host file?
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Tue Oct 14, 2014 9:14 am    Post subject: Reply with quote

The hosts file has a man page, but it's truly as simple as it can get. I see to way to "include" additional files into it. Howerver that shouldn't really be a problem. This task is better suited for a cron job, so I have this:

Code:
if `wget http://winhelp2002.mvps.org/hosts.txt -O /tmp/hosts > /dev/null 2>&1`
then
        rm -f /etc/hosts
        mv /tmp/hosts /etc/hosts
        echo "# Custom list" >> /etc/hosts
        cat /home/i92guboj/hosts.mine >> /etc/hosts
fi


The script basically attempts to retrieve the list (you can use whatever list you want as long as it has an "x.x.x.x domain.name" layout). If it can be downloaded then /etc/hosts is removed, the new file is copied, and the custom lists are added to the end of the file (you can put softonic and whatever you want in there ;) ).

That list I use contains entries for localhost, so you don't have to add those. If your list hasn't those then you can add them either in your hosts.mine file or copy the default file in /etc/hosts.default and add that file to the list with another cat >> sentence. Whatever suits you. This way I don't have to worry about that at all, and it works surprisingly well without hogging resources.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Tue Oct 14, 2014 9:19 am    Post subject: Reply with quote

Thanks. It's a good idea simply to create a hosts file by a script. And I like the idea of including an external list automatically.

Update:
Found this:

https://bbs.archlinux.org/viewtopic.php?pid=1311801#p1311801

I can simply include your downloaded host file into pdnsd. No need to create a combined hosts file.


Last edited by musv on Tue Oct 14, 2014 10:13 am; edited 1 time in total
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Tue Oct 14, 2014 9:23 am    Post subject: Reply with quote

Just adjust it to your liking, then test it by running it manually.

Once you are sure it works ok you can just put it into any of the cron directories. I put it at /etc/cron.daily/hosts, you can choose a different frequency if you wish. Make sure it's executable in any case. It's easy to add a notification via e-mail or whatever if you want that stuff.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Tue Oct 14, 2014 11:29 am    Post subject: Reply with quote

Another question of fine-tuning:

Blocking the ad-sites works quite well. But if the webpages have a fixed size container for the ads, I can see there an error message:

Code:
Fehler: Verbindung fehlgeschlagen

Firefox kann keine Verbindung zu dem Server unter googleads.g.doubleclick.net aufbauen.

    Die Website könnte vorübergehend nicht erreichbar sein, versuchen Sie es bitte später nochmals.
    Wenn Sie auch keine andere Website aufrufen können, überprüfen Sie bitte die Netzwerk-/Internetverbindung.
    Wenn Ihr Computer oder Netzwerk von einer Firewall oder einem Proxy geschützt wird, stellen Sie bitte sicher, dass Firefox auf

Means:
Code:
Error connection failed

Firefox cannot establish a connection to the server at googleads.g.doubleclick. …

Did you find a solution to replace that container by an empty website? It means just deliver an empty html file instead of rejecting the certain domain.
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Tue Oct 14, 2014 11:43 am    Post subject: Reply with quote

I really never looked into it. Maybe the sites I use don't feature such big iframes or maybe my brain just skips them without even noticing.

I guess their css could be set to "display: none;" using greasemonkey or stylish, as long as you find a selector which will fit them all (probably by contents).

That would probably defeat the purpose of not using a proper blocking extension such as AdBlock though, since greasemonkey will take its good amount of ram as well...
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
musv
Advocate
Advocate


Joined: 01 Dec 2002
Posts: 3239
Location: de

PostPosted: Wed Oct 15, 2014 9:09 am    Post subject: Reply with quote

Ok, one step more. It's a first approach.

How I wrote in the posting above this. It's annoying, that sometimes the ads are loaded into iframes with a fixed size. Due to the ad blocking, the browser can't load the content and shows an error message, that the page can't be loaded.

The goal is to replace that error by a blank content.

The idea:
Deliver a blank page on port 80. If there's already a webserver running, then return a blank page as error 404 page.

The own webserver:
If not, write a mini webserver, which delivers just a blank page. This I got from rosettacode.org. I replaced the issued text and removed the "got connection" line. I moved that thing to /usr/local/bin.

'Cause I'm using Systemd, here's the Unit:

/etc/systemd/system/blankserver.service:
[Unit]
Description = replaces empty ad iframe error message.
Conflicts=nginx.service apache.service lighttpd.service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/blankserver

[Install]
WantedBy=multi-user.target


Enable and start the service:
Code:
systemctl enable blankserver.service
systemctl start blankserver


Change the hosts file:
I took the downloaded hosts file, removed the comments and the localhost and replaced:

0.0.0.0 by 127.0.0.1

so it looks like this:
/etc/hosts-pdnsd:
127.0.0.1 0.datacollector.coin.scribol.com
127.0.0.1 0.r.msn.com
127.0.0.1 005.free-counter.co.uk
127.0.0.1 006.free-counter.co.uk
127.0.0.1 007.free-counter.co.uk
127.0.0.1 008.free-counter.co.uk
127.0.0.1 008.free-counters.co.uk
127.0.0.1 00fun.com
127.0.0.1 011707160008.c.mystat-in.net


Loading the file in pdnsd:
/etc/pdnsd.conf (Source Section):
source {
    ttl=1w;                           # Caching one week
    owner=localhost;   
    serve_aliases=on;
    file="/etc/hosts";             # read local hosts file
    file="/etc/hosts-pdnsd";    # block
}


Result:
On the certain pages with fixed-size iframes there's now a blank area, where the error message was shown before. It looks a little bit confusing, having some space between the contents. But nevertheless it's a step forward.

Comments, critics, improvements are welcome!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum