View previous topic :: View next topic |
Author |
Message |
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Fri Oct 10, 2014 11:13 am Post subject: [solved] Blocking ad-sites with pdnsd doesn't work overall |
|
|
Hi there,
I'm using pdnsd to block the ads of a lot of sites. I'm using this list.
Works quite good. But there are site, which are still tainted by annoying ads, e.g. webfail.com. The ad-site criteo.com is linked in the ads. Criteo is also blacklisted in my pdnsd.conf.
ping criteo.com: | ping: unknown host criteo.com |
Nevertheless:
- the ad is shown.
- the linked content leads me to the destination site. (In my case teufel.com)
How can I get rid of this?
Last edited by musv on Tue Oct 14, 2014 9:20 am; edited 1 time in total |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Fri Oct 10, 2014 11:53 am Post subject: |
|
|
Is there any other reason why you are using pdns?
If all you want is to block a bunch of servers the easy and clean way for me is to use /etc/hosts, there are lots around, or you can create your own one, it's straightforward. Just google for "hosts blacklist" and pick a reputable site, in any case, examine it before dumping it into your /etc/hosts
Note that either method will only block DNS resolution. If a given site has hardcoded IPs into it, the only thing that can help you is an iptables rule. |
|
Back to top |
|
|
Roman_Gruber Advocate
Joined: 03 Oct 2006 Posts: 3846 Location: Austro Bavaria
|
Posted: Fri Oct 10, 2014 12:21 pm Post subject: |
|
|
i92guboj wrote: | Is there any other reason why you are using pdns?
If all you want is to block a bunch of servers the easy and clean way for me is to use /etc/hosts, there are lots around, or you can create your own one, it's straightforward. Just google for "hosts blacklist" and pick a reputable site, in any case, examine it before dumping it into your /etc/hosts
Note that either method will only block DNS resolution. If a given site has hardcoded IPs into it, the only thing that can help you is an iptables rule. |
I did the same but i never updated this list. I even manually add some sites when they get too annoying like facebook for example.
We definitely need an ebuild with an up to date host list easy to pull in on any gentoo installation than. I wonder if there is such list already?
And browser plugins help too |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Fri Oct 10, 2014 12:33 pm Post subject: |
|
|
An ebuild for that would be nice, though we would also need a dedicated tool to do the merge. I don't know if just etc-update would be fine for complex hosts files. I just use a cron job to pick the file and merge it with my /etc/hosts.mine file, that's trivial to do and works nicely
Browser plugins... well, they add a unnecessary level of complexity and the problem with them is that they suck lots of ram. If you use /etc/hosts you get all this for free without any extra resource consumption.
edit: typos |
|
Back to top |
|
|
Roman_Gruber Advocate
Joined: 03 Oct 2006 Posts: 3846 Location: Austro Bavaria
|
Posted: Fri Oct 10, 2014 12:56 pm Post subject: |
|
|
i92guboj wrote: | An ebuild for that would be nice, though we would also need a dedicated tool to do the merge. I don't know if just etc-update would be fine for complex hosts files. I just use a cron job to pick the file and merge it with my /etc/hosts.mine file, that's trivial to do and works nicely
Browser plugins... well, they add a unnecessary level of complexity and the problem with them is that they suck lots of ram. If you use /etc/hosts you get all this for free without any extra resource consumption.
edit: typos |
Is there a way to just make an easy ebuild which just downloads the source text files and add them together to a large list?
I think it does not matter if there are any duplilcate entries
And there should be an option to have a custom list which is also added.
I agree these browser plugins suck badly and the browser is not hte right spot for this at all. |
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Fri Oct 10, 2014 2:19 pm Post subject: |
|
|
i92guboj wrote: | Is there any other reason why you are using pdns? |
Yes, there are several reasons.
First of all, Pdnsd was proposed quite long time ago in the German subforum. I've tried it, and it worked. It was also a good solution, because I've found a prefilled list of Ad-servers.
And last but not least, I've installed Pdnsd on my Nas as DNS-cache. So I don't need to update every machine in my network and have a DNS cache at the same time.
But back to the topic:
Any idea, how criteo.com bypasses the blacklisting? |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Fri Oct 10, 2014 2:27 pm Post subject: |
|
|
Well, being a dns proxy, and given that all the entries in that file are domain names and not IPs, I'll insist that maybe the are using hardcoded IPs to bypass this kind of add blocker.
My suggestion: find the ip for criteo.com (ping should do, if not use whois or traceroute or whatever you prefer) then block incoming and outgoing connections to that ip. Failing that you'll have to chech the site source code and see what is going on. |
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Mon Oct 13, 2014 11:04 am Post subject: |
|
|
Tried to block the IP:
Code: | iptables -A INPUT -s 178.250.0.87 -j DROP |
The rule seems to be working:
Code: | ping 178.250.8.87
PING 178.250.8.87 (178.250.8.87) 56(84) bytes of data.
From 77.75.248.97 icmp_seq=1 Time to live exceeded |
But I still see the ad on the webpage. And if I click on that thing, I still get to the destination side (www.teufel.com).
Not nice. |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Mon Oct 13, 2014 3:25 pm Post subject: |
|
|
According to whois teufel.com IP is 67.23.28.153 |
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Tue Oct 14, 2014 3:17 am Post subject: |
|
|
i92guboj wrote: | According to whois teufel.com IP is 67.23.28.153 |
That's right. But teufel.com or teufel.de is only the destination, to where it will be referred to.
The ad is included by an iframe on the webpage. The link there:
Code: | <a id="logo" class="last" href="http://cas.fr.eu.criteo.com/delivery/ck.php?ckmode=9&cb=f94d6094f8&did=f94d6094f8&nlsid=60122&c1=onbg-DAny2qV4gcQWIEEgg56HREz8nLPwBtqU9ScJF0NewYjX9kKNo4NEYmJX6KvHdu1N6hilxQH4JxKZN_b54jLATYhHBBSaQakBpT5opFR8J6GhZ72yzxWptNL_3Mh3XLan47tspMcH05sj6Y4 WltmkVgED8IE16NBS1NFbth6gTvyzBJwxiWY3Xq4l3rD0kACW5lvTWiieyIoA77NmgvfBrSqNtP4UGeC_IHTSFMzLtCqr36pDv9UpytsJFFu729gksgRtgfNImaeYTObsaRRDbVntP91HN6SSH
PzP4rGDBYpscQCiWEj4UsEPRDU&maxdest=http%3a%2f%2fwww.teufel.de%2fzero-heroes.html%3fpartner_id%3drem.criteo.all.logo.1441082" target="_blank" "=""></a> |
That means, it links to cas.fr.eu.criteo.com, which is definitely blocked by my pdnsd and iptables. The destination url teufel.de is only specified as a parameter. But nevertheless the link works.
Last edited by musv on Wed Oct 15, 2014 9:10 am; edited 1 time in total |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Tue Oct 14, 2014 6:27 am Post subject: |
|
|
Well, something is going on, since the ip I see from here for that domain is still different from the one you used in the iptables rule, I see 178.250.0.101.
You could try blocking the whole 178.250.0.0 IP range and see if that effectively cuts down the spam.
By the way, I took a quick tour in that web site and I can't see any add, can you share a concrete URL or instructions on how to navigate to find it? I have a hosts-based block list and several lines for criteo on it, but I examined the source for the web site and I don't see "criteo" anywhere on it, and a hosts-based block can't definitely mangle the source code for any web site.
This are the sites I am locking, in case you are interested:
Code: |
0.0.0.0 sslwidget.criteo.com
0.0.0.0 cas.criteo.com
0.0.0.0 dis.criteo.com
0.0.0.0 dis.eu.criteo.com
0.0.0.0 dis.ny.us.criteo.com
0.0.0.0 dis.sv.us.criteo.com
0.0.0.0 dis.us.criteo.com
0.0.0.0 ld2.criteo.com
0.0.0.0 rta.criteo.com
0.0.0.0 rtax.criteo.com
0.0.0.0 sapatoru.widget.criteo.com
0.0.0.0 static.criteo.net
0.0.0.0 static.eu.criteo.net
0.0.0.0 widget.criteo.com
0.0.0.0 www.criteo.com
|
|
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Tue Oct 14, 2014 8:59 am Post subject: |
|
|
Muchas gracias. It works.
After putting your your list into my pdnsd, the ads are gone. I guess the problem were the different domains. I tought wrongly, that e.g. sslwidget.criteo.com would be blocked, if the domain criteo.com is blocked. That's not the case.
Btw. your solution, to put the ad-domains into a host file seems a better solution to me than the mine one. Did you manage it to include a separate host file with the blocked hosts into the main host file? |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Tue Oct 14, 2014 9:14 am Post subject: |
|
|
The hosts file has a man page, but it's truly as simple as it can get. I see to way to "include" additional files into it. Howerver that shouldn't really be a problem. This task is better suited for a cron job, so I have this:
Code: | if `wget http://winhelp2002.mvps.org/hosts.txt -O /tmp/hosts > /dev/null 2>&1`
then
rm -f /etc/hosts
mv /tmp/hosts /etc/hosts
echo "# Custom list" >> /etc/hosts
cat /home/i92guboj/hosts.mine >> /etc/hosts
fi |
The script basically attempts to retrieve the list (you can use whatever list you want as long as it has an "x.x.x.x domain.name" layout). If it can be downloaded then /etc/hosts is removed, the new file is copied, and the custom lists are added to the end of the file (you can put softonic and whatever you want in there ).
That list I use contains entries for localhost, so you don't have to add those. If your list hasn't those then you can add them either in your hosts.mine file or copy the default file in /etc/hosts.default and add that file to the list with another cat >> sentence. Whatever suits you. This way I don't have to worry about that at all, and it works surprisingly well without hogging resources. |
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Tue Oct 14, 2014 9:19 am Post subject: |
|
|
Thanks. It's a good idea simply to create a hosts file by a script. And I like the idea of including an external list automatically.
Update:
Found this:
https://bbs.archlinux.org/viewtopic.php?pid=1311801#p1311801
I can simply include your downloaded host file into pdnsd. No need to create a combined hosts file.
Last edited by musv on Tue Oct 14, 2014 10:13 am; edited 1 time in total |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Tue Oct 14, 2014 9:23 am Post subject: |
|
|
Just adjust it to your liking, then test it by running it manually.
Once you are sure it works ok you can just put it into any of the cron directories. I put it at /etc/cron.daily/hosts, you can choose a different frequency if you wish. Make sure it's executable in any case. It's easy to add a notification via e-mail or whatever if you want that stuff. |
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Tue Oct 14, 2014 11:29 am Post subject: |
|
|
Another question of fine-tuning:
Blocking the ad-sites works quite well. But if the webpages have a fixed size container for the ads, I can see there an error message:
Code: | Fehler: Verbindung fehlgeschlagen
Firefox kann keine Verbindung zu dem Server unter googleads.g.doubleclick.net aufbauen.
Die Website könnte vorübergehend nicht erreichbar sein, versuchen Sie es bitte später nochmals.
Wenn Sie auch keine andere Website aufrufen können, überprüfen Sie bitte die Netzwerk-/Internetverbindung.
Wenn Ihr Computer oder Netzwerk von einer Firewall oder einem Proxy geschützt wird, stellen Sie bitte sicher, dass Firefox auf |
Means:
Code: | Error connection failed
Firefox cannot establish a connection to the server at googleads.g.doubleclick. … |
Did you find a solution to replace that container by an empty website? It means just deliver an empty html file instead of rejecting the certain domain. |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Tue Oct 14, 2014 11:43 am Post subject: |
|
|
I really never looked into it. Maybe the sites I use don't feature such big iframes or maybe my brain just skips them without even noticing.
I guess their css could be set to "display: none;" using greasemonkey or stylish, as long as you find a selector which will fit them all (probably by contents).
That would probably defeat the purpose of not using a proper blocking extension such as AdBlock though, since greasemonkey will take its good amount of ram as well... |
|
Back to top |
|
|
musv Advocate
Joined: 01 Dec 2002 Posts: 3333 Location: de
|
Posted: Wed Oct 15, 2014 9:09 am Post subject: |
|
|
Ok, one step more. It's a first approach.
How I wrote in the posting above this. It's annoying, that sometimes the ads are loaded into iframes with a fixed size. Due to the ad blocking, the browser can't load the content and shows an error message, that the page can't be loaded.
The goal is to replace that error by a blank content.
The idea:
Deliver a blank page on port 80. If there's already a webserver running, then return a blank page as error 404 page.
The own webserver:
If not, write a mini webserver, which delivers just a blank page. This I got from rosettacode.org. I replaced the issued text and removed the "got connection" line. I moved that thing to /usr/local/bin.
'Cause I'm using Systemd, here's the Unit:
/etc/systemd/system/blankserver.service: | [Unit]
Description = replaces empty ad iframe error message.
Conflicts=nginx.service apache.service lighttpd.service
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/blankserver
[Install]
WantedBy=multi-user.target |
Enable and start the service:
Code: | systemctl enable blankserver.service
systemctl start blankserver |
Change the hosts file:
I took the downloaded hosts file, removed the comments and the localhost and replaced:
0.0.0.0 by 127.0.0.1
so it looks like this:
/etc/hosts-pdnsd: | 127.0.0.1 0.datacollector.coin.scribol.com
127.0.0.1 0.r.msn.com
127.0.0.1 005.free-counter.co.uk
127.0.0.1 006.free-counter.co.uk
127.0.0.1 007.free-counter.co.uk
127.0.0.1 008.free-counter.co.uk
127.0.0.1 008.free-counters.co.uk
127.0.0.1 00fun.com
127.0.0.1 011707160008.c.mystat-in.net
… |
Loading the file in pdnsd:
/etc/pdnsd.conf (Source Section): | source {
ttl=1w; # Caching one week
owner=localhost;
serve_aliases=on;
file="/etc/hosts"; # read local hosts file
file="/etc/hosts-pdnsd"; # block
} |
Result:
On the certain pages with fixed-size iframes there's now a blank area, where the error message was shown before. It looks a little bit confusing, having some space between the contents. But nevertheless it's a step forward.
Comments, critics, improvements are welcome! |
|
Back to top |
|
|
|