| View previous topic :: View next topic |
| Author |
Message |
shimitar
Guru
Joined: 23 Nov 2003
Posts: 320
Location: Italy, Torino
|
 Posted: Sun Feb 11, 2024 3:06 pm Post subject: Ad blocking at DNS level |
 |
|
Anybody has installe AdGuard DNS software on Gentoo?
Is there a better choice?
I could set AdGuard DNS directly, but it would be nice to host it directly.
_________________
Willy Gardiol
willy@gardiol.org |
|
| Back to top |
|
 |
user
Apprentice
Joined: 08 Feb 2004
Posts: 202
|
| Posted: Sun Feb 11, 2024 7:19 pm Post subject: |
|
|
Hi shimitar,
choose open source DNS resolver of your choice like bind, dnsmasq, unbound and import dns blocklist from https://oisd.nl/downloads |
|
| Back to top |
|
|
Banana
Veteran
Joined: 21 May 2004
Posts: 1392
Location: Germany
|
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA
|
| Posted: Tue Feb 13, 2024 5:28 am Post subject: |
|
|
I hand-jam a custom /etc/hosts in order to block nuisance and malicious sites using a pair of script that I crafted following the concept of an adblocker that's included in antiX and MX-Linux. It's easy, doesn't require much maintenance (I update the list maybe very couple of months). It's low tech, and only one arrow in my quiver. The first script retrieves lists of sites to block, and the second script concatenates the lists, sorts, deduplicates, and adjusts the contents using ordinary Unix/Linux text editing tools.
My current blocklist that I append to /etc/hosts runs 347,727 rows. It speeds up browsing and keeps a lot of crap out of my web browsers.
I've posted details along with the scripts on the MX-Linux forums under Tips & Tricks at the following URL:
https://forum.mxlinux.org/viewtopic.php?p=558095#p558095
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
| Back to top |
|
|
shimitar
Guru
Joined: 23 Nov 2003
Posts: 320
Location: Italy, Torino
|
| Posted: Tue Feb 13, 2024 8:13 am Post subject: |
|
|
| Banana wrote: | | Or if you want to maintain it and use it for all your devices at home, use https://pi-hole.net/ and set it up as your DNS server |
Is there a practical way to have pihole on Gentoo? I already have one server at home and i don't want to add a second one
_________________
Willy Gardiol
willy@gardiol.org |
|
| Back to top |
|
|
Banana
Veteran
Joined: 21 May 2004
Posts: 1392
Location: Germany
|
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Tue Feb 13, 2024 10:15 am Post subject: |
|
|
This one combines several large hosts files for ad/malware/whatnot blocking.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA
|
| Posted: Tue Feb 13, 2024 4:38 pm Post subject: |
|
|
| Zucca wrote: | | This one combines several large hosts files for ad/malware/whatnot blocking. |
Do you use referenced resources from Ultimate-Hosts-Blacklist? If yes, please share your personal experience with them.
Thanks for the link.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Tue Feb 13, 2024 5:02 pm Post subject: |
|
|
| figueroa wrote: | | Do you use referenced resources from Ultimate-Hosts-Blacklist? If yes, please share your personal experience with them. |
I have used someonewhocares.org hosts list before and my plan is to try out this Ultimate-Hosts-Blacklist soon enough... So at the moment I have no experience with it.
EDIT: | Code: | $ curl -s --head https://hosts.ubuntu101.co.za/hosts | grep ^content-length:
content-length: 19504858 |
... oh boy.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Tue Feb 13, 2024 5:52 pm Post subject: |
|
|
I added the little over 600k hosts now on my laptop.
I'll report if something goes south.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
mrbassie
l33t
Joined: 31 May 2013
Posts: 772
Location: over here
|
| Posted: Tue Feb 13, 2024 6:45 pm Post subject: |
|
|
| shimitar wrote: | | Banana wrote: | | Or if you want to maintain it and use it for all your devices at home, use https://pi-hole.net/ and set it up as your DNS server |
Is there a practical way to have pihole on Gentoo? I already have one server at home and i don't want to add a second one |
https://github.com/Tatsh/tatsh-overlay/tree/master/net-dns
Not mine btw, I looked it up on zugaina. There's a docker image on github too. I can't opine on either, I have basically the same setup as @Banana.
_________________
Bus conductors learned to code. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Tue Feb 13, 2024 9:38 pm Post subject: |
|
|
| Zucca wrote: | | This one combines several large hosts files for ad/malware/whatnot blocking. |
I made a preliminary ebuild for it.
I haven't crafted any live ebuilds that don't use git, so I'll bet this one breaks quite some rules.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA
|
| Posted: Wed Feb 14, 2024 5:10 am Post subject: |
|
|
Zucca, I think this would do for the end of the ebuild. It works for me without need to match whitespace with "\s" as part of the expression. Lifted directly from my own script so I know it works. The caret (^) anchors the first expression to the beginning of the line.
| Code: | | sed 's/^127\.0\.0\.1/0\.0\.0\.0/g' |
Reverse it, of course, if you want 127.0.0.1 in place of 0.0.0.0 as you did in the ebuild. I do use 0.0.0.0 for blocked URLs.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Wed Feb 14, 2024 8:55 am Post subject: |
|
|
| figueroa wrote: | It works for me without need to match whitespace with "\s" as part of the expression. Lifted directly from my own script so I know it works. The caret (^) anchors the first expression to the beginning of the line.
| Code: | | sed 's/^127\.0\.0\.1/0\.0\.0\.0/g' |
Reverse it, of course, if you want 127.0.0.1 in place of 0.0.0.0 as you did in the ebuild. I do use 0.0.0.0 for blocked URLs. |
I was just extra careful by using "\s". But you shouldn't need to have sed perform "globally" (g at the end) as fas as I know.
The hosts list is, by default, in 0.0.0.0 form, but if USE="to-127" is enabled then sed acts. Yeah, the flag could have better name...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Wed Feb 14, 2024 9:32 am Post subject: |
|
|
Using the ultimate blocklist hasn't caused any problems so far, except that it seems some cloudflare hosted sites seem to present me the | Code: | | [ ] I'm not a robot |
... confirmation. I also have Cookie AutoDelete installed, so that in conjunction with the blocklist may cause the confirmation to appear.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA
|
| Posted: Wed Feb 14, 2024 6:09 pm Post subject: |
|
|
Zucca,
Regarding the g (globally) at the end of the sed invocation, when I leave g out, the output file is somewhat larger, apparently due to failure to delete redundant whitespace in the row where a row contains comments. Example using diff:
In other words, the outputs are functionally identical.
Here is my entire script so you can see it in context. The objective of this script is to concatenate, normalize, and sort each line in a group of text files into a new output file.
| Code: | #!/bin/sh
# Compansion script to hosts-wget.scr to automatically concatenate files
# retrieved by that script, normalizing them, and finally doing a sort unique
# into an output file named adlist-all, an ordinary text file.
# Runs as ordinary user and requires directories /home/username/hosts and
# /home/username/hosts/hosts.bak/ to pre-exist as working directories.
# Adjust script to account for personal layout differences or changes.
# The output file may be APPENDED to your /etc/hosts file. Don't overwrite it!
# Comments follow:
# 0. Starts with copy (cp) to backup previous adlist-all file, then
# concatenate (cat) all files ending with .txt and pipe through ...
# 1. print only lines beginning with number 0 or 1,
# sed -n '/^[01]/p' | \
# This step was moved to top of the list since it also removes
# both comments that begin with a hashtag and blank lines.
# Originally, it was the final instruction, sed -n '/^[0-9]/p' | \
# 2. deleting lines that contain the string localhost,
# sed '/localhost/d' | \
# 3. (removed) suppress comments, # sed '/^#/d' | \
# 4. (removed) suppress empty lines, # sed '/^$/d' | \
# 5. replaces tabs with spaces, sed 's/[\t]/ /g' | \
# 6. replaces double spaces with single spaces, sed 's/ / /g' | \
# 7. In lines beginning with 127.0.0.1 substitute 0.0.0.0,
# sed 's/^127\.0\.0\.1/0\.0\.0\.0/g' | \
# also, unblock listed block lines adding # to row
# 8. suppress \r at end of line, tr -d '\015' | \
# 9. then sort unique by field 2 (url) into adlist-all,
# sort -u -k 2 > ~/hosts/adlist-all
# begin script:
cp ~/hosts/adlist-all ~/hosts/hosts.bak/
cat ~/hosts/*.txt | \
sed -n '/^[01]/p' | \
sed '/localhost/d' | \
sed 's/[\t]/ /g' | \
sed 's/ / /g' | \
sed 's/^127\.0\.0\.1/0\.0\.0\.0/g' | \
sed 's/0.0.0.0 eepurl.com/#0.0.0.0 eepurl.com/g' | \
sed 's/0.0.0.0 mailchi.mp/#0.0.0.0 mailchi.mp/g' | \
sed 's/0.0.0.0 smetrics.ups.com/#0.0.0.0 smetrics.ups.com/g' | \
sed 's/0.0.0.0 payusatax.com/#0.0.0.0 payusatax.com/g' | \
sed 's/0.0.0.0 www.payusatax.com/#0.0.0.0 www.payusatax.com/g' | \
sed 's/0.0.0.0 t.co$/#0.0.0.0 t.co/g' | \
tr -d '\015' | \
sort -u -k 2 > ~/hosts/adlist-all |
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3627
|
| Posted: Wed Feb 14, 2024 11:32 pm Post subject: |
|
|
| Zucca wrote: | | Zucca wrote: | | This one combines several large hosts files for ad/malware/whatnot blocking. |
I made a preliminary ebuild for it.
I haven't crafted any live ebuilds that don't use git, so I'll bet this one breaks quite some rules. |
Interesting... 
Thks 4 ur attention, interest & support
_________________
USE="-* ..." in /etc/portage/make.conf here.
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Thu Feb 15, 2024 6:58 am Post subject: |
|
|
figueroa,
I didn't realize you had such script with several sed commands.
I meant that the sed commands which changes 0.0.0.0 to 127.0.0.1 (or other way around) do not benefit from g(lobal), since with sed command of | Code: | | s/^\0\.0\.0\.0/127.0.0.1/ |
... there's no need for g because each line only has one start. ;) It might even yield to faster processing, since the first search & replace action sed will stop processing the line and print it, then advancing to the next line. Although I'd suspect sed could be smart and drop the g flag automatically if it sees that the searched regex string could only be matched once at the start of a line.
Few observations:- You could get rid of (almost) all unnecessary white spaces with
| Code: | | sed -e 's/\s\{2,\}/ /g' |
This replaces every concurrent set of white spaces with one (matches whitespace two or more times).
You can pass several actions/scripts for sed with -e | Code: | | sed -e '<script>' -e '<script>' -e '<script>' |
... however this may actually be slower performing way, since by piping, sed processes can each run on separate cpu cores.
To avoid much more steering off the topic we should maybe start another topic, if needed.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
mrbassie
l33t
Joined: 31 May 2013
Posts: 772
Location: over here
|
| Posted: Sun Feb 18, 2024 6:26 pm Post subject: |
|
|
| Zucca wrote: | | This one combines several large hosts files for ad/malware/whatnot blocking. |
That appears to block ads on the youtube app on android which none of the firebog ones do. Nice one!
_________________
Bus conductors learned to code. |
|
| Back to top |
|
|
mrbassie
l33t
Joined: 31 May 2013
Posts: 772
Location: over here
|
| Posted: Mon Feb 19, 2024 6:50 pm Post subject: |
|
|
| mrbassie wrote: | | Zucca wrote: | | This one combines several large hosts files for ad/malware/whatnot blocking. |
That appears to block ads on the youtube app on android which none of the firebog ones do. Nice one! |
EDIT: No it doesn't, at least not all of them. Thanks all the same, can't have too many blocklists.
_________________
Bus conductors learned to code. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3137
|
| Posted: Mon Feb 19, 2024 8:31 pm Post subject: |
|
|
YT used to serve all adds from a different domain, I've been exploiting that for years with a custom adblock rule. Unfortunately they started mixing thing up; the adds domain appears to have been retired, and adds are now served from different paths and inserted in between clips, distinguished by their class attribute, and so on. Some time ago I saw a few different versions of their UI with adds inserted in different ways, like they were doing A/B tests for conversions.
Still, adblock lists keep doing a pretty good job filtering the garbage out, even if domain alone is not sufficient anymore.
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA
|
| Posted: Mon Feb 19, 2024 9:21 pm Post subject: |
|
|
Brave Browser does a fine job of blocking ads on Youtube.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
| Back to top |
|
|
mrbassie
l33t
Joined: 31 May 2013
Posts: 772
Location: over here
|
| Posted: Tue Feb 20, 2024 6:39 pm Post subject: |
|
|
@figueroa what I'm thinking about is why these blacklists are ignored by "apps" on "smart" things like phones or tellyiies.
@szatox are you talking about the seemingly never ending collection of domains beginning with rr?
_________________
Bus conductors learned to code. |
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA
|
| Posted: Wed Feb 21, 2024 1:02 am Post subject: |
|
|
| mrbassie wrote: | @figueroa what I'm thinking about is why these blacklists are ignored by "apps" on "smart" things like phones or tellyiies.
... |
With regard to your phone and TV, it depends on where the blacklist is in relationship to where the phone and TV are getting their network connection. If the blacklist is on the router (i.e. pihole) then there shouldn't be any "ignored" for devices connected to it.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Wed Feb 21, 2024 2:17 am Post subject: |
|
|
| I expect DNS-based blocking is countered by DNS-over-HTTPS (DoH), since that sends the query over TLS to a server "in the cloud" and gets the answer back over that channel, so your local DNS resolver is unaware that any DNS resolution happened and therefore has no opportunity to return a result different from what the domain owner intended. |
|
| Back to top |
|
|
|
Networking & Security
