Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Tracking internet activity
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2546

PostPosted: Tue May 02, 2017 2:58 am    Post subject: Tracking internet activity Reply with quote

I want to monitor all sites my box visits. To be clear I'm not trying to monitor another (legitimate) user. My reason is to track (and block) ad/tracking sites or, in the worst case, detect malware calling home.

My ideal format would be a list of human readable urls which I can filter out trusted sites like forums.gentoo.org and eliminate duplicates. The list will be e-mailed daily in a cron job. The last part is fairly easy so once I have the log it shouldn't be a problem. I can't seem to find a good unix way to generate said log. Any tips and tricks? And did I miss something obvious? Google has not been helpful.

Thanks.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
saturnalia0
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2016
Posts: 94

PostPosted: Tue May 02, 2017 5:31 am    Post subject: Re: Tracking internet activity Reply with quote

The Doctor wrote:
My reason is to track (and block) ad/tracking sites


Besides the usual extensions such as uBlock Origin, uMatrix/NoScript, etc you can use lists such as this: https://github.com/StevenBlack/hosts

I have found that particularly useful (using /etc/hosts for blocking ads) on my old CyanogenMod phone since ads were blocked on apps as well.

As to your actual question, would it suffice to log DNS queries? If so, `tcpdump dst port 53` plus some grepping should suffice.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Tue May 02, 2017 6:08 am    Post subject: Reply with quote

The Doctor ...

one possible solution is to use a caching proxy like net-proxy/polipo, you would then set logLevel=4 (for LOGGING_MAX). The advantage to this might be that as it's cached you would be able to do some diagnostics should you need to (though you would probably want to rotate the logs on a regular basis).

@saturnalia0 ... I don't recommend /etc/hosts for ad blocking and the like, something like this is better managed by a dns proxy such as net-dns/unbound (using an 'include' file generated via unbound-block-hosts).

best ... khay
Back to top
View user's profile Send private message
cboldt
l33t
l33t


Joined: 24 Aug 2005
Posts: 829

PostPosted: Tue May 02, 2017 11:51 am    Post subject: Reply with quote

I use dnsmasq to return 127.0.0.1 for ad servers, and get (monthly update) a list of adserver names from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq

Edit to add, dnsmasq can also directly log inquiries made to it. The logging function is quite flexible.


Last edited by cboldt on Tue May 02, 2017 11:59 am; edited 1 time in total
Back to top
View user's profile Send private message
pietinger
n00b
n00b


Joined: 17 Oct 2006
Posts: 46

PostPosted: Tue May 02, 2017 11:55 am    Post subject: Re: Tracking internet activity Reply with quote

The Doctor wrote:
[...] My reason is to track (and block) ad/tracking sites or, in the worst case, detect malware calling home.


Iam using "privoxy" since years and it filters out all ads. You can set the log-level just as you want.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1469
Location: KUUSANKOSKI, Finland

PostPosted: Tue May 02, 2017 1:51 pm    Post subject: Reply with quote

I don't monitor the traffic (but I should).

Instead I use /etc/hosts mainly and update it from here. While it's not intended for that use case, it's perhaps the most simple way.

pietinger wrote:
Iam using "privoxy" since years and it filters out all ads. You can set the log-level just as you want.
Can it filter https served ads too?
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1717

PostPosted: Tue May 02, 2017 5:24 pm    Post subject: Reply with quote

Iptables offers LOG target. You can use it to catch new connections and define information you want to keep in log (e.g. IP and port number).
Not a very clever solution, but it certainly does let you collect some stats.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2546

PostPosted: Wed May 03, 2017 6:20 am    Post subject: Reply with quote

Thanks for the replies.

I neglected to mention that I've already got ublock and some /etc/hosts blocking going already. My problem is simply figuring out what I might want to block in my case.

net-proxy/polipo looks close to what I want, but it is dead upstream. This led me to squid and I spent most of the day playing with it. Maybe I'll make some progress with it tomorrow.

Actually the problem seems much harder than I thought it would be. Redirecting system wide traffic to the proxy is straight forward, but it can't handle https traffic and leaves a lot of activity unmonitored and, with my uber impressive computer skills, more than half the internet inaccessible.

I considered the iptables solution but I'm hesitant because it might get spammy and it only gets ips. Perhaps there is an idea for a project here. Write a utility that takes an iptables log and converts it into something readily usable. I could use host to convert the ips back then a little shell magic to make the log. I think I'll play with that tomorrow as well.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
pietinger
n00b
n00b


Joined: 17 Oct 2006
Posts: 46

PostPosted: Wed May 03, 2017 11:48 am    Post subject: Reply with quote

Zucca wrote:
Can it filter https served ads too?


yes.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1469
Location: KUUSANKOSKI, Finland

PostPosted: Wed May 03, 2017 2:45 pm    Post subject: Reply with quote

The Doctor wrote:
net-proxy/polipo looks close to what I want, but it is dead upstream. This led me to squid and I spent most of the day playing with it. Maybe I'll make some progress with it tomorrow.

Actually the problem seems much harder than I thought it would be. Redirecting system wide traffic to the proxy is straight forward, but it can't handle https traffic and leaves a lot of activity unmonitored and, with my uber impressive computer skills, more than half the internet inaccessible.
I had squid as a transparent proxy. One of its jobs was to make "pre-emptive" ad-blocking where /etc/hosts was not suitable. I had a large list regexp url patterns handling the task. To make it work I had to learn a bit of iptables and port forwardings.
I never tried, but IIRC Squid has https support. I'm not sure if it's offical, but I read that from somewhere.

As I just learned that privoxy also has https support (thanks pietinger), I'm keen to try it out. If it fails I'll "fall back" to Squid eventually.
I think it's about ten years back when I had a privoxy install. Back then it seemed complex... I'll give it a go... after I have sorted out few other things.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum