| View previous topic :: View next topic |
| Author |
Message |
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
 Posted: Wed Sep 28, 2005 4:43 am Post subject: How do you manage your log files? (Best practices?) |
 |
|
Administration involves many boring duties, and log management is probably high on that list.
What do you do to make that job easier?
Do you do anything besides a basic logrotate setup?
Do you try to minimize what goes into the messages log, or do you just let a tool organize its contents for easier viewing?
Have you ever come across a huge log file (hundreds of megs/multiple gigs), and had to go through it? What have you found to be the best methods?
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
 |
groovin
Guru
Joined: 07 Feb 2004
Posts: 429
Location: California, USA
|
| Posted: Wed Sep 28, 2005 5:24 am Post subject: |
|
|
| our logs are kind of messy right now... working on some perl scripts to chop them up and make some nice html pages so i can view logs from all servers based upon server > date via apache. might even add some kind of basic search to it as well. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Wed Sep 28, 2005 5:58 am Post subject: |
|
|
Well, I'm finally getting around to taking a look at mine.
My firewall had a log ~650M or so. My DSL "modem" was broadcasting, which is apparently "normal operation." My iptables rules were logging those broadcasts. After removing those broadcasts, the log file was only a few meg (for a few years worth of operation). After removing another ~3M of iptables logs, its under 300k. (Still need to go through the last 3M of iptables logs to see what it is though).
My main box had a log of ~9.6G (yes, G). After removing "usb-storage" messages, it dropped to 454k. I knew there were some usb messages, but ignored them, since the drives appeared to be working OK. Looks like I need to figure out whats going on there.
I wonder how common it is to have "single events" filling logs like that. It's making me wonder about using fam (or something) to monitor growth.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
groovin
Guru
Joined: 07 Feb 2004
Posts: 429
Location: California, USA
|
| Posted: Wed Sep 28, 2005 6:23 am Post subject: |
|
|
| 9.6G?? holy overgrown log files batman!! |
|
| Back to top |
|
|
tsunam
Retired Dev
Joined: 23 Feb 2004
Posts: 343
|
| Posted: Wed Sep 28, 2005 7:26 am Post subject: |
|
|
you sir win the prize for largest, I actually looked at the log prize 
9.6 G is wow...I think my largest is 7 meg, and that'll be rotated in the morning. Generally I have about a week to look at logs before they get rotated, then 2 in storage before they go away.
_________________
I'm not afraid of happy endings, just afraid my life wont work that way. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Wed Sep 28, 2005 7:38 am Post subject: |
|
|
I couldn't find the thread, but someone had a RAID drive fail, which generated ~25G in a handful of seconds IIRC. But yeah, 9.6G is large.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Shadow Skill
Veteran
Joined: 04 Dec 2004
Posts: 1023
|
| Posted: Wed Sep 28, 2005 8:04 am Post subject: |
|
|
| pjp wrote: | | I couldn't find the thread, but someone had a RAID drive fail, which generated ~25G in a handful of seconds IIRC. But yeah, 9.6G is large. |
I remember the thread you are thinking of and it was truly horrific to even think about having a 25g log file.
_________________
Ware wa mutekinari.
Wa ga kage waza ni kanau mono nashi.
Wa ga ichigeki wa mutekinari.
"First there was nothing, so the lord gave us light. There was still nothing, but at least you could see it." |
|
| Back to top |
|
|
stahlsau
Guru
Joined: 09 Jan 2004
Posts: 584
Location: WildWestwoods
|
| Posted: Wed Sep 28, 2005 11:18 am Post subject: |
|
|
| Quote: | | My main box had a log of ~9.6G (yes, G). After removing "usb-storage" messages, it dropped to 454k. |
Eventually you've got "usb-debug" enabled in the kernel, i would check that. Just a shot in the blue 
I've got all my progs to write their logs to /var/log, and most of the logs enabled to get rotated once in a week. The rotated are bzipped and stored 4 weeks before deletion.
Second, with syslog-ng there are plenty of thing you can finetune in the settings, like routing some logs to other files etc... |
|
| Back to top |
|
|
shadow_dancer
n00b
Joined: 25 Feb 2005
Posts: 68
Location: germany
|
| Posted: Wed Sep 28, 2005 11:55 am Post subject: |
|
|
here is mine "bad_guys_searcher"...
based in python cookbook with some tweaks..
| Code: |
#!/usr/bin/python
# checking logs
# based: python cookbook
# original author: Mark Nevadov
# this is script is free! do what you want with it...
import sys
def CalculateApacheHits(logfile_pathname):
IpHitListing={}
Contents = open(logfile_pathname,"r").xreadlines()
for line in Contents:
Ip = line.split(" ")[0]
if 6 < len(Ip) <= 15:
IpHitListing[Ip] = IpHitListing.get(Ip,0) + 1
return IpHitListing
#
def GetIpErrorAuth(logfile_pathname):
IpDict={}
File = open(logfile_pathname,"r").xreadlines()
for line in File:
if 'allowed' in line:
Ip1=line.split("from ")[1]
Ip=Ip1.split(" not")[0]
IpDict[Ip] = IpDict.get(Ip,0)+1
if 'Invalid' in line:
Ip1=line.split("from ")[1]
Ip=Ip1.split("\n")[0]
IpDict[Ip] = IpDict.get(Ip,0)+1
return IpDict
def GetErrorApache(logfile_pathname):
IpDict={}
File = open(logfile_pathname,"r").xreadlines()
for line in File:
if 'error' in line:
Ip1=line.split("client ")[1]
Ip=Ip1.split("]")[0]
IpDict[Ip] = IpDict.get(Ip,0)+1
return IpDict
#
print "Apache Hits ..."
print "---------------------"
dict=CalculateApacheHits("/var/log/httpd/access_log")
for x in dict.keys():
print x,'\t',dict[x]
print " "
print "---------------------"
print "Apache Error ..."
print "---------------------"
dict=GetErrorApache("/var/log/httpd/error_log")
for x in dict.keys():
print x,'\t',dict[x]
print " "
print "---------------------"
print "SSH Error ..."
print "---------------------"
dict=GetIpErrorAuth("/var/log/auth.log")
for x in dict.keys():
print x,'\t',dict[x]
|
cheers,
sd |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Wed Sep 28, 2005 4:46 pm Post subject: |
|
|
| stahlsau wrote: | | Eventually you've got "usb-debug" enabled in the kernel, i would check that. Just a shot in the blue |
Hmmm.... was that a suggestion that it should be enabled to discover the problem, or that it is the problem because it is enabled (it is)?
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Monkeh
Veteran
Joined: 06 Aug 2005
Posts: 1656
Location: England
|
| Posted: Wed Sep 28, 2005 5:55 pm Post subject: |
|
|
| pjp wrote: | | I couldn't find the thread, but someone had a RAID drive fail, which generated ~25G in a handful of seconds IIRC. |
He unplugged it by accident. https://forums.gentoo.org/viewtopic-t-381804-highlight-raid.html
I myself would like some good tips on handling log files, I'm not sure where to start. |
|
| Back to top |
|
|
drseergio
Apprentice
Joined: 28 Sep 2005
Posts: 236
|
| Posted: Wed Sep 28, 2005 7:00 pm Post subject: How to conviently manage logs? |
|
|
| I have a small server with basic features such as mail/web/sftp... and I have lots of large logs, how to organize correct management of them and frequent auto cleaning? As a log server I have syslog-ng. |
|
| Back to top |
|
|
Dlareh
Advocate
Joined: 06 Aug 2005
Posts: 2102
|
| Posted: Wed Sep 28, 2005 7:05 pm Post subject: |
|
|
logrotate, perhaps...
_________________
"Mr Thomas Edison has been up on the two previous nights discovering 'a bug' in his phonograph." --Pall Mall Gazette (1889)
Are we THERE yet? |
|
| Back to top |
|
|
adaptr
Watchman
Joined: 06 Oct 2002
Posts: 6730
Location: Rotterdam, Netherlands
|
| Posted: Wed Sep 28, 2005 7:06 pm Post subject: |
|
|
emerge logrotate ?
It rotates, compresses and mails them - if you want to.
_________________
>>> emerge (3 of 7) mcse/70-293 to /
Essential tools: gentoolkit eix profuse screen |
|
| Back to top |
|
|
drseergio
Apprentice
Joined: 28 Sep 2005
Posts: 236
|
| Posted: Wed Sep 28, 2005 7:14 pm Post subject: |
|
|
| Thanks for replies, I will try it. |
|
| Back to top |
|
|
DNAspark99
Guru
Joined: 03 Sep 2004
Posts: 321
|
| Posted: Wed Sep 28, 2005 7:55 pm Post subject: |
|
|
example of my setup :
/etc/syslog-ng/syslog-ng.conf :
| Code: |
options {
long_hostnames(off);
sync(0);
stats(43200);
};
source src {
unix-stream("/dev/log");
internal();
pipe("/proc/kmsg");
};
##REMOTE logging through stunnel##
destination remote {
tcp("127.0.0.1"
port(514));
};
log {source(src); destination(remote); };
#########################
#cron
destination cron { file("/var/log/cron.log"); };
filter f_cron { facility(cron); };
log { source(src); filter(f_cron); destination(cron); };
#mail
destination mail { file("/var/log/mail.log"); };
filter f_mail { facility(mail); };
log { source(src); filter(f_mail); destination(mail); };
#openvpn
destination openvpn { file("/var/log/openvpn.log"); };
filter f_openvpn { match ("openvpn"); };
log { source(src); filter(f_openvpn); destination (openvpn); };
#grsec
destination grsec {
file("/var/log/grsec.log"
owner(root) group(logger)
perm(0640) dir_perm(0700));
};
filter f_grsec { match ("grsec:"); };
log { source(src); filter(f_grsec); destination (grsec); };
#pax
destination pax {
file("/var/log/pax.log"
owner(root) group(logger)
perm(0640) dir_perm(0700));
};
filter f_pax { match ("PAX:"); };
log { source(src); filter(f_pax); destination (pax); };
#firewall
destination firewall { file ("/var/log/firewall.log"); };
filter f_firewall {
match ("IN-") or
match ("OUT-") or
match ("PASS-") or
match ("NEW TCP w/o SYN:") or
match ("SYN FLOOD:");
};
log { source(src); filter(f_firewall); destination (firewall); };
filter f_messages {
not facility(mail) and
not facility(cron) and
not filter(f_firewall) and
not filter(f_pax) and
not filter(f_grsec);
};
####################################################
#Default loging
log { source(src); filter(f_messages); destination(messages); };
destination messages { file("/var/log/messages"); };
destination console_all { file("/dev/tty12"); };
log { source(src); destination(console_all); };
|
/etc/logrotate.d/syslog-ng : (you'll need to add files to this dir for various application logfiles like apache, etc....read manpage for logrotate details  )
| Code: |
/var/log/messages {
sharedscripts
olddir /var/log/archive
postrotate
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
endscript
}
/var/log/cron.log {
olddir /var/log/archive
weekly
rotate 4
}
/var/log/mail.log {
olddir /var/log/archive
weekly
rotate 4
}
/var/log/firewall.log {
olddir /var/log/archive
size=512M
rotate 2
}
/var/log/grsec.log {
monthly
olddir /var/log/archive
rotate 6
}
/var/log/pax.log {
monthly
olddir /var/log/archive
rotate 6
}
|
|
|
| Back to top |
|
|
Jerem
Apprentice
Joined: 11 Jun 2004
Posts: 177
|
| Posted: Thu Jun 15, 2006 10:41 am Post subject: |
|
|
Had once a 1.7G logfile.
There was no log rotation for the antivirus... |
|
| Back to top |
|
|
JeliJami
Veteran
Joined: 17 Jan 2006
Posts: 1086
Location: Belgium
|
| Posted: Thu Jun 15, 2006 11:39 am Post subject: |
|
|
strange that noone mentioned logwatch or logcheck
_________________
Unanswered Post Initiative | Search | FAQ
Former username: davjel |
|
| Back to top |
|
|
Bones McCracker
Veteran
Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.
|
| Posted: Thu Oct 12, 2006 11:52 pm Post subject: |
|
|
Disappointing that there is no gentoo-specific config files for logcheck. One might as well write customized scripts.
I think there should be a logcheck.d directory in which applications deposit files to be included in the logcheck config files. |
|
| Back to top |
|
|
tcort
Retired Dev
Joined: 10 Jul 2004
Posts: 17
Location: Gatineau, QC, Canada
|
| Posted: Thu Oct 19, 2006 12:55 pm Post subject: |
|
|
| Once you get centralized logging working with syslog-ng over stunnel, you can have those logs piped into a MySQL database and view/search them with php-syslog-ng. There is a Gentoo Wiki article about it... http://gentoo-wiki.com/HOWTO_setup_PHP-Syslog-NG |
|
| Back to top |
|
|
|
Gentoo Chat
