Log Management

bokuholio.RemoveThis@gmail.com wrote:
> I would like to find a tool, where from a central server, I can
> collect those daily rolled logs from each of the servers. Ideally it
> would verify that the file was received, remove it from the host, gzip
> it up and retain it for some predetermined period in a directory
> hierarchy.

you've got a lot of options:

REMOTE LOGHOST
someone in this thread already mentioned that you can configure your webservers
to log to a centralized syslog loghost:
http://www.oreilly.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html

REMOTE THIRD PARTY ANALYTICS
if you just need marketing analytics (e.g. unique cookied visitor behavior), not
operational metrics, then you can install third-party analytics on each machine,
and disable the local logging altogether: http://google.com/analytics/

REMOTE MESSAGING PROTOCOL
if your machines are on the same physical network, you can use the spread
messaging library and mod_log_spread to send all log data to a central recorder:
http://www.backhand.org/mod_log_spread/
https://alioth.debian.org/frs/?group_id=30823

REMOTE DATABASE
if you use a database like postgresql or mysql, you can configure your web
servers to record their log entries to a remote database server:
http://pgfoundry.org/projects/modpglogger
http://bitbrook.de/software/mod_log_mysql/

UTILITIES FOR LOCAL LOG MAINTENANCE

for local log file maintenance, i don't know of anything that eliminates the
scripting "glue" that is necessary to push or pull rotated logs to a central
location, but you can use a few things to make it easier:

cronolog can be configured as an inline logging application that will write out
uniquely named logfiles as often as you need them: http://cronolog.org/

rsync is a differential / incremental file transfer tool, you can use it to pull
files over to your loghost, and delete remote files after they've been safely
written to the central machine. if a transfer is interrupted, rsync will pick up
where it left off, which can safe you a lot of time: http://rsync.samba.org/

if you're trying to merge logs from load-balanced web servers into a single
comprehensive log corpus, you can use perl or shell commands to sort and de-dupe
the data, but you'll find this utility to be much faster:
http://mergelog.sourceforge.net/

hth
--sean

--
sean dreilinger - http://durak.org/sean/
 >> Stay informed about: Log Management 

On Feb 21, 10:55 pm, mels... RemoveThis @aragorn.rgmhome.net (Robert Melson)
wrote:
> In article <26442534-50d2-4a93-9d5e-276f02c4d... RemoveThis @s12g2000prg.googlegroups.com>,
>         bokuho... RemoveThis @gmail.com writes:
>
> > I'm looking for someone's recommendation on a tool for log management.
>
> > I have a number of physical webservers running various Apache/IHS,
> > Siteminder, Websphere, etc.  All the webservers have very small
> > internal storage on them, and reside in locations that don't readily
> > have access to SAN/NAS connectivity.
>
> > I would like to find a tool, where from a central server, I can
> > collect those daily rolled logs from each of the servers.  Ideally it
> > would verify that the file was received, remove it from the host, gzip
> > it up and retain it for some predetermined period in a directory
> > hierarchy.
>
> > All the tools that I've seen seem more geared for analysis
> > (statistics) or security (firewall log management).
>
> > I would prefer open source, but at this point anything will do.
>
> > Is anyone able to recommend a product?
>
> > Thank you in advance!
>
> I can't recommend a product, but why not "roll your own" with
> a perl or python script?  It would be a trivial exercise to
> write a script that'd reside on each box, rather than polling
> the collection.  The script could grab the logfile(s), gzip'em
> and email'em or ftp'em  to your central workstation for archiving.
>
> Bob
>
> r
> --
> Robert G. Melson | Rio Grande MicroSolutions | El Paso, Texas
> -----
> Thinking is the hardest work there is, which is the probable
> reason so few engage in it.  -- Henry Ford

This is what I was planning to do, but I was hoping that there might
be a tool that would be more usable by my management who want to have
access to the logs, establish their own retention schedules, grant /
deny access for members of their team.

As a geek, I was going to just script it all to collect - but was
hoping someone, somewhere made a product that actually put some
management around logs in a nice, p.h.b. friendly package.

The other suggestions are great, and I really appreciate your time in
responding.
 >> Stay informed about: Log Management