slurp

0.5 ISC License (ISCL)    
  UNRATED

  152 downloads

Log file slurper

description

download

specs

slurp iterates over "entries" in log files (sources), parsed them into something structured and passes them along to something else (sinks). A log file is something that:

- is created
- has strings appended to it
- is then possibly deleted.

If a file does not conform to this lifestyle it is not suitable for use with slurp.

In the slurp world files are mapped to consumers which are Python dictionaries describing:

- what files are associated with the consumer
- how to identify raw "entry" strings in them
- how to parse those "entries" to something structured
- where to send those parsed entries

The motivating use-case for slurp is feeding entries streamed to centralized syslog spool(s) to elastic search and other data mining tools.

Install

Simply::

     pip install slurp

or if you prefer::

     easy_install slurp

Usage

Slurp has both programming and command-line interfaces.

To use the programming interface import it and read doc strings::

    $ python
    >>> import slurp


To use the command-line interface run the slurp script::

    $ slurp --help
    Usage:
    slurp s|seed path-1 .. path-n [options]
    slurp m|monitor path-1 .. path-n [options]
    slurp e|eat path-1 .. path-n [options]

    Options:
      -h, --help            show this help message and exit
      -s STATE_PATH, --state-path=STATE_PATH
      -c CONSUMERS, --consumer=CONSUMERS
      -l LOG_LEVEL, --log-level=LOG_LEVEL
      --enable-syslog
      --disable-stderrlog
      -d, --daemonize
      --disable-locking
      --lock-timeout=LOCK_TIMEOUT
      --disable-tracking
      --pid-file=PID_FILE
      --sink=SINK
      --batch-size=BATCH_SIZE


Another common use case is to run the slurp script as a monitor daemon. See extras/slurp.init for an example init script.
read more   
Last updated on June 21st, 2012

0 User reviews so far.

SUBMIT

#log slurper #slurp logs #log #slurper #logging #Python