slurp iterates over "entries" in log files (sources), parsed them into something structured and passes them along to something else (sinks). A log file is something that:
- is created
- has strings appended to it
- is then possibly deleted.
If a file does not conform to this lifestyle it is not suitable for use with slurp.
In the slurp world files are mapped to consumers which are Python dictionaries describing:
- what files are associated with the consumer
- how to identify raw "entry" strings in them
- how to parse those "entries" to something structured
- where to send those parsed entries
The motivating use-case for slurp is feeding entries streamed to centralized syslog spool(s) to elastic search and other data mining tools.
pip install slurp
or if you prefer::
Slurp has both programming and command-line interfaces.
To use the programming interface import it and read doc strings::
>>> import slurp
To use the command-line interface run the slurp script::
$ slurp --help
slurp s|seed path-1 .. path-n [options]
slurp m|monitor path-1 .. path-n [options]
slurp e|eat path-1 .. path-n [options]
-h, --help show this help message and exit
-s STATE_PATH, --state-path=STATE_PATH
-c CONSUMERS, --consumer=CONSUMERS
-l LOG_LEVEL, --log-level=LOG_LEVEL
Another common use case is to run the slurp script as a monitor daemon. See extras/slurp.init for an example init script.