Throughput 0.1

Throughput monitor is a log analyzer. In general notation it is a event counter per timeframe - in short frquency monitor. If it detects too high event rate, utility notifies about it. If rate drops below predefined value, also notify with statistics gathered meanwhile. It can analyze past logs or realtime logs.

Throughput monitor can analyze every single-line log (1 event per line). Good examples are of this kind of log producers are syslog and apache. Log lines are filtered through regex. Log can contain random crap, except on line we are interested in. Usually most of log is useless. The interesting lines must be somewhat consistent and regex must be able to detect:

timestamp in format: Day Mon dd hh:mm:ss yyyy
observerion object like username or hostname, etc

Output can be only hwm or lwm line. If amount of events per timeframe rises above hwm, then hwm line is generated. The format of hwm line is following:
Day Mon dd hh:mm:ss yyyy hwm obj

Day Mon dd hh:mm:ss yyyy - timestamp
hwm - fixed string
obj - can be username or hostname or whatever user's regex matches as observation object

Similarly lwm line is generated when rate drops below lwm. Also additional statistics is included. The format of lwm line is following:
Day Mon dd hh:mm:ss yyyy lwm obj max=zz count=xx duration=sec/hh:mm:ss interval=hh:mm:ss

Day Mon dd hh:mm:ss yyyy - timestamp
lwm - fixed string
obj - can be username or hostname or whatever user's regex matches as observation object
max=zz - the peak amount (zz) of event in timeframe
count=xx - count of events while rate is over hwm. May be smaller than max. Because when rate crosses hwm line, count is initalized to 1, but max at the same moment is hwm.
duration=sec/hh:mm:ss - how long object were in hwm state. Duration is given in both forms: seconds for automation and hhmmss for humans.
interval=hh:mm:ss - average delay between events. Equals to duration/count.

One may ask, why do i need lwm output when detecting spam? Because statistics included gives feedback about parameters to tune and also we get quick info about incident.

Know that the utility may complain about timetravel when log is not linear. In the other words, every next matched log entry must have timestamp, which is greater or equal then previous one of the same object. Unfortunately this is too common problem with logs in general.

Product's homepage