syslog-ng Changelog

What's new in syslog-ng 3.24.1

Oct 8, 2019
  • Highlights:
  • Add a new template function called $(format-flat-json), which generates
  • flattened json output. This is useful for destinations, where the json
  • parsing does not handle nested json format. (#2890)
  • Add ISO 8601 compliant week numbering. Use it with the ${ISOWEEK} macro
  • and and all its variants: S_ISOWEEK, R_ISOWEEK and C_ISOWEEK. (#2878)
  • Add add-contextual-data() glob selector. It matches the message with shell
  • style globbing. Enable it by setting selector(glob("$my_template") in the
  • add-contextual-data() block. (#2936)
  • Add new rewrite operations to manipulate the timezone portion of timestamps have
  • been added. set-timezone() to set the timezone value to a specific value,
  • fix-timezone() to fix up an incorrectly recognized timezone and guess-timezone()
  • to automatically deduce the timezone value on the assumption that the message
  • is received in near real time. (#2818)
  • Send Server Name Identification (SNI) information with transport(tls).
  • Enable it by setting the sni(yes) option in the tls block in your
  • destination. (#2930)
  • Features:
  • templates: change the $LOGHOST macro to honour use-fqdn() (#2894)
  • Define syslog-ng-sysconfdir (#2932)
  • dqtool: add assign dqfile to persist file feature (#2872)
  • Bugfixes:
  • Fix backtick subsitution of defines/environment variables in the main configuration file. (#2906, #2909)
  • Fix SCL block parameter substitution of quoted escaped newline (#2901)
  • python, diskq, random-generator source: crash after failed reload (#2907)
  • Fix crash at shutdown on 32bit systems (#2893, #2895)
  • Invalidate the value of the LEGACY_MSGHDR macro in case either the PID or the PROGRAM
  • macros are unset() using a rewrite rule. Previously LEGACY_MSGHDR would retain the old values. (#2896)
  • on 32bit platform diskq ftruncate could fail due to size 32/64 interface (#2892)
  • Support new tzdata format, starting from version 2009.XXX, in tzinfo parser. (#2898)
  • udp, udp6, tcp, tcp6, syslog, network destination: Correctly detect and set IP_MULTICAST_TTL
  • in case of multicast ip address (#2905)
  • Fix hostname resolve on systems with only the loopback network interface configured (#2933)
  • wildcard-file(): Add multi-line(), pad_size() and multi-line-mode() option validation. (#2922)
  • kafka-c: Fix multiple memleaks (#2944)
  • Other changes:
  • geoip: remove deprecated module, geoip2 database location detection (#2780)
  • various refactor, build issue fixes (#2902)
  • Notes to the developers:
  • LightRunWithStrace: Run syslog-ng behind strace (#2921)
  • LightVerboseLogOnError: Increase default pytest verbosity on error (#2919)
  • Dbld image caching (#2858)
  • Dbld gradle caching (#2857)
  • logreader,logsource: move scratch-buffer mark and reclaim into LogSource (#2903)

New in syslog-ng 3.22.1 (Jun 27, 2019)

  • Highlights:
  • Sending SNMP traps: Using the new snmp() destination, incoming log messages
  • can be converted to SNMP traps, as the fields of the SNMP messages can be
  • customized with macros. (#2693)
  • $(template) dynamic binding: Extends the $(template) template function to
  • allow dynamic binding. For example, the name of the template to be invoked
  • can come from the message (name-value pairs). (#2716)
  • syslog(), network(): Add dynamic-window-size() option to enable dynamic
  • flow control that distributes the specified amount of window between active
  • connections at runtime. This can be used in low-memory environments, where
  • only a small subset of the active clients sends messages at high rate.
  • (#2772)
  • Features:
  • match(): Add support for the template() option (#2715)
  • add-contextual-data(): Allow using templates in name-value pairs (#2711)
  • Add support for floating point operations in template functions (#2742)
  • Add support for usec precision when parsing time (#2709)
  • Bugfixes:
  • Fix null pointer access when destinations are suspended (#2778)
  • Fix grouping-by() deadlock (#2758)
  • Fix a general source-related crash and enhance wildcard-file()'s bookmark
  • handling (#2589)
  • Fix infinite loop (reload/reopen) (#2739)
  • Fix python() package/module name collision (#2438)
  • Fix escaped quote in block argument (#2781)
  • Reintroduce test on SYSLOG_NG_HAVE_TIMEZONE (#2774)
  • snmp(): Fix template leak (#2746)
  • Other changes:
  • Never drop flow-controlled messages: The meaning of log-fifo-size() has
  • changed to avoid dropping flow-controlled messages when log-fifo-size() is
  • misconfigured. From now on, log-fifo-size() only affects messages that are
  • not flow-controlled. (#2753)
  • The -d/--debug syslog-ng command line flag no longer implies
  • -e/--stderr. If you want to redirect internal() source to stderr,
  • use the -e/--stderr option explicitly. (#2731)
  • dbld, RPM and DEB packaging improvements (#2724)
  • Checkpoint parser improvements (#2740)
  • Reset the timezone on config reload event (#2691)
  • geoip2(): Include IP into the error message (#2743)
  • Improve regexp error messages (#2796)
  • http(): Warn if less workers used than urls (#2757)
  • http(): Allow URLs to be specified by a space/comma separated string
  • (#2699)
  • loggen: Change message rate at runtime using signals (#2756)
  • debun: add acquire_running_syslog_config function (#2752)
  • FreeBSD fixes for the test suite (#2783)

New in syslog-ng 3.6.2 (Jan 7, 2015)

  • Features:
  • New parameter added to loggen: --permanent (-T) wich is for sending logs indefinitely.
  • Fixes:
  • From now, syslog-ng won't crash when using a Riemann destination and no attributes are set.
  • In some cases program destination respawned during syslog-ng stop/restart.
  • Max packet length for spoof source is set to 1024 (previously : 256).
  • Removed syslog.socket from service file on systems using systemd. Syslog-ng reads the messages directly from journal on systems with systemd.
  • In some cases, localtime related macros had a wrong value(eg.:$YEAR).
  • Transaction handling fixed in SQL destination. In some circumstances when both select and insert commands were run within a single transaction and the select failed (eg.: in case of mssql), the log messages related to the insert commands, broken by the invalid transaction, were lost.
  • Fixed a memleak in SQL destination driver. The memleak occured during one of the transaction failures.
  • A certificate which is not contained by the list of fingerprints is rejected from now.
  • Hostname check in tls certificate is case insensitive from now.
  • Fix spinning on EOF for `unix-stream()` sockets. Root cause of the spinning was that a unix-dgram socket was created even in case of unix-stream.

New in syslog-ng 3.6.1 (Nov 12, 2014)

  • This is the first production ready version of syslog-ng OSE 3.6. More than 25000 lines fof code changed, with about 500 file modified.
  • New dependencies:
  • PCRE is now a required dependency of syslog-ng, and is not optional anymore.
  • Changed defaults:
  • Threaded mode is now **enabled** by default. To turn it off, use `threaded(no)` in the global options section.
  • The versioning of the `libsyslog-ng` internal library has changed: instead of always using the current release number, we will now try to maintain ABI compatibility during the lifetime of a stable branch. Therefore, we use only the first two components of our version as the base of the library version. Another number will be part of the SONAME too, but that will only change when we break compatibility.
  • The SONAME is currently set to `libsyslog-ng-3.6.so.0`, and will remain the same during alpha and beta releases, even when the ABI changes. We will start bumping the version after the first stable release from this branch, if needed.
  • The `flush-lines()` setting now defaults to *100*, rather than *1*, for increased speed.
  • Features:
  • New options:
  • A new `custom-domain()` global setting was introduced, which allows the administrator to override the local domain name used by syslog-ng. It affects all locally generated log messages.
  • Added a `use-rcptid()` global option, that tells syslog-ng to assign a reception ID to each message received and generated by syslog-ng. This ID is available as the `$RCPTID` macro, and is unique on a given host. The counter wraps around at 48 bits and is never zero.
  • New drivers:
  • The `pseudofile()` destination driver is a very simple driver, aimed at delivering messages to special files in `/proc` or `/dev`. It opens and closes the file on each message, instead of keeping it open. It does not support templates in the filename, and does not have a queue (and as such, is not adequate in high traffic situations).
  • The new `nodejs()` source driver (implemented as an SCL macro) adds a source driver that allows syslog-ng to accept messages from node.js applications that use the `winston` logging API.
  • The new `systemd-syslog()` source replaces the former implicit support for the same thing. Users who use systemd are advised to use either the `system()` source, or this new one when they want to receive logs from systemd via the `/run/systemd/journal/syslog` socket.
  • The new source driver systemd-journal() reads from the Journal directly, not via the syslog forwarding socket. The `system()` source defaults to using this source when systemd is detected.
  • Added groupset rewrite object. Groupset allows the user to modify multiple log message properties at once. It also allows referencing the old value of the property as the $_ macro.
  • Features from the [Incubator][incubator]:
  • The `$(or)` template function that returns the first non-empty argument is now included in syslog-ng itself.
  • The `$(padding)` template function, to pad text with custom padding to a given length is also included.
  • The `$(graphite-output)` template function, to be used for sending metrics to [Graphite][graphite] was ported over from the Incubator. The `graphite()` destination SCL block is also available now, to make it even easier to talk to Graphite.
  • The `riemann()` destination, which allows sending metrics to the [Riemann][riemann] monitoring system was also ported over from the Incubator.
  • [graphite]: http://graphite.wikidot.com/
  • [incubator]: https://github.com/balabit/syslog-ng-incubator
  • [riemann]: http://riemann.io/
  • Threaded destinations:
  • A number of features were implemented for all threaded destinations:
  • `amqp()`, `mongodb()`, `redis()`, `riemann()`, `smtp()` and `stomp()`.
  • The destinations gained support for `SEQNUM` persistence: the counter will be preserved across reloads and restarts.
  • A new option called `retries()` was implemented for all of these, which controls how many times a message delivery is retried before dropping it.
  • The `throttle()` option is now implemented, and works for all of the aforementioned destination drivers.
  • The message delivery loop was optimised to do less sleep/wakeup cycles, which should make the drivers not only faster, but more CPU friendly too.
  • Miscellaneous new features:
  • The `multi-line-mode()` option gained a new setting: `prefix-suffix`, which works similarly to the `prefix-garbage` (which is the new name for `regexp`), except it appends the garbage part to the message, instead of discarding it.
  • This new mode can be used to work around the absence of a timeout.
  • Filters default to PCRE matching, instead of the previous POSIX regexp default.
  • The `system()` source will now parse `@cim` marked messages as JSON, if the JSON module is available at run-time. This improves inter-operation with other software that uses the *Common Information Model*.
  • One can now use multiple elements in the `key()` and `exclude()` options of any value-pairs declaration.
  • It is now possible to load not only a single certificate when using TLS, but a certificate chain.
  • Statistics:
  • The stats counter for PROGRAM counters now includes the timestamp of the last update.
  • A new `stats-lifetime()` global option was introduced, which controls how often dynamic counters are expired. The timer is not exact, some timers may live a little bit longer than the specified time.
  • Dynamic counters are now cleaned up every `stats-lifetime()` minutes (defaulting to 10 minutes) instead of only on reloads. This change was done to reduce the memory used by dynamic counters.
  • There is now an `internal_queue_length` statistic, which shows the length of the internal queue. This is most useful to see if the `internal()` source is not connected, or if it is not being emptied fast enough (which, again, indicates a more serious error).
  • MongoDB:
  • The `mongodb()` driver now supports authentication, even when using replica sets. When re-connecting to another member of the set, the driver will automatically re-authenticate.
  • The `--with-libmongo-client` option of the configure script now supports `auto` as a value, and will then detect whether to use the system version of the library or the internal copy. We default to `auto` now, which prefers the system library over the internal copy.
  • The driver does not automatically add an `_id` field to the message: the server will do that automatically, if none is present. This allows users to override the field from within their syslog-ng config.
  • A new `retries()` option can be used to tell the driver how many times it should try to insert a message into the database before giving up (defaults to 3). This fixes the case where a rogue message could hold up the entire queue, as it was retried forever.
  • The driver now enables `safe-mode()` by default.
  • There is now a one-minute timeout for MongoDB operations. If an operation times out, it will be considered failed.
  • The driver can now connect to MongoDB via UNIX domain sockets.
  • The `double()` type hint is now supported by the driver.
  • In the MongoDB destination, reconnecting in a replica-set environment now works correctly, and reliably.
  • To build syslog-ng with the MongoDB destination, libmongo-client version 0.1.8+ is now required. (The internal copy has been updated accordingly.)
  • SMTP destination changes:
  • The `smtp()` destination now supports a `retries()` option, which controls how many times a message delivery will be attempted before dropping it.
  • The templates used in the destination now honor the time-zone settings.
  • The driver will abort if required options (any of `to()`, `cc()`, `bcc()` and `from()`, and `subject()` and `body()`) are not set.
  • Unix Domain Sockets:
  • The `unix-dgram()` and `unix-stream()` sources now extract UNIX credentials (PID, UID and GID of the sending application) from the passed messages, if any. On Linux, and FreeBSD, the path of the executable belonging to PID is extracted too, along with command-line arguments.
  • The extracted values are available in `${.unix.pid}`, `${.unix.uid}`, `${.unix.gid}`, `${.unix.exe}` and `${.unix.cmdline}`, respectively.
  • The `system()` source will overwrite the PID macro with the value of `${.unix.pid}`, if present.
  • JSON:
  • The json-parser gained an `extract-prefix()` option, which can be used to tell the parser to only extract JSON members from a specific subtree of the incoming object.
  • Example: `json-parser(extract-prefix("foo.bar[5]"));`
  • Assuming that the incoming object is named msg, this is equivalent to the following javascript code: `msg.foo.bar[5]`
  • The resulting expression must be a JSON object, so that syslog-ng can extract its members into LogMessage name-value pairs.
  • This also works when the top-level object is an array, as `extract-prefix()` allows the use of an array index at the first indirection level, for example: `json-parser(extract-prefix("[5]"));`, which translates to `msg[5]`.
  • The `$(format-json)` template function now handles the `double()` type hint.
  • Debugging:
  • When sending messages to stderr in debug mode, prepend a timestamp to the messages.
  • The new `$RUNID` macro is available for templates, which changes its value every time syslog-ng is restarted, but not when reloaded.
  • A Valgrind suppression file was added (available under `contrib/valgrind/`), to aid in debugging memory leaks in syslog-ng. It supresses a couple of known false positives, and a few other things in third-party libraries.
  • A new utility, `system-expand`, was added, which returns what the `system()` source would expand to.
  • Bugfixes:
  • The reliability of the `usertty()` destination driver was greatly improved. Previously, some parts of it were not thread-safe, which could result in strange behaviour.
  • The handling of escape related flags of `csvparser()` was changed: instead of these flags overwriting all other (even non-escape related) flags, if the flag to set is an escape-flag, it will keep all non-escape flags, and set the new one. If it is a not such a flag, then it will clear all flags, and set the previous escape flags, and the new flag.
  • This, in essence, means that when setting flags on a `csvparser()`, if it is an escape flag, only escape flags will be affected. If not, then escape flags will not be affected at all.
  • The SQL destination now correctly continues $SEQNUM counting after a reload, instead of starting afresh.
  • Casting error eliminated in Riemann destination when metric is applied to an empty field.
  • From now, syslog-ng always exclude attributes that conflict with properties in Riemann destination (otherwise value of the attribute would override the property).
  • When tring to stop syslog-ng while a reload is in progress, syslog-ng will now correctly shut down cleanly.
  • Reloading a config file containing runtime error now not ends in a crash, it is able to fallback to the original config. (runtime error: config file is grammatically valid but containing invalid value, eg.: wrong database column name)
  • When the local hostname is not an FQDN, and the local resolver fails to return an FQDN too, syslog-ng does not abort anymore, but continues using a non-FQDN hostname after emitting a warning on the internal source.
  • Furthermore, syslog-ng will try to resolve the FQDN harder: when multiple names are returned, it will search for the first FQDN one, instead of stopping at the primary name.
  • The `update-patterndb` script will now work correctly when the current working directory contains .pdb files.
  • Patterndb fixed to apply condition even if context-id is missing.
  • We will now correctly handle time going backwards in patterndb: it will realign its idea of current time with the system. This corrects a bug where timeouts did not function properly when system time was set backwards.
  • The `pdbtool merge` command will now generate version 4 patterndb files.
  • The Linux capability support is now correctly auto-detected by the configure script, and defaults to off on FreeBSD 9+, as it should.
  • The `file()` and `network()` (including `tcp()` et al) sources will now properly set the `$SOURCE` macro.
  • The basicfuncs module was fixed to work correctly on 32-bit architectures.
  • The `stored` statistics is no longer incremented by various drivers when they mean `processed`.
  • The type hinting feature is now more picky about what kind of type hints it accepts, allowing one to use template functions in - for example - `$(format-json)` pairs.
  • All the various crypto-related template functions now check that the desired length of the digest is not larger than the digest itself. If a larger value is requested, they will truncate it to the digest length.
  • The `$(geoip)` template function now works with `threaded(yes)` too.
  • The `in-list()` filter was fixed to look at all elements of the list, instead of only the last one.
  • Fixed an assertion when using the `match()` filter under certain circumstances.
  • The `system()` source will not add `/dev/kmsg` (or `/proc/kmsg` on older kernels) to the default sources if using the systemd journal, because kernel logs are included in the journal.
  • The `system()` source will not include `/dev/kmsg` (or `/proc/kmsg`) when running inside a Linux container.
  • Various memory leak fixes around the code base.
  • Change control socket message from notice to debug
  • Opening control socket disabled when syslog-ng is used for only syntax-checking.
  • Fixes for retries() functionality. Retry counter incremented by every message write error (including network connection errors) which can lead to message lost.
  • Miscellaneous changes:
  • We now ship a "Contributors Guide" in the `CONTRIBUTING.md` file.

New in syslog-ng 3.5.1 (Nov 11, 2013)

  • This is the first stable release in the 3.5 series, adding a number of features compared to 3.4, a result of about eight months of development. This release includes all the fixes of the recent 3.4.5 release, and a host of new features.
  • Bugfixes:
  • A race condition in log message reference counting code that sometimes led to crashes was fixed. [#255]
  • A use-after-free error that sometimes happened after a reload, and caused memory corruption was also fixed. [#253]
  • patterndb was corrected not to create a new instance on reload: this way, the old one is not leaked, and db-parser() does not forget the correlation state, nor its idea of current time on reload. [#257]
  • The syslog-ng.spec file does not try to install the long-removed ChangeLog file anymore.

New in syslog-ng 3.5.0 Beta 3 (Oct 16, 2013)

  • Template escaping:
  • Template escaping was changed in an incompatible way: previously, both the lexer and the template compiler used the '\' character for escaping, which was confusing. The template compiler uses '$$' to escape the '$' char, and '@@' to escape '@'.
  • If a non-numeric value follows '@', a warning will be printed.
  • Bugfixes:
  • syslog-ng should compile again on non-Linux platforms.
  • The flush() and fsync() options of the file destination were fixed, and they should work the same way now as they do in 3.3 and 3.4.
  • The hiredis library should be detected on the default include paths, so one does not necessarily need to specify --with-libhiredis non non-Debian systems. Said option was also made to work.
  • A memory corruption was fixed in the @STRING@ parser of db-parser().
  • Excludes now work properly with value-pairs(), and they do not get ignored if the value to exclude is in the default set.

New in syslog-ng 3.3.2 (Nov 15, 2011)

  • Stability and memory leak fixes.

New in syslog-ng 3.3.1 (Oct 10, 2011)

  • Integrated support for MongoDB, JSON formatted events, and a multi-threaded architecture that scales syslog-ng up into the 800000 message/second range.

New in syslog-ng 3.2.1 (Nov 29, 2010)

  • This is the first release in the new major version of syslog-ng, containing the longest list of features ever since the start of the syslog-ng project such as log message correlation and plugin support.

New in syslog-ng 3.2 Alpha 2 (Aug 8, 2010)

  • Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64)
  • Fixed pdbtool match --debug-pattern output for ESTRING parsers.
  • Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs.
  • Fixed Solaris STREAMS device support.
  • Forward ported all bugfixes from syslog-ng OSE 3.0 & 3.1
  • Disable process accounting module by default as it doesn't compile on non-Linux platforms.
  • Added "pdbtool match --file" option to read and parse an existing logfile.
  • Added "pdbtool test" to check the log samples in the patterndb file.
  • Added "dont-create-tables" flag for the SQL destination to inhibit automatic table creation.
  • Added "condition()" support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression.
  • Added "--module-path" command line option to control where modules are loaded from from the command line.

New in syslog-ng 3.1 Beta 1 (Dec 3, 2009)

  • Support for patterndb v3
  • pdbtool
  • Message tags
  • Rewrite structured data
  • Macro and name-value integration
  • Name-value pair performance improvements
  • Patterndb parser enhancements
  • Information about non-portable facilities
  • Name-value pair performance improvements
  • Patterndb parser enhancements
  • Information about non-portable facilities

New in syslog-ng 3.0.2 (May 31, 2009)

  • The first official version to feature binary packages for Linux and BSD platforms.

New in syslog-ng 3.0.1 (Mar 5, 2009)

  • Version 3.0 of syslog-ng supports the new syslog protocol standards developed by the Internet Engineering Task Force (IETF)
  • The capability to encrypt log connections using TLS has been added.
  • Log statements can be embedded into each other, making it possible to design complex log paths.
  • The encoding of source files can be set for proper character conversion (internally syslog-ng represents every message in UTF-8).
  • The syslog-ng application assigns a unique message identification number to every log message, making it easy to detect if any messages are lost.
  • The syslog-ng application can read, process, and rewrite structured messages (e.g., Apache webserver logs) using templates and regular expressions. Both messages with fixed field sizes and fields separated with delimiters (e.g., comma-separated values) are supported.