pmacct Changelog

New in version 1.5.0 RC1

December 18th, 2013
  • Introducing custom-defined aggregation primitives: primitives are defined via a file pointed by aggregate_primitives config directive. The feature applies to NetFlow v9/IPFIX fields only, and with a pre-defined length. Semantics supported are: 'u_int' (unsigned integer, presented as decimal number), 'hex' (unsigned integer, presented as hexa- decimal number), 'ip' (IP address), 'mac' (MAC address)and 'str' (string). Syntax along with examples are available in the 'examples/primitives.lst' file.
  • Introducing JSON output in addition to tabular and CSV formats. Suitable for injection in 3rd party tools, JSON has the advantage of being a self- consisting format (ie. compared to CSV does not require a table title). Library leveraged is Jansson, available at: http://www.digip.org/jansson/
  • Introducing RabbitMQ/AMQP pmacct plugin to publish network traffic data to message exchanges. Unicast, broadcast, load-balancing scenarios being supported. amqp_routing_key supports dynamic elements, like the value of peer_src_ip and tag primitives or configured post_tag value, enabling selective delivery of data to consumers. Messages are encoded in JSON format.
  • pre_tag_map (and other maps): 'ip' key, which is compared against the IP address originating NetFlow/IPFIX or the AgentId field in sFlow, can now be an IP prefix, ie. XXX.XXX.XXX.XXX/NN, so to apply tag statements to set of exporters or 0.0.0.0/0 to apply to any exporter. Many thanks to Stefano Birmani for his support.
  • Re-introducing support for Cisco ASA NSEL export. Previously it was just a hack. Now most of the proper work done for Cisco NEL is being reused: post_nat_src_host (field type #40001), post_nat_dst_host (field type #40002), post_nat_src_port (field type #40003), post_nat_dst_port (field type #40004), fw_event (variant of nat_event, field type #40005) and timestamp_start (observation time in msecs, field type #323).
  • Introducing MPLS-related aggregation primitives decoded from NetFlow v9/ IPFIX, mpls_label_top mpls_label_bottom and mpls_stack_depth, so to give visibility in export scenarios on egress towards core, MPLS interfaces.
  • mpls_vpn_rd: primitive value can now be sourced from NetFlow v9/IPFIX field types #234 (ingressVRFID) and #235 (egressVRFID). This is in addition to existing method to source value from a flow_to_rd_map file.
  • networks_file: AS field can now be defined as "_", Useful also to define (or override) elments of an internal port-to-port traffic matrix.
  • print plugin: creation of intermediate directory levels is now supported; directories can contain dynamic time-based elements hence the amount of variables in a given pathname was also lifted to 32 from 8.
  • print plugin: introduced print_history configuration directive, which supports same syntax as, for example, sql_history. When enabled, time- related variables substitution of dynamic print_output_file names are determined using this value instead of print_refresh_time one.
  • Introducing IP prefix labels, ie. for custom grouping of own IP address space. The feature can be enabled by a --enable-plabel when configuring the package for compiling. Labels can be defined via a networks_file.
  • mongo_user and mongo_passwd configuration directive have been added in order to support authentication with MongoDB. If both are omitted, for backward compatibility, authentication is disabled; if only one of the two is specified instead, the other is set to its default value.
  • Introducing mongo_indexes_file config directive to define indexes in collections with dynamic name. If the collection does not exist yet, it is created. Index names are picked by MongoDB.
  • print plugin: introduced print_output_file_append config directive: if set to true allows the plugin to append to an output file rather than overwrite.
  • bgp_agent_map: added bgp_port key to lookup a NetFlow agent also against a BGP session port (in addition to BGP session IP address/router ID): it aims to support scenarios where BGP sessions do NAT traverals.
  • peer_dst_ip (BGP next-hop) can now be inferred by MPLS_TOP_LABEL_ADDR (NetFlow v9/IPFIX field type #47). This field might replace BGP next-hop when NetFlow is exported egress on MPLS-enabled core interfaces.
  • Introducing [nf|pm|sf|u]acctd_proc_name config directives to define the name of the core process (by default always set to 'default'). This is the equivalent to instantiate named plugins but for the core process. Thanks to Brian Rak for bringing this up.
  • pre_tag_map: introduced key 'flowset_id' to tag NetFlow v9/IFPIX data records basing on their flowset ID value, part of the flowset header.
  • pmacct client: introduced '-V' command-line option to verify version, build info and compile options passed to the configure script; also a new -a option now allows to retrieve supported aggregation primitives and their description.
  • Check for mallopt() has been added at configure time. mallopt() calls are introduced in order to disable glibc malloc() boundary checks.
  • ! flow_to_rd_map replaces iface_to_rd_map, increasing its scope: it is now possible to map couples to BGP/ MPLS VPN Route Distinguishers (RD). This is in addition to existing mapping method basing on .
  • ! fix, nfacctd, sfacctd: Setsocksize() call effectiveness is now verified via a subsequent getsockopt(). If result is different than expected, an informational log message is issued.
  • ! fix, building system: removed stale check for FreeBSD4 and introduced check for BSD systems. If on a BSD system, -DBSD is now passed over to the compiler.
  • ! fix, tee plugin: transparent mode now works on FreeBSD systems. Patch is courtesy by Nikita V. Shirokov.
  • ! fix, peer_dst_ip: uninitialized pointer variable was causing unexpected behaviours. Thanks to Maarten Bollen for his support resolving this.
  • ! fix, IMT plugin: selective queries with -M and -N switches verified not working properly. Thanks to Acipia organization for providing a patch.
  • ! fix, sql_common.c: src_port and dst_port primitives correctly spelled if used in conjunction with BGP primitives. Thanks to Brent Van Dussen and Elisa Jasinska for flagging the issue.
  • ! fix, building system: added library checks in /usr/lib64 for OS's where it is not linked to /lib where required.
  • ! fix, print, MongoDB and AMQP plugins: P_test_zero_elem() obsoleted. Instead, the cache structure 'valid' field is used to commit entries to the backend.
  • ! fix, nfacctd: in NetFlow v9/IPFIX, if no time reference is specified as part of records, fall back to time reference in datagram header.
  • ! fix, MongoDB plugin: mongo_insert_batch() now bails out with MONGO_FAIL if something went wrong while processing elements in the batch and an error message is issued. Typical reason for such condition is batch is too big for the resources, mainly memory, available. Thanks very much to Maarten Bollen for his support.
  • ! fix, cfg_handlers.c: all functions parsing configuration directives, and expecting string arguments, are now calling lower_string() so to act as case insensitive.
  • ! fix, IPv6 & NetFlow exporter IP address: upon enabling IPv6, NetFlow exporter IP addresses were written as IPv4-mapped IPv6 address. This was causing confusion when composing maps since the 'ip' field would change depending on whether IPv6 was enabled or not. This is now fixed and IPv4- mapped IPv6 addresses are now internally translated to plain IPv4 ones.
  • ! fix, nfacctd: NetFlow v9/IPFIX source/destination peer ASN information elements have been found mixed up and are now in proper order.

New in version 0.14.0 (May 9th, 2012)

  • This version integrates an IS-IS daemon, which is being run as a parallel thread within the collector core process.
  • It implements a single L2 P2P neighborship, i.e. over a GRE tunnel, P2P Hello, CSNP, and PSNP, and does not send any LSP information out.
  • A new aggregation primitive 'etype' is introduced in order to support accounting against the EtherType field of Ethernet frames.
  • Support for samples generated on ACL matches in Brocade (sFlow sample type: Enterprise: #1991, Format: #1) is now also introduced.
  • Several bugfixes are also included in this release.

New in version 0.14.0 RC3 (December 21st, 2011)

  • The BGP daemon now features an implementation of BGP/MPLS VPNs (rfc4364): it correlates couples to Route Distinguisher (RD) values.
  • The print plugin can now write network traffic data to flat-files (print_output_file) featuring formatted or CSV output, dynamic filenames, historical breakdown, and triggers.
  • pmacctd now supports the DLT_LOOP link-type (i.e. OpenBSD tunnel interfaces).
  • Several enhancements to the uacctd daemon and other minor new features and miscellaneous fixes are included.

New in version 0.14.0 RC2 (November 21st, 2011)

  • A sampling_map feature has been introduced, allowing definition of static traffic sampling mappings.
  • Further work on the NetFlow v9/IPFIX sampling includes support for 16 bits SAMPLER_IDs (seen against IOS-XR) and support for (FLOW)_SAMPLING_INTERVAL fields as part of the NetFlow v9/IPFIX data record.
  • [ns]facctd_as_new and [ns]facctd_net both feature a new "fallback" option to look up BGP-related primitives against BGP first and, if not successful, against the export protocol.
  • Other minor new features and misc.
  • Fixes are also included in this release.

New in version 0.12.5 (December 30th, 2010)

  • nfacctd, the pmacct NetFlow collector, has been enhanced to support NAT L3/L4 fields (IPFIX field types 225, 226, 227, 228, 281, and 282), Cisco ASA NetFlow v9 NSEL (field types 40001, 40002, 40003, 40004 152, 153, and 323) and application classification (aligning to Cisco NBAR-NetFlow v9 integration).
  • The NetFlow probe plugin, nfprobe, introduces egress IPv4/IPv6 NetFlow v9 templates.
  • The BGP thread now supports IPv6 transport and mapping to sFlow/NetFlow agents; BGP peer information is linked into the status table for caching purposes, which results in good CPU savings on bigger deployments.

New in version 0.12.4 (October 4th, 2010)

  • BGP daemon: a new memory model is introduced by which IP prefixes are being shared among the BGP peers RIBs - leading to consistent memory savings whenever multiple BGP peers export full tables due to the almost total overlap of information. Longest match nature of IP lookups required to raise BGP peer awareness of the lookup algorithm.
  • Updated INTERNALS document to support estimation of the memory footprint of the daemon.
  • BGP daemon: a new bgp_table_peer_buckets configuration directive is introduced: per-peer routing information is attached to IP prefixes and now hashed onto buckets with conflict chains. This parameter sets the number of buckets of such hash structure; the value is directly related to the number of expected BGP peers, should never exceed such amount and is best set to 1/10 of the expected number of peers.
  • nfprobe: support has been added to export direction field (NetFlowv9 field type #61); its value, 0=ingress 1=egress, is determined via nfprobe_direction configuration directive.
  • nfacctd: introduced support for Cisco ASA bytes counter, NetFlow v9 field type #85. Thanks to Ralf Reinartz for his support.
  • nfacctd: improved flow recognition heuristics for cases in which Pv4/IPv6/input/output data are combined within the same NetFlowv9 template. Thanks to Carsten Schoene for his support.
  • fix, BGP daemon: bgp_nexthop_followup was not working correctly if pointed to a non-existing next-hop.
  • fix, nfv9_template.c: ignoring unsupported NetFlow v9 field types; improved template logging. Thanks to Ralf Reinartz for his support.
  • fix, print plugin: support for interfaces and network masks has been added. Numeric variables output converted to unsigned from signed.

New in version 0.12.3 (July 30th, 2010)

  • 'cos' aggregation primitive has been implemented providing support for 802.1p priority. Collection is supported via sFlow, libpcap and ULOG; export is supported via sFlow.
  • BGP daemon: TCP MD5 signature implemented. New 'bgp_daemon_md5_file' configuration directive is being added for the purpose of defining peers and their respective MD5 keys, one per line, in CSV format. The map is reloadable at runtime: existing MD5 keys are removed via setsockopt(), new ones are installed as per the newly supplied map. Sample map added in 'examples/bgp_md5.lst.example'.
  • BGP daemon: added support for RFC3107 (SAFI=4 label information) to enable receipt of labeled IPv4/IPv6 unicast prefixes.
  • nfprobe, sfprobe: introduced the concept of traffic direction. As a result, [ns]fprobe_direction and [ns]fprobe_ifindex configuration directives have been implemented.
  • [ns]fprobe_direction defines traffic direction. It can be statically defined via 'in' or 'out' keywords; values can also be dynamically determined through a pre_tag_map (1=input, 2=output) by means of 'tag' and 'tag2' keywords.
  • [ns]fprobe_ifindex either statically associate an interface index (ifIndex) to a given [ns]fprobe plugin or semi-dynamically via lookups against a pre_tag_map by means of 'tag' and 'tag2' keywords.
  • sfprobe: sfprobe_ifspeed configuration directive is introduced and aimed at statically associating an interface speed to an sfprobe plugin.
  • sfprobe: Switch Extension Header support added. Enabler for this development was support for 'cos' and in/out direction. Whereas VLAN information was already supported as an aggregation primitive.
  • sfprobe: added support for Counter Samples for multiple interfaces. Sampling function has been brought to the plugin so that Counter Samples can be populated with real bytes/packets traffic levels. ! nfprobe, sfprobe: send buffer size is now aligned to plugin_pipe_size, if specified, providing a way to tune buffers in case of sustained exports. ! fix, addr.c: pm_ntohll() and pm_htonll() routines rewritten. These are aimed at changing byte ordering of 64-bit variables. ! fix, BGP daemon: support for IPv6 global address/link-local address next-hops as part of MP_REACH_NLRI parsing. ! fix, cfg_handlers.c: bgp_daemon and bgp_daemon_msglog parsing was not correct, ie. enabled if specified as 'false'. Thanks to Brent Van Dussen for reporting the issue. ! fix, bgp.c: found a CPU hog issue caused by missing cleanup of the select() descriptors vector. ! fix, pmacct.c: in_iface/out_iface did erroneously fall inside a section protected by the "--disable-l2" switch. Thanks to Brent Van Dussen for reporting the issue.

New in version 0.12.1 (April 8th, 2010)

  • Input/output interfaces (SNMP indexes) have now been implemented natively; it's therefore not required anymore to pass through the (Pre-)tag infrastructure. As a result two aggregation primitives are being introduced: 'in_iface' and 'out_iface'.
  • Support for source/destination IP prefix masks is introduced via two new aggregation primitives: src_mask and dst_mask. These are populated as defined by the [nf|sf|pm|u]acctd_net directive: NetFlow/sFlow protocols, BGP, Network files (networks_file) or static (networks_mask) being valid data sources.
  • A generic tunnel inspection infrastructure has been developed to benefit both pmacctd and uacctd daemons. Handlers are defined via configuration file. Once enabled daemons will account basing upon tunnelled headers rather than the envelope. Currently the only supported tunnel protocol is GTP, the GPRS tunnelling protocol (which can be configured as: "tunnel_0: gtp, "). Up to 8 different tunnel stacks and up to 4 tunnel layers per stack are supported. First matching stack, first matching layer wins.
  • uacctd: support for the MAC layer has been added for the Netlink/ ULOG Linux packet capturing framework.
  • 'nfprobe_source_ip' feature introduced: it allows to select the IPv4/IPv6 address to be used to export NetFlow datagrams to the collector.
  • nfprobe, sfprobe: network masks are now exported via NetFlow and sFlow. 'pmacctd_net' and its equivalent directives define how to populate src_mask and dst_mask values. ! cleanup, nfprobe/sfprobe: data source for 'src_as' and 'dst_as' primitives is now expected to be always explicitely defined (in line with how 'src_net' and 'dst_net' primitives work). See the UPGRADE doc for the (limited) backward compatibility impact. ! Updated SQL documentation: sql/README.iface guides on 'in_iface' and 'out_iface' primitives; sql/README.mask guides on 'src_mask' and 'dst_mask' primitives; sql/README.is_symmetric guides on 'is_symmetric' primitive. ! fix, nfacctd.h: source and destination network masks were twisted in the NetFlow v5 export structure definition. Affected releases are: 0.12.0rc4 and 0.12.0. ! fix, nfprobe_plugin.c: l2_to_flowrec() was missing some variable declaration when the package was configured for compilation with --disable-l2. Thanks to Brent Van Dussen for reporting the issue. ! fix, bgp.c: bgp_attr_munge_as4path() return code was not defined for some cases. This was causing some BGP messages to be marked as malformed. ! fix, sfprobe: a dummy MAC layer was created whenever this was not included as part of the captured packet. This behaviour has been changed and header protocol is now set to 11 (IPv4) or 12 (IPv6) accordingly. Thanks to Neil McKee for pointing the issue. ! workaround, building sub-system: PF_RING enabled libpcap was not recognized due to missing of pcap_dispatch(). This is now fixed.

New in version 0.12.0 (February 17th, 2010)

  • 'is_symmetric' aggregation primitive has been implemented: aimed at easing detection of asymmetric traffic. It's based on rule definitions supplied in a 'bgp_is_symmetric_map' map, reloadable at runtime.
  • A new 'bgp_daemon_allow_file' configuration directive allows to specify IP addresses that can establish a BGP session with the collector's BGP thread. Many thanks to Erik van der Burg for contributing the idea.
  • 'nfacctd_ext_sampling_rate' and 'sfacctd_ext_sampling_rate' are introduced: they flag the daemon that captured traffic is being sampled. Useful to tackle corner cases, ie. the sampling rate reported by the NetFlow/sFlow agent is missing or incorrect.
  • The 'bgp_follow_nexthop' feature has been extended so that extra IPv4/IPv6 prefixes can be supplied. Up to 32 IP prefixes are now supported and a warning message is generated whenever a supplied string fails parsing.
  • Pre-Tagging: implemented 'src_local_pref' and 'src_comms' keys. These allow tagging based on source IP prefix local_pref (sourced from either a map or BGP, ie. 'bgp_src_local_pref_type: map', 'bgp_src_local_pref_type: bgp') and standard BGP communities.
  • Pre-Tagging: 'src_peer_as' key was extended in order to match on BGP-sourced data (bgp_peer_src_as_type: bgp).
  • Pre-Tagging: introduced 'comms' key to tag basing on up to 16 standard BGP communities attached to the destination IP prefix. The lookup is done against the BGP RIB of the exporting router. Comparisons can be done in either match-any or match-all fashion; xidDocumentation and examples updated. ! fix, util.c: load_allow_file(), empty allow file was granting a connection to everybody being confused with a 'no map' condition. Now this case is properly recognized and correctly translates in a reject all clause. ! fix, sql_common.c: log of NetFlow micro-flows to a SQL database (nfacctd_sql_log directive) was not correctly getting committed to the backend, when sql_history was disabled. ! fix, mysql|pgsql|sqlite_plugin.c: 'flows' aggregation primitive was not suitable to mix-and-match with BGP related primitives (ie. peer_dst_as, etc.) due to an incorrect check. Many thanks to Zenon Mousmoulas for the bug report. ! fix, pretag_handlers.c: tagging against NetFlow v9 4-bytes in/out interfaces was not working properly. Thanks to Zenon Mousmoulas for reporting the issue.