Data::Consumer 0.13

Data::Consumer - repeatedly consume a data resource in a robust way.

SYNOPSIS

use Data::Consumer;

my $consumer = Data::Consumer->new(
type => $consumer_name,
unprocessed => $unprocessed,
working => $working,
processed => $processed,
failed => $failed,
max_passes => $num_or_undef,
max_process => $num_or_undef,
max_elapsed => $seconds_or_undef,
);

$consumer->consume( sub {
my $id = shift;
print "processed $idn";
} );

It is a common requirement to need to process a feed of items of some sort in a robust manner. Such a feed might be records that are inserted into a table, or files dropped in a delivery directory. Writing a script that handles all the edge cases, like getting "stuck" on a failed item, and manages things like locking so that the script can be parallelized can be tricky and is certainly repetitive.

The aim of Data::Consumer is to provide a framework to allow writing such consumer type scripts as easy as writing a callback that processes each item. The framework handles the rest.

The basic idea is that one need only use, or in the case of a feed type not already supported, define a Data::Consumer subclass which implements a few reasonably well defined primitive methods which handle the required tasks, and then the Data::Consumer methods use those to provide a DWIMily consistant interface to the end consumer.

Currently Data::Consumer is distributed with two subclasses, Data::Consumer::MySQL for handling records in a MySQL db (using the MySQL GET_LOCK() function), and Data::Consumer::Dir for handling a drop directory scenario (like for FTP or a mail directory).

Once a resource type has been defined as a Data::Consumer subclass the use pattern is to construct the subclass with the appropriate arguments, and then call consume with a callback.

Product's homepage

Requirements:

· Perl