Devel::Tokenizer::C 0.08

Devel::Tokenizer::C is a Perl module that can generate C source for fast keyword tokenizer.

SYNOPSIS

use Devel::Tokenizer::C;

$t = new Devel::Tokenizer::C TokenFunc => sub { "return U$_[0];n" };

$t->add_tokens(qw( bar baz ))->add_tokens(['for']);
$t->add_tokens([qw( foo )], 'defined DIRECTIVE');

print $t->generate;

The Devel::Tokenizer::C module provides a small class for creating the essential ANSI C source code for a fast keyword tokenizer.

The generated code is optimized for speed. On the ANSI-C keyword set, it's 2-3 times faster than equivalent code generated with the gprof utility.

The above example would print the following C source code:
switch (tokstr[0])
{
case 'b':
switch (tokstr[1])
{
case 'a':
switch (tokstr[2])
{
case 'r':
if (tokstr[3] == '�')
{ /* bar */
return BAR;
}

goto unknown;

case 'z':
if (tokstr[3] == '�')
{ /* baz */
return BAZ;
}

goto unknown;

default:
goto unknown;
}

default:
goto unknown;
}

case 'f':
switch (tokstr[1])
{
case 'o':
switch (tokstr[2])
{
#if defined DIRECTIVE
case 'o':
if (tokstr[3] == '�')
{ /* foo */
return FOO;
}

goto unknown;
#endif /* defined DIRECTIVE */

case 'r':
if (tokstr[3] == '�')
{ /* for */
return FOR;
}

goto unknown;

default:
goto unknown;
}

default:
goto unknown;
}

default:
goto unknown;
}

So the generated code only includes the main switch statement for the tokenizer. You can configure most of the generated code to fit for your application.

Product's homepage

Requirements:

· Perl