MarpaX::Repa::Lexer - simplify lexing for Marpa parser
Most details are in MarpaX::Repa.
Returns a new lexer instance. Takes named arguments.
my $lexer = MyLexer->new( tokens => { word => qr{\b\w+\b}, }, store => 'array', recognizer => $recognizer, debug => 1, );
Possible arguments:
Hash with names of terminals as keys and one of the following as values:
Just a string to match.
'a token' => "matches this long string",
A qr{} compiled regexp.
qr{}
'a token' => qr{"[^"]+"},
Note that regexp MUST match at least one character. At this moment look behind to look at chars before the current position is not supported.
With hash you can define token specific options. At this moment 'store' option only (see below). Use match key to set what to match (string or regular expression).
match
'a token' => { match => "a string", store => 'hash', },
What to store (pass to Marpa's recognizer). The following variants are supported:
{ token => 'a token', value => 'a value' }
[ 'a token', 'a value' ]
'a value'
undef is stored so later Repa's actions will skip it.
A function will be called with token name and reference to its value. Should return a reference or undef that will be passed to recognizer.
Marpa::R2::Recognizer object or its subclass.
If true then lexer prints debug log to STDERR.
Minimal size of the buffer (4*1024 by default).
Setups instance and returns $self. Called from constructor.
$self
Takes a file handle and parses it. Dies on critical errors, not when parser lost its way. Returns recognizer that was passed to "new".
Returns reference to the current buffer.
Called when "buffer" needs a re-fill with a file handle as argument. Returns true if there is still data to come from the handle.
Returns first 20 chars of the buffer with everything besides ASCII encoded with \x{####}. Use argument to control size, zero to mean whole buffer.
\x{####}
To install MarpaX::Repa, copy and paste the appropriate command in to your terminal.
cpanm
cpanm MarpaX::Repa
CPAN shell
perl -MCPAN -e shell install MarpaX::Repa
For more information on module installation, please visit the detailed CPAN module installation guide.