Plucene::Analysis::CharTokenizer - base class for character tokenisers
# isa Plucene::Analysis::Tokenizer my $next = $chartokenizer->next;
This is an abstract base class for simple, character-oriented tokenizers.
This should be defined in subclasses.
This will normalise the character before it is added to the token.
my $next = $chartokenizer->next;
This will return the next token in the string, or undef at the end of the string.
To install Plucene, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Plucene
CPAN shell
perl -MCPAN -e shell install Plucene
For more information on module installation, please visit the detailed CPAN module installation guide.