Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer
# isa Plucene::Analysis::CharTokenizer
This is the standard tokenizer.
This should be a good tokenizer for most European-language documents.
The regular expression for tokenising.
Remove 's and .
To install Plucene, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Plucene
CPAN shell
perl -MCPAN -e shell install Plucene
For more information on module installation, please visit the detailed CPAN module installation guide.