String::Tokeniser - Perl extension for, uhm, tokenising strings.
use String::Tokeniser;
String::Tokeniser provides an interface to a tokeniser class, allowing one to manipulate strings on a token-by-token basis without having to keep track of list element numbers and so on.
String::Tokeniser
Create a String::Tokeniser, tokenises $sentence and resets the token counter.
The next argument determines how a ``token'' is defined: a value of 0 or undef determines that underscores are included in a token; -1 states that they are not. Alternatively, you can supply your own regular expression which will be fed to a split to determine the tokens.
undef
split
Then may optionally follow a list of exceptions: tokens that would be split in two, but should be treated as one.
Tells you if you have any more tokens left to deal with.
Move the `pointer' forward one (or n) tokens.
n
Return the current token; that is, the token under the `pointer'.
Return the previous token; that is, the one just past the `pointer'.
Equivalent to skiptoken;gettoken - the usual way of grabbing the next token in the list in turn.
skiptoken;gettoken
Looks ahead one token, but does not change the `pointer' position.
Returns a string composed of the next n tokens, but does not change the `pointer' position.
Assuming a string of tokens will end in $string, returns everything from the current `pointer' position until the string is found. Returns a two-element list: firsly, why the search terminated, (either EOF meaning we hit the end of the token list without success, or FOUND meaning $string was found.) and the rest of the tokens upto and including $string (or the end of the list, whichever was soonest).
$string
EOF
FOUND
Saves one's pointer position. Can be used multiply as a save stack.
Restores a previously saved position.
At present, there is no support for exceptions which spread over three or more tokens, although this is planned.
Originaly written by Simon Cozens; Maintained by Alberto Simoes <ambs@cpan.org>
<ambs@cpan.org
WEBPerl::Changetie
To install String::Tokeniser, copy and paste the appropriate command in to your terminal.
cpanm
cpanm String::Tokeniser
CPAN shell
perl -MCPAN -e shell install String::Tokeniser
For more information on module installation, please visit the detailed CPAN module installation guide.