The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Search-Tools-1.007
River stage two • 17 direct dependents • 34 total dependents
7 ++
/ Search::Tools::TokenListPP

NAME

Search::Tools::TokenListPP - a bunch of tokens from a Tokenizer

SYNOPSIS

 use Search::Tools::Tokenizer;
 my $tokenizer = Search::Tools::Tokenizer->new();
 my $tokens = $tokenizer->tokenize_pp('quick brown red dog');
 
 # use like an iterator
 while ( my $token = $tokens->next ) {
    # token isa Search::Tools::Token
 }
 # iterate the other way
 while ( my $token = $tokens->prev ) {
    # ...
 }
 
 # fetch a particular token
 my $token = $tokens->get_token( $position );
 
 # reset the iterator
 $tokens->reset;
 
 # get the current iterator position
 my $pos = $tokens->pos;
 
 # set the iterator position
 $tokens->set_pos( $pos + 1 );
 
 # how many tokens originally?
 my $num = $tokens->num;
 
 # treat like array
 push( @{ $tokens->as_array }, $new_token );
 
 # now how many tokens?
 my $len = $tokens->len;    # $len != $num
 
 # get all the hot tokens
 my $heat = $tokens->get_heat;
 
 # get all the matches to the regex in Tokenizer
 my $matches = $tokens->get_matches;
 
 # just the number of matches
 my $num_matches = $tokens->num_matches;

DESCRIPTION

A TokenListPP is a pure-Perl version of TokenList. See the docs for TokenList for more details.

METHODS

See Search::Tools::TokenListUtils for other methods available on this class.

This class inherits from Search::Tools::Object. Only new or overridden methods are documented here.

next

Get the next Token.

prev

Get the previous Token.

pos

Get the iterator position.

set_pos

Set the iterator position.

reset

Same as calling:

 $tokens->set_pos(0);

len

The number of Tokens in the TokenList.

num

The number of Tokens initially parsed by the Tokenizer. This is the same value as len() unless you alter the TokenList via as_array().

as_array

Returns an array ref to the internal array of tokens. If you alter the array, it will alter the len() value but not the num() value.

get_heat

Returns an array ref to the internal array of tokens with is_hot() set by the original Tokenizer. This method will return an empty list unless you have passed a heat_seeker to the tokenize_pp() method. See Search::Tools::Tokenizer.

matches

Returns an array ref of all the Tokens with is_match() set. The array is constructed at the time you call the method so if you alter the array it will not affect the TokenListPP object, but if you alter a Token in the array it will affect the Token in the TokenListPP object.

num_matches

Like calling:

 my $num = scalar @{ $tokens->matches };

get_token( position )

Returns the Token at position. If position is invalid returns undef.

AUTHOR

Peter Karman <karman@cpan.org>

BUGS

Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

    perldoc Search::Tools

You can also look for information at:

COPYRIGHT

Copyright 2009 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.