NAME

MMM::Text::Search - Perl module for indexing and searching text files and web objects

SYNOPSIS

  use MMM::Text::Search;
          
  my $srch = new MMM::Text::Search {    #for indexing...
        #index main file location...  
                IndexPath => "/tmp/myindex.db",
        #local files... (optional)
                FileMask  => '(?i)(\.txt|\.htm.?)$',
                Dirs      => [ "/usr/doc", "/tmp" ] ,
                FollowSymLinks => 0|1, (default = 0)
        #web objects... (optional)
                URLs      => [ "http://localhost/", ... ],
                Level     => recursion-level (0=unlimited)              
        #common options...              
                IgnoreLimit =>  0.3,   (default = 2/3)
                Verbose => 0|1                          
        };
  
  $srch->start_indexing_session();
        
  $srch->commit_indexing_session();
  
  $srch->index_default_locations();
        
  $srch->index_content( { title =>   '...', 
                          content=>  '...', 
                          id =>      '...'  } );
         
  $srch->makeindex;
       (Obsolete.) 


        
        

  my $srch = new MMM::Text::Search (  #for searching....
                  "/tmp/myindex.db", verbose_flag );
  
  my $hashref = $srch->query("pizza","ciao", "-pasta" );  
  my $hashref = $srch->advanced_query("(pizza OR ciao) AND NOT pasta");  

  $srch->errstr()       # returns last error 
                        # (only query syntax-errors for the moment being)

  
  $srch->dump_word_stats(\*FH)  

DESCRIPTION

• Indexing

  When a session is closed the following files will have been created (assuming IndexPath = /path/myindex.db, see constructor):

          /path/myindex.db             word index database
          /path/myindex-locations.db   filename/URL database
          /path/myindex-titles.db      html title database
          /path/myindex.stopwords      stop-words list
          /path/myindex.filelist       readable list of indexed files/URLs
          /path/myindex.deadlinks      broken http links
  
          [... lots of important things missing ... ]

  start_indexing_session() starts session.

  commit_indexing_session() commits and closes current session.

  index_default_locations() indexes all files and URLs specified on construction.

  index_content() pushes content into indexing engine. Argument must have the following structure

   { title =>   '...', content=>  '...', id =>      '...'  }

  makeindex() is obsolete. Equivalent to: $srch->start_indexing_session(); $srch->index_default_locations(); $srch->commit_indexing_session();

  dump_word_stats(\*FH) dumps all words sorted by occurence frequency using FH file handle (or STDOUT if no parameter is specified). Stop-words get a frequency value of 1.

• Searching

  Both query() and advanced_query() return a reference to a hash with the following structure:

          (
           ignored  => [ string, string, ... ], # ignored words
           searched => [ string, string, ... ], # words searched for
           entries    => [  hashref, hashref, ... ] # list of records 
                                                  # found
           )
          

  The 'entries' element is a reference to an array of hashes, each having the following structure:

          (
           location => string,  # file path or URL or anything
           score    => number,  # score 
           title    => string   # HTML title               
          )

NOTES

Note on implementation: The technique used for indexing is substantially derived from that exposed by Tim Kientzle on Dr. Dobbs magazine.

BUGS

Many, I guess.

AUTHOR

Max Muzi <maxim@comm2000.it>

SEE ALSO

perl(1).