The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
CPAN-Search-Lite-0.78
River stage zero No dependents
++
/ CPAN::Search::Lite::Index

NAME

CPAN::Search::Lite::Index - set up or update database tables.

SYNOPSIS

 my $index = CPAN::Search::Lite::Index->new(config => 'cpan.conf', setup => 1);
 $index->index();

DESCRIPTION

This is the main module used to set up or update the database tables used to store information from the CPAN and ppm indices. The creation of the object

 my $index = CPAN::Search::Lite::Index->new(%args);

accepts three arguments:

  • config => /path/to/config.conf

    This argument specifies where to find the configuration file used to determine the remaining options. In lieu of this option, the environment variable CSL_CONFIG_FILE pointing to the configuration file may be specified.

  • setup => 1

    This (optional) argument specifies that the database is being set up. Any existing tables will be dropped.

  • reindex => value

    This (optional) argument specifies distribution names that one would like to reindex in an existing database. These may be specified as either a scalar, for a single distribution, or as an array reference for a list of distributions.

CONFIGURATION

Most of the options used to control the behaviour of the indexing are contained in a configuration file. An example of the format of such a file is

 [CPAN]

 CPAN = /var/ftp/pub/CPAN
 pod_root = /usr/local/POD
 html_root = /usr/local/httpd/htdocs/CPAN

 [DB]

 db = pause
 user = sarah
 passwd = lianne

 [WWW]

 css = cpan.css
 up_img = up.gif
 tt2 = /usr/local/tt2
 geoip = /usr/local/share/geoip/cpan.txt

This consists of 3 sections.

CPAN

This is associated with various things related to CPAN.

  • CPAN = /var/ftp/pub/CPAN

    This specifies the root directory of the local CPAN mirror, if this exists, or the location where the CPAN index files will be downloaded and kept, if the no_mirror option is specified.

  • pod_root = /usr/local/POD

    This specifies where the extracted pod files from a distribution will be kept. A subdirectory dist_name under this directory will be created corresponding to the name of the distribution.

  • pod_only = 1

    This specifies that, if the module files are to be extracted, fetch only those that contain pod.

  • split_pod = 1

    This specifies that, if the module files are to be extracted, when generating the html pages create two pages for each module: one containing just the documentation, and the other containing the code run through Perl::Tidy. For a module such as Foo::Bar, the documentation will be saved as a file Foo/Bar.html, while the sources will be saved as Foo/Bar.pm.html.

  • html_root = /usr/local/httpd/htdocs/CPAN

    This specifies where the html files created from the pod files will be kept. A subdirectory dist_name under this directory will be created corresponding to the name of the distribution.

  • ignore = some_dist_name_to_ignore

    This specifies a name of a distribution (without a version number) to ignore in indexing. This option may be given a number of times to specify an array of values, or may be specified as

      ignore = <<EOL
  Module-CPANTS-asHash
  CORBA-IDL
  EOL

    This array of values (which may include regular expressions) is joined together as

      $pat = join '|', @ignore_dists

    and if the distribution name matches

      $dist_name =~ /^($pat)$/

    the distribution is ignored.

  • no_mirror = 1

    This specifies that a local CPAN mirror isn't available, and as such no pod or html files will be extracted nor created.

  • no_ppm = 1

    This can be used to signal to not gather information on Win32 ppm packages from the repositories specified in $repositories of CPAN::Search::Lite::Util.

  • remote_mirror = http://cpan.wherever.edu

    If no_mirror is specified, the value of remote_mirror will be used to fetch the CPAN indices. If not given, http://www.cpan.org will be used.

  • multiplexer = http://cpan.redirect.edu/cpan

    This can be used to specify a multiplexer to redirect downloads to nearby CPAN mirrors. See, for example, Apache::GeoIP for one implementation.

  • cat_threshold = 0.99

    Many modules do not have a category (chapter) associated with them. In such cases, when populating the database, the AI::Catgorizer module is used to guess which category should be assigned to such a module, based on available information for those modules that do have a category. The value of cat_threshold is used to determine if the guessed category should be accepted (a perfect match has a score of 1, and no match has 0). If no such value is given, a default of 0.995 is used.

  • no_cat = 1

    Set no_cat equal to a true value if you don't want AI::Categorizer to try categorizing modules which don't have a category assigned.

DB

This is used to store connection information to the database used to populate the tables.

  • db = pause

    This is the name of the database used. It is assumed here that the database has already been created, and that appropriate read, write, update, create, and delete permissions for the user specified below have already been granted.

  • user = sarah

    This is the user under which to connect to the database.

  • passwd = lianne

    This is the password to use for the user.

WWW

This is used for various information related to a web interface.

  • css = cpan.css

    If specified, this will be used as the css file when generating the html files from the pod files of the distribution. It is assumed this file appears directly beneath the html_root of the CPAN section.

  • up_img = up.gif

    If specified, this will be used as an image in the generated html files linking each section to the top-most index. If not specifed, the text __top will be used. It is assumed this image appears directly beneath the html_root of the CPAN section.

  • dist_info = http://cpan.uwinnipeg.ca/dist/

    If specified, this will be used to provide a link on the generated html pages to information on the distribution. The name of the distribution will be added at the end of the link (for example, using http://cpan.uwinnipeg.ca/dist/ will result, for the libnet distribution, in a link to http://cpan.uwinnipeg.ca/dist/libnet.

  • tt2 = /usr/local/tt2

    This gives the location of the Template-Toolkit pages used to provide a web interface. This is used to place a file mirror_list (extracted from $CPAN/MIRRORED.BY) containing a list of CPAN mirrors.

  • geoip = /usr/local/share/geoip/cpan.txt

    If the module Geo::IP or Apache::GeoIP is used to provide a redirection service to a nearby CPAN mirror based on the location of origin, this file will be created to provide the necessary country of origins of the CPAN mirrors.

DETAILS

Calling

  $index->index();

will start the indexing procedure. Various messages detailing the progress will written to STDOUT, which by default will be captured into a file cpan_search_log.dddddddddd, where the extension is the time that the method was invoked. Passing index an argument of log => log_file will save these messages into log_file. Error messages are not captured, and will appear in STDERR.

The steps of the indexing procedure are as follows.

  • fetch index data

    If the no_mirror option is specified, the necessary CPAN index files $CPAN/MIRRORED.BY, $CPAN/indices/ls-lR.gz, $CPAN/authors/01mailrc.txt.gz, $CPAN/modules/02packages.details.txt.gz, and $CPAN/modules/03modlist.data.gz will be fetched from the CPAN mirror specified by the $cpan variable at the beginning of CPAN::Search::Lite::Index. If you are using this option, it is recommended to use the same CPAN mirror with subsequent updates, to ensure consistency of the database. As well, the information on the locations of the CPAN mirrors used for Template-Toolkit and GeoIP is written.

  • get index information

    Information from the CPAN indices and, if desired, the ppm repositories is extracted. This is done through CPAN::Search::Lite::Info (for the CPAN indices) and CPAN::Search::Lite::PPM (for the ppm repositories).

  • get state information

    Unless the setup argument within the new method of CPAN::Search::Lite::Index is specified, this will get information on the state of the database through CPAN::Search::Lite::State. A comparision is then made between this information and that gathered from the CPAN indices, and if there's a discrepency in some items, those items are marked for either insertion, updating, or deletion, as appropriate.

  • extract files

    Unless the no_mirror option is specified, this will extract, through CPAN::Search::Lite::Extract, the available pod sections of files of a distribution, placing them under a subdirectory dist_name (corrsponding to the name of the distribution) under the specified pod_root in the configuration file. pod2html is then run on them, with the results placed under dist_name of html_root. Also, a README, Changes, INSTALL, and META.yml file, if present, will be copied over into dist_name under pod_root. Finally, information on the prerequisites of the distribution, and distribution and module descriptions, if available and needed, is extracted.

  • populate the database

    At this stage the gathered information is used to populate the database, through CPAN::Search::Lite::Populate, either inserting new items, updating existing ones, or deleting obsolete items.

    As well, unless the no_mirror option is specified, the html files created under html_root will be edited to adjust the links to module files. This is necessary because when the html files are created no cache is used (in order to maintain consistency between updates), and consequently links to packages outside of a given package may be incorrect. This is fixed by querying the database to see what module documentation is actually present, and then adjusting the links in the html files accordingly (or removing a link if the indicated documentation is missing).

SEE ALSO

CPAN::Search::Lite::Info, CPAN::Search::Lite::PPM, CPAN::Search::Lite::State, CPAN::Search::Lite::Extract, CPAN::Search::Lite::Populate, and CPAN::Search::Lite::Util. Development takes place on the CPAN-Search-Lite project at http://sourceforge.net/projects/cpan-search/.

COPYRIGHT

This software is copyright 2004 by Randy Kobes <randy@theoryx5.uwinnipeg.ca>. Use and redistribution are under the same terms as Perl itself.