NAME

WWW::Extractor - Semi-automated extraction of records from WWW pages

SYNOPSIS

   use strict;
   use WWW::Extractor;

   my($extractor) = WWW::Extractor->new();

   $extractor->process($string);

DESCRIPTION

WWW::Extractor is a tool for semi automated extraction of records from a string containing HTML. One record within the string is marked up with extraction markups and the modules uses a pattern matching algorithm to match up the remaining records.

Extraction markup

The user markups up one record withing the HTML stream with the following symbols.

(((BEGIN)))

Begin a record

(((fieldname)))

Begin a field named fieldname

[[[literal string]]]

This identifies a block of text that the extractor attempts to match. This string is dumped out when the records are extracted.

{{{literal string}}}

This identifies a block of text that the extractor attempts to match. This string is not dumped out when the records are extracted.

(((nodump)))

This marks an area of text that is not to be dumped out.

(((/nodump)))

This ends a section of text that is not to be dumped out.

(((END)))

End a record.

ALGORITHM

The algorithm used is based on the edit distance wrapper generation method described in

@inproceedings{ chidlovskii00automatic, author = "Boris Chidlovskii and Jon Ragetli and Maarten de Rijke", title = "Automatic Wrapper Generation for Web Search Engines", booktitle = "Web-Age Information Management", pages = "399-410", year = "2000", url = "citeseer.nj.nec.com/chidlovskii00automatic.html" }

but with two major enhancements.

1 Before calculating edit distance, the system divides the tokens into different classification groups.

2 Instead of creating a general grammar from all of the records in a file, the data extractor creates one grammar from the sample entry and then matches the rest of the text to that one grammar.

METHODS

$self->new()

Standard constructor

$self->open($string)

Opens a string for processing.

$self->read($hashref)

Read the next processed entry into a hash pointer.

$self->close()

Closes off the string

$self->capture($field, $regexp)

Captures a string that is outside of an entry and inserts it into an entry hash. The routine takes two arguments. One is the field that the entry hash is to be included in. The other is the regular expression which matches the field.

$self->debug(i)

Set the debug level. Higher numbers turn on more debug levels.

$self->expand_hrefs(i)

If set to one then expand out the internal A tags and IMG tags to moving the href outside the tag.

$self->start_tags(i)

Set to the number of tags to match at the beginning of each entry. Default is 2.

$self->end_tags(i)

Number of tags to match at the end of an entry, The default is 1.

If set to one then match table tags exactly. Otherwise ignore internal table items.

EXAMPLES

The distribution contains a sample driver application and test data in the examples directory. To look at the markup for the test data, search for the (((BEGIN))) tag.

Direct mode

To run

./learn.wrapper < ./sample1.html

To run an example with --expand-hrefs

./learn.wrapper --expand-hrefs < ./sample1.html

Saving a grammar

To save off a grammar

./learn.wrapper --extract-grammar < ./sample1.html > /tmp/saved.grammar

You can then use this grammar to do future parsing

./learn.wrapper --load-grammar /tmp/saved.grammar < ./sample1.html

DISCUSSION AND DEVELOPMENT

A wiki on this module is located at

http://www.gnacademy.org/twiki/bin/view/Gna/AutomatedDataExtraction

Please contact gna@gnacademy.org for ideas on improvements.

COPYRIGHT AND LICENSE

Copyright (C) 2002-2004 Globewide Network Academy

Redistributed under the terms of the Lesser GNU Public License

cpanm WWW::Extractor

perl -MCPAN -e shell
install WWW::Extractor