HTTP::RobotsTag::Rules - A Representation Of Robots Exclusion Rules
$rules = $p->parse_rules($html); $bool = $rules->can_index(); $bool = $rules->can_follow(); $bool = $rules->can_archive(); $bool = $rules->can_snippet(); $bool = $rules->is_available( $dt );
HTTP::RobotsTag::Rules represents the robots exclusion policies that are described within HTTP headers.
Returns true if the resource is available on $dt.
To install HTTP::RobotsTag, copy and paste the appropriate command in to your terminal.
cpanm
cpanm HTTP::RobotsTag
CPAN shell
perl -MCPAN -e shell install HTTP::RobotsTag
For more information on module installation, please visit the detailed CPAN module installation guide.