Basic Crawler Plugin
The Basic Crawler Plugin implements a CLI Tool extending Rover to add site crawling capabilities.
The tool can be used to extract semantic content from a small/medium size sites.
To use it make sure to have correctly configured the basic-crawler plugin to be found by the any23tools script (follow the Plugins section instructions):
core/bin/$ ./any23tools Crawler
usage: [{<url>|<file>}]+ [-d <arg>] [-e <arg>] [-f <arg>] [-h] [-l <arg>]
[-maxdepth <arg>] [-maxpages <arg>] [-n] [-numcrawlers <arg>] [-o
<arg>] [-p] [-pagefilter <arg>] [-politenessdelay <arg>] [-s]
[-storagefolder <arg>] [-t] [-v]
-d,--defaultns <arg> Override the default namespace used to produce
statements.
-e <arg> Specify a comma-separated list of extractors,
e.g. rdf-xml,rdf-turtle.
-f,--Output format <arg> [turtle (default), rdfxml, ntriples, nquads,
trix, json, uri]
-h,--help Print this help.
-l,--log <arg> Produce log within a file.
-maxdepth <arg> Max allowed crawler depth. Default: no limit.
-maxpages <arg> Max number of pages before interrupting crawl.
Default: no limit.
-n,--nesting Disable production of nesting triples.
-numcrawlers <arg> Sets the number of crawlers. Default: 10
-o,--output <arg> Specify Output file (defaults to standard
output).
-p,--pedantic Validate and fixes HTML content detecting
commons issues.
-pagefilter <arg> Regex used to filter out page URLs during
crawling. Default:
'.*(\.(css|js|bmp|gif|jpe?g|png|tiff?|mid|mp2|
mp3|mp4|wav|wma|avi|mov|mpeg|ram|m4v|wmv|rm|sm
il|pdf|swf|zip|rar|gz|xml|txt))$'
-politenessdelay <arg> Politeness delay in milliseconds. Default: no
limit.
-s,--stats Print out extraction statistics.
-storagefolder <arg> Folder used to store crawler temporary data.
Default:
[/var/folders/d5/c_0b4h1d7t1gx6tzz_dn5cj40000g
q/T/]
-t,--notrivial Filter trivial statements (e.g. CSS related
ones).
-v,--verbose Show debug and progress information.