## Getting started with Apache Any23

Apache Any23 can be used:

- via CLI (command line interface) from your preferred shell environment;
- as a RESTful Webservice;
- as a library.

### Apache Any23 Modules

Apache Any23 is composed of the following modules:

- `api/` The base API definitions e.g. The Any23 API.
- `core/` The core library containing all extractor functionality.
- `cli/` A command line interface enabling easy invocation of Any23 tools.
- `csvutils/` Utility code for CSV extractions.
- `encoding/` Characterset detection and encoding.
- `mime/` Media-type detection.
- `service/` The REST service.
- `plugins/` The core additional plugins.
- `openie/` Additional extractor logic for the Open Information Extraction (Open IE) system.

### Use the Apache Any23 CLI

The command-line tools support is provided by the cli module.

Once Apache Any23 has been correctly installed, if you want to use it as a command line tool, use the shell script within the `cli/target/appassembler/bin/` directory. These are provided both for Unix (Linux/OSX) and Windows.

The `any23` script provides analysis, documentation, testing and debugging utilities.

Simply running ./any23 without options will show the usage options.

    $ cli/target/appassembler/bin/any23

    A command must be specified.
    Usage: any23 [options] [command] [command options]
      Options:
        -h, --help
           Display help information.
           Default: false
            --plugins-dir
           The Any23 plugins directory.
           Default: /Users/lmcgibbn/.any23/plugins
        -X, --verbose
           Produce execution verbose output.
           Default: false
        -v, --version
           Display version information.
           Default: false
      Commands:
        extractor      Utility for obtaining documentation about metadata extractors.
          Usage: extractor [options] Extractor name
            Options:
              -a, --all
                 shows a report about all available extractors
                 Default: false
              -i, --input
                 shows example input for the given extractor
                 Default: false
              -l, --list
                 shows the names of all available extractors
                 Default: false
              -o, --outut
                 shows example output for the given extractor
                 Default: false

        microdata      Commandline Tool for extracting Microdata from file/HTTP source.
          Usage: microdata [options] Input document URL, {http://path/to/resource.html|file:/path/to/localFile.html}

        mimes      MIME Type Detector Tool.
          Usage: mimes [options] Input document URL, {http://path/to/resource.html|file:///path/to/local.file|inline:// some inline content}

        verify      Utility for plugin management verification.
          Usage: verify [options] plugins-dir

        rover      Any23 Command Line Tool.
          Usage: rover [options] input IRIs {<url>|<file>}+
            Options:
              -d, --defaultns
                 Override the default namespace used to produce statements.
              -e, --extractors
                 a comma-separated list of extractors, e.g. rdf-xml,rdf-turtle
                 Default: []
              -f, --format
                 the output format
                 Default: json
              -l, --log
                 Produce log within a file.
              -n, --nesting
                 Disable production of nesting triples.
                 Default: false
              -t, --notrivial
                 Filter trivial statements (e.g. CSS related ones).
                 Default: false
              -o, --output
                 Specify Output file (defaults to standard output)
                 Default: java.io.PrintStream@5204062d
              -p, --pedantic
                 Validate and fixes HTML content detecting commons issues.
                 Default: false
              -s, --stats
                 Print out extraction statistics.
                 Default: false

        vocab      Prints out the RDF Schema of the vocabularies used by Any23.
          Usage: vocab [options]
            Options:
              -f, --format
                 Vocabulary output format
                 Default: N-Quads (mimeTypes=application/n-quads, text/x-nquads, text/nquads; ext=nq)

The `any23` script detects a list of available utilities within the core and plugins classpath and allows to activate them.

The any23-core CLI tools are:

- `extractor`: a utility for obtaining useful information about extractors.
- `microdata`: commandline parser to extract specific Microdata content from a web page (local or remote) and produce a JSON output compliant with the Microdata specification (http://www.w3.org/TR/microdata/).
- `mimes`: detects the MIME Type for any HTTP / file / direct input resource.
- `verify`: a utility for verifying Apache Any23 plugins.
- `rover`: the RDF extraction tool.
- `vocab`: allows to dump all the RDFSchema vocabularies declared within Apache Any23.

#### The Rover tool

Rover is the main extraction tool. It allows to extract metadata from local and remote (HTTP) resources, specify a custom list of extractors, specify the desired output format and other flags to suppress noise and generate advanced reports.

Extract metadata from an HTML page:

    cli$ any23 rover http://yourdomain/yourfile

Extract metadata from a local resource:

    cli$ any23 rover myfoaf.rdf

Specify the output format, use the option "-f" or "--format": (Default output format is TURTLE).

    cli$ any23 rover -f quad myfoaf.rdf

Filtering trivial statements

By default, Apache Any23 will extract HTML/head meta information, such as links to CSS stylesheets or meta information like the author or the software used to create the html. Hence, if the user is only interested in the structured content from the HTML/body tag we offer a filter functionality, activated by the "-t" command line argument.

    core$ any23 rover -t -f quad myfoaf.rdf

#### The ExtractorDocumentation tool

The ExtractorDocumentation returns human readable information about the registered extractors.

List all the available extractors:

    cli$ any23 extractor --list
                          csv [org.apache.any23.extractor.csv.CSVExtractorFactory] [text/csv;q=0.1]
         html-embedded-jsonld [org.apache.any23.extractor.html.EmbeddedJSONLDExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
               html-head-icbm [org.apache.any23.extractor.html.ICBMExtractorFactory] [text/html;q=0.01, application/xhtml+xml;q=0.01]
              html-head-links [org.apache.any23.extractor.html.HeadLinkExtractorFactory] [text/html;q=0.05, application/xhtml+xml;q=0.05]
               html-head-meta [org.apache.any23.extractor.html.HTMLMetaExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
              html-head-title [org.apache.any23.extractor.html.TitleExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
                  html-mf-adr [org.apache.any23.extractor.html.AdrExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
                  html-mf-geo [org.apache.any23.extractor.html.GeoExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
            html-mf-hcalendar [org.apache.any23.extractor.html.HCalendarExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
                html-mf-hcard [org.apache.any23.extractor.html.HCardExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
             html-mf-hlisting [org.apache.any23.extractor.html.HListingExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
              html-mf-hrecipe [org.apache.any23.extractor.html.HRecipeExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
              html-mf-hresume [org.apache.any23.extractor.html.HResumeExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
              html-mf-hreview [org.apache.any23.extractor.html.HReviewExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
    html-mf-hreview-aggregate [org.apache.any23.extractor.html.HReviewAggregateExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
              html-mf-license [org.apache.any23.extractor.html.LicenseExtractorFactory] [text/html;q=0.01, application/xhtml+xml;q=0.01]
              html-mf-species [org.apache.any23.extractor.html.SpeciesExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
                  html-mf-xfn [org.apache.any23.extractor.html.XFNExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
               html-microdata [org.apache.any23.extractor.microdata.MicrodataExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
                  html-rdfa11 [org.apache.any23.extractor.rdfa.RDFa11ExtractorFactory] [application/xhtml+xml;q=0.3, application/html;q=0.3, text/html;q=0.3]
                   html-xpath [org.apache.any23.extractor.xpath.XPathExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
                         ical [org.apache.any23.extractor.calendar.ICalExtractorFactory] [text/calendar]
                         jcal [org.apache.any23.extractor.calendar.JCalExtractorFactory] [application/calendar+json]
               owl-functional [org.apache.any23.extractor.rdf.FunctionalSyntaxExtractorFactory] [text/owl-functional]
               owl-manchester [org.apache.any23.extractor.rdf.ManchesterSyntaxExtractorFactory] [text/owl-manchester]
                   rdf-jsonld [org.apache.any23.extractor.rdf.JSONLDExtractorFactory] [application/ld+json;q=0.1]
                       rdf-nq [org.apache.any23.extractor.rdf.NQuadsExtractorFactory] [application/n-quads, text/x-nquads;q=0.1, text/rdf+nq;q=0.1, text/nq;q=0.1, text/nquads;q=0.1, text/n-quads;q=0.1]
                       rdf-nt [org.apache.any23.extractor.rdf.NTriplesExtractorFactory] [application/n-triples;q=0.1, text/nt;q=0.1, text/ntriples;q=0.1, text/plain;q=0.1]
                     rdf-trix [org.apache.any23.extractor.rdf.TriXExtractorFactory] [application/trix]
                   rdf-turtle [org.apache.any23.extractor.rdf.TurtleExtractorFactory] [text/turtle, text/rdf+n3, text/n3, application/n3, application/x-turtle, application/turtle]
                      rdf-xml [org.apache.any23.extractor.rdf.RDFXMLExtractorFactory] [application/rdf+xml, text/rdf, text/rdf+xml, application/rdf]
                         xcal [org.apache.any23.extractor.calendar.XCalExtractorFactory] [application/calendar+xml]
                         yaml [org.apache.any23.extractor.yaml.YAMLExtractorFactory] [text/x-yaml;q=0.5]

#### The MicrodataParser tool

The MicrodataParser tool allows to apply the only MicrodataExtractor on a specific input source and returns the extracted data in the JSON format declared in the Microdata specification section JSON.

    cli$ any23 microdata http://path/to/resource.html

#### The VocabPrinter tool

The VocabPrinter Tool prints out the RDFSchema declared by all the Apache Any23 declared vocabularies.

Just launch the command below to see all the managed vocabularies.

    cli$ any23 vocab

NOTE: This tool is still in beta version.

#### The MimeDetector tool

The MimeDetector Tool extracts the MIME Type for a given source (http:// file:// inline://).

Examples:

    cli$ any23 mimes http://www.michelemostarda.com/foaf.rdf
    application/rdf+xml

    cli$ any23 mimes file://../src/test/resources/application/trix/test1.trx
    application/trix

    cli$ any23 mimes 'inline://<http://s> <http://p> <http://o> .'
    text/n3

#### The PluginVerifier tool

The PluginVerifier tool allows checking installed plugin in the specified input directory

Just launch the command below to sanity-check the input plugins directory

    cli$ any23 verify [/path/to/plugins/dir]

### Apache Any23 CLI Plugins

The Apache Any23 ToolRunner CLI (bin/any23) supports the auto detection of Tool plugins within the classpath. For further details see Plugins section.

The default any23 CLI plugins are enlisted below.

#### Crawler Plugin

crawler-tool The Crawler Plugin provides basic site crawling and metadata extraction capabilities.

    cli$ any23 -h
    [...]
        crawler      Any23 Crawler Command Line Tool.
          Usage: crawler [options] input IRIs {<url>|<file>}+
      Options:
              -d, --defaultns          Override the default namespace used to
                                       produce statements.
              -e, --extractors         a comma-separated list of extractors, e.g.
                                       rdf-xml,rdf-turtle
                                       Default: []
              -f, --format             the output format
                                       Default: turtle
              -l, --log                Produce log within a file.
              -md, --maxdepth          Max allowed crawler depth.
                                       Default: 2147483647
              -mp, --maxpages          Max number of pages before interrupting
                                       crawl.
                                       Default: 2147483647
              -n, --nesting            Disable production of nesting triples.
                                       Default: false
              -t, --notrivial          Filter trivial statements (e.g. CSS related
                                       ones).
                                       Default: false
              -nc, --numcrawlers       Sets the number of crawlers.
                                       Default: 10
              -o, --output             Specify Output file (defaults to standard
                                       output)
                                       Default: java.io.PrintStream@2911a3a4
              -pf, --pagefilter        Regex used to filter out page URLs during
                                       crawling.
                                       Default: .*(\.(css|js|bmp|gif|jpe?g|png|tiff?|mid|mp2|mp3|mp4|wav|wma|avi|mov|mpeg|ram|m4v|wmv|rm|smil|pdf|swf|zip|rar|gz|xml|txt))$
              -p, --pedantic           Validate and fixes HTML content detecting
                                       commons issues.
                                       Default: false
              -pd, --politenessdelay   Politeness delay in milliseconds.
                                       Default: 2147483647
              -s, --stats              Print out extraction statistics.
                                       Default: false
              -sf, --storagefolder     Folder used to store crawler temporary data.
                                       Default: /var/folders/zz/9vvv_lbn1cs8dpwz859nmq080000gn/T/crawler-metadata-9ff4c650-10c2-41a1-9d99-ebeb3e7d21ce

A usage example:

    cli$ any23 crawler -s -f ntriples http://www.repubblica.it 1> out.nt 2> repubblica.log

### Use Apache Any23 as a RESTful Web Service

Apache Any23 provides a Web Service that can be used to extract RDF from Web documents. Apache Any23 services can be accessed through a RESTful API.

Running the server

The server command line tool is defined within the service module. Run the `any23server` script

    service$ ./bin/any23server

from the command line in order to start up the server, then go to to access the web interface. A live demo version of such service is running at . You can also start the server from Java by running the Apache Any23 Servlet class. Maven can be used to create a WAR file for deployment into an existing servlet container such as Apache Tomcat.

### Use Apache Any23 as a Library

See our Developers guide for more details.