org.apache.nutch.protocol

## Interface Protocol

- All Superinterfaces:
    Configurable, Pluggable
  All Known Implementing Classes:
    File, Ftp, Http, Http, Http, Http, Http, HttpBase, OkHttp

      public interface Protocol
      extends Pluggable, Configurable

  A retriever of url content. Implemented by protocol extensions.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `X_POINT_ID` The name of the extension point.

  - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method and Description
    --- | ---
    `ProtocolOutput` | `getProtocolOutput(Text url, CrawlDatum datum)` Returns the `Content` for a fetchlist entry.
    `crawlercommons.robots.BaseRobotRules` | `getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)` Retrieve robot rules applicable for this URL.

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Field Detail

    - #### X_POINT_ID

          static final String X_POINT_ID

      The name of the extension point.

  - ### Method Detail

    - #### getProtocolOutput

          ProtocolOutput getProtocolOutput(Text url,
                                           CrawlDatum datum)

      Returns the `Content` for a fetchlist entry.

    - #### getRobotRules

          crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                             CrawlDatum datum,
                                                             List<Content> robotsTxtContent)

      Retrieve robot rules applicable for this URL.
      Parameters:
        `url` - URL to check
        `datum` - page datum
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robot rules (specific for this URL or default), never null

Copyright © 2021 The Apache Software Foundation