org.apache.nutch.protocol

## Class RobotRulesParser

- java.lang.Object
- - org.apache.nutch.protocol.RobotRulesParser

- All Implemented Interfaces:
    Configurable, Tool
  Direct Known Subclasses:
    FtpRobotRulesParser, HttpRobotRulesParser

      public abstract class RobotRulesParser
      extends Object
      implements Tool

  This class uses crawler-commons for handling the parsing of `robots.txt` files. It emits SimpleRobotRules objects, which describe the download permissions as described in SimpleRobotRulesParser. Protocol-specific implementations have to implement the method `getRobotRulesSet`.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected String` | `agentNames`
    `protected static Hashtable<String,crawlercommons.robots.BaseRobotRules>` | `CACHE`
    `protected Configuration` | `conf`
    `static crawlercommons.robots.BaseRobotRules` | `EMPTY_RULES` A `BaseRobotRules` object appropriate for use when the `robots.txt` file is empty or missing; all requests are allowed.
    `static crawlercommons.robots.BaseRobotRules` | `FORBID_ALL_RULES` A `BaseRobotRules` object appropriate for use when the `robots.txt` file is not fetched due to a `403/Forbidden` response; all requests are disallowed.
    `protected Set<String>` | `whiteList` set of host names or IPs to be explicitly excluded from robots.txt checking

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `RobotRulesParser()`
    `RobotRulesParser(Configuration conf)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Configuration` | `getConf()` Get the `Configuration` object
    `crawlercommons.robots.BaseRobotRules` | `getRobotRulesSet(Protocol protocol, Text url, List<Content> robotsTxtContent)` Fetch robots.txt (or it's protocol-specific equivalent) which applies to the given URL, parse it and return the set of robot rules applicable for the configured agent name(s).
    `abstract crawlercommons.robots.BaseRobotRules` | `getRobotRulesSet(Protocol protocol, URL url, List<Content> robotsTxtContent)` Fetch robots.txt (or it's protocol-specific equivalent) which applies to the given URL, parse it and return the set of robot rules applicable for the configured agent name(s).
    `boolean` | `isWhiteListed(URL url)` Check whether a URL belongs to a whitelisted host.
    `static void` | `main(String[] args)`
    `crawlercommons.robots.BaseRobotRules` | `parseRules(String url, byte[] content, String contentType, String robotName)` Parses the robots content using the `SimpleRobotRulesParser` from crawler commons
    `int` | `run(String[] args)`
    `void` | `setConf(Configuration conf)` Set the `Configuration` object

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### CACHE

          protected static final Hashtable<String,crawlercommons.robots.BaseRobotRules> CACHE

    - #### EMPTY_RULES

          public static final crawlercommons.robots.BaseRobotRules EMPTY_RULES

      A `BaseRobotRules` object appropriate for use when the `robots.txt` file is empty or missing; all requests are allowed.

    - #### FORBID_ALL_RULES

          public static crawlercommons.robots.BaseRobotRules FORBID_ALL_RULES

      A `BaseRobotRules` object appropriate for use when the `robots.txt` file is not fetched due to a `403/Forbidden` response; all requests are disallowed.

    - #### conf

          protected Configuration conf

    - #### agentNames

          protected String agentNames

    - #### whiteList

          protected Set<String> whiteList

      set of host names or IPs to be explicitly excluded from robots.txt checking

  - ### Constructor Detail

    - #### RobotRulesParser

          public RobotRulesParser()

    - #### RobotRulesParser

          public RobotRulesParser(Configuration conf)

  - ### Method Detail

    - #### setConf

          public void setConf(Configuration conf)

      Set the `Configuration` object
      Specified by:
        `setConf` in interface `Configurable`

    - #### getConf

          public Configuration getConf()

      Get the `Configuration` object
      Specified by:
        `getConf` in interface `Configurable`

    - #### isWhiteListed

          public boolean isWhiteListed(URL url)

      Check whether a URL belongs to a whitelisted host.

    - #### parseRules

          public crawlercommons.robots.BaseRobotRules parseRules(String url,
                                                                 byte[] content,
                                                                 String contentType,
                                                                 String robotName)

      Parses the robots content using the `SimpleRobotRulesParser` from crawler commons
      Parameters:
        `url` - A string containing url
        `content` - Contents of the robots file in a byte array
        `contentType` - The content type of the robots file
        `robotName` - A string containing all the robots agent names used by parser for matching
      Returns:
        BaseRobotRules object

    - #### getRobotRulesSet

          public crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol protocol,
                                                                       Text url,
                                                                       List<Content> robotsTxtContent)

      Fetch robots.txt (or it's protocol-specific equivalent) which applies to the given URL, parse it and return the set of robot rules applicable for the configured agent name(s).
      Parameters:
        `protocol` - `Protocol`
        `url` - URL to check
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robot rules (specific for this URL or default), never null

    - #### getRobotRulesSet

          public abstract crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol protocol,
                                                                                URL url,
                                                                                List<Content> robotsTxtContent)

      Fetch robots.txt (or it's protocol-specific equivalent) which applies to the given URL, parse it and return the set of robot rules applicable for the configured agent name(s).
      Parameters:
        `protocol` - `Protocol`
        `url` - URL to check
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robot rules (specific for this URL or default), never null

    - #### run

          public int run(String[] args)

      Specified by:
        `run` in interface `Tool`

    - #### main

          public static void main(String[] args)
                           throws Exception

      Throws:
        `Exception`

Copyright © 2021 The Apache Software Foundation