org.apache.nutch.protocol

## Class RobotRulesParser

- java.lang.Object
- - org.apache.nutch.protocol.RobotRulesParser

- All Implemented Interfaces:
    Configurable
  Direct Known Subclasses:
    FtpRobotRulesParser, HttpRobotRulesParser

      public abstract class RobotRulesParser
      extends java.lang.Object
      implements Configurable

  This class uses crawler-commons for handling the parsing of `robots.txt` files. It emits SimpleRobotRules objects, which describe the download permissions as described in SimpleRobotRulesParser.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected java.lang.String` | `agentNames`
    `protected static java.util.Hashtable<java.lang.String,crawlercommons.robots.BaseRobotRules>` | `CACHE`
    `static crawlercommons.robots.BaseRobotRules` | `EMPTY_RULES` A `BaseRobotRules` object appropriate for use when the `robots.txt` file is empty or missing; all requests are allowed.
    `static crawlercommons.robots.BaseRobotRules` | `FORBID_ALL_RULES` A `BaseRobotRules` object appropriate for use when the `robots.txt` file is not fetched due to a `403/Forbidden` response; all requests are disallowed.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `RobotRulesParser()`
    `RobotRulesParser(Configuration conf)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Configuration` | `getConf()` Get the `Configuration` object
    `crawlercommons.robots.BaseRobotRules` | `getRobotRulesSet(Protocol protocol, java.lang.String url)`
    `abstract crawlercommons.robots.BaseRobotRules` | `getRobotRulesSet(Protocol protocol, java.net.URL url)`
    `static void` | `main(java.lang.String[] argv)` command-line main for testing
    `crawlercommons.robots.BaseRobotRules` | `parseRules(java.lang.String url, byte[] content, java.lang.String contentType, java.lang.String robotName)` Parses the robots content using the `SimpleRobotRulesParser` from crawler commons
    `void` | `setConf(Configuration conf)` Set the `Configuration` object

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### CACHE

          protected static final java.util.Hashtable<java.lang.String,crawlercommons.robots.BaseRobotRules> CACHE

    - #### EMPTY_RULES

          public static final crawlercommons.robots.BaseRobotRules EMPTY_RULES

      A `BaseRobotRules` object appropriate for use when the `robots.txt` file is empty or missing; all requests are allowed.

    - #### FORBID_ALL_RULES

          public static crawlercommons.robots.BaseRobotRules FORBID_ALL_RULES

      A `BaseRobotRules` object appropriate for use when the `robots.txt` file is not fetched due to a `403/Forbidden` response; all requests are disallowed.

    - #### agentNames

          protected java.lang.String agentNames

  - ### Constructor Detail

    - #### RobotRulesParser

          public RobotRulesParser()

    - #### RobotRulesParser

          public RobotRulesParser(Configuration conf)

  - ### Method Detail

    - #### setConf

          public void setConf(Configuration conf)

      Set the `Configuration` object
      Specified by:
        `setConf` in interface `Configurable`

    - #### getConf

          public Configuration getConf()

      Get the `Configuration` object
      Specified by:
        `getConf` in interface `Configurable`

    - #### parseRules

          public crawlercommons.robots.BaseRobotRules parseRules(java.lang.String url,
                                                                 byte[] content,
                                                                 java.lang.String contentType,
                                                                 java.lang.String robotName)

      Parses the robots content using the `SimpleRobotRulesParser` from crawler commons
      Parameters:
        `url` - A string containing url
        `content` - Contents of the robots file in a byte array
        `contentType` - The content type of the robots file
        `robotName` - A string containing all the robots agent names used by parser for matching
      Returns:
        BaseRobotRules object

    - #### getRobotRulesSet

          public crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol protocol,
                                                                       java.lang.String url)

    - #### getRobotRulesSet

          public abstract crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol protocol,
                                                                                java.net.URL url)

    - #### main

          public static void main(java.lang.String[] argv)

      command-line main for testing

Copyright © 2019 The Apache Software Foundation