org.apache.nutch.protocol

## Interface RobotRules

-     public interface RobotRules

  This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.

- - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method and Description
    --- | ---
    `long` | `getCrawlDelay()` Get Crawl-Delay, in milliseconds.
    `long` | `getExpireTime()` Get expire time
    `boolean` | `isAllowed(java.net.URL url)` Returns `false` if the `robots.txt` file prohibits us from accessing the given `url`, or `true` otherwise.

- - ### Method Detail

    - #### getExpireTime

          long getExpireTime()

      Get expire time

    - #### getCrawlDelay

          long getCrawlDelay()

      Get Crawl-Delay, in milliseconds. This returns -1 if not set.

    - #### isAllowed

          boolean isAllowed(java.net.URL url)

      Returns `false` if the `robots.txt` file prohibits us from accessing the given `url`, or `true` otherwise.

Copyright © 2019 The Apache Software Foundation