org.apache.nutch.protocol.file

## Class File

- java.lang.Object
- - org.apache.nutch.protocol.file.File

- All Implemented Interfaces:
    Configurable, Pluggable, Protocol

      public class File
      extends Object
      implements Protocol

  This class is a protocol plugin used for file: scheme. It creates `FileResponse` object and gets the content of the url from it. Configurable parameters are `file.content.limit` and `file.crawl.parent` in nutch-default.xml defined under "file properties" section.
  Author:
    John Xing

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected static org.slf4j.Logger` | `LOG`

    - ### Fields inherited from interface org.apache.nutch.protocol.Protocol

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `File()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Configuration` | `getConf()` Get the `Configuration` object
    `ProtocolOutput` | `getProtocolOutput(Text url, CrawlDatum datum)` Creates a `FileResponse` object corresponding to the url and return a `ProtocolOutput` object as per the content received
    `crawlercommons.robots.BaseRobotRules` | `getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)` No robots parsing is done for file protocol.
    `static void` | `main(String[] args)` Quick way for running this class.
    `void` | `setConf(Configuration conf)` Set the `Configuration` object
    `void` | `setMaxContentLength(int maxContentLength)` Set the length after at which content is truncated.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### LOG

          protected static final org.slf4j.Logger LOG

  - ### Constructor Detail

    - #### File

          public File()

  - ### Method Detail

    - #### setConf

          public void setConf(Configuration conf)

      Set the `Configuration` object
      Specified by:
        `setConf` in interface `Configurable`

    - #### getConf

          public Configuration getConf()

      Get the `Configuration` object
      Specified by:
        `getConf` in interface `Configurable`

    - #### setMaxContentLength

          public void setMaxContentLength(int maxContentLength)

      Set the length after at which content is truncated.

    - #### getProtocolOutput

          public ProtocolOutput getProtocolOutput(Text url,
                                                  CrawlDatum datum)

      Creates a `FileResponse` object corresponding to the url and return a `ProtocolOutput` object as per the content received
      Specified by:
        `getProtocolOutput` in interface `Protocol`
      Parameters:
        `url` - Text containing the url
        `datum` - The CrawlDatum object corresponding to the url
      Returns:
        `ProtocolOutput` object for the content of the file indicated by url

    - #### main

          public static void main(String[] args)
                           throws Exception

      Quick way for running this class. Useful for debugging.
      Throws:
        `Exception`

    - #### getRobotRules

          public crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                                    CrawlDatum datum,
                                                                    List<Content> robotsTxtContent)

      No robots parsing is done for file protocol. So this returns a set of empty rules which will allow every url.
      Specified by:
        `getRobotRules` in interface `Protocol`
      Parameters:
        `url` - URL to check
        `datum` - page datum
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robot rules (specific for this URL or default), never null

Copyright © 2021 The Apache Software Foundation