Package org.apache.nutch.protocol.http.api

## Class HttpBase

- java.lang.Object
- - org.apache.nutch.protocol.http.api.HttpBase

- All Implemented Interfaces:
    `Configurable`, `Pluggable`, `Protocol`
  Direct Known Subclasses:
    `Http`, `Http`, `Http`, `Http`, `Http`, `OkHttp`

      public abstract class HttpBase
      extends Object
      implements Protocol

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `protected String` | `accept` | The "Accept" request header value.
    `protected String` | `acceptCharset` | The "Accept-Charset" request header value.
    `protected String` | `acceptLanguage` | The "Accept-Language" request header value.
    `static int` | `BUFFER_SIZE` |
    `static Text` | `COOKIE` |
    `protected boolean` | `enableCookieHeader` | Controls whether or not to set Cookie HTTP header based on CrawlDatum metadata
    `protected boolean` | `enableIfModifiedsinceHeader` | Configuration directive for If-Modified-Since HTTP header
    `protected int` | `maxContent` | The length limit for downloaded content, in bytes.
    `protected long` | `maxCrawlDelay` | Skip page if Crawl-Delay longer than this value.
    `protected int` | `maxDuration` | The time limit to download the entire content, in seconds.
    `protected boolean` | `partialAsTruncated` | Whether to save partial fetches as truncated content.
    `protected HashMap<String,​String>` | `proxyException` | The proxy exception list.
    `protected String` | `proxyHost` | The proxy hostname.
    `protected int` | `proxyPort` | The proxy port.
    `protected Proxy.Type` | `proxyType` | The proxy port.
    `static Text` | `RESPONSE_TIME` |
    `protected boolean` | `responseTime` | Record response time in CrawlDatum's meta data, see property http.store.responsetime.
    `protected boolean` | `storeHttpHeaders` | Record the HTTP response header in the metadata, see property `store.http.headers`.
    `protected boolean` | `storeHttpRequest` | Record the HTTP request in the metadata, see property `store.http.request`.
    `protected boolean` | `storeIPAddress` | Record the IP address of the responding server, see property `store.ip.address`.
    `protected boolean` | `storeProtocolVersions` | Record the HTTP and SSL/TLS protocol versions and the SSL/TLS cipher suites, see property `store.protocol.versions`.
    `protected int` | `timeout` | The network timeout in millisecond
    `protected boolean` | `tlsCheckCertificate` | Whether to check TLS/SSL certificates
    `protected Set<String>` | `tlsPreferredCipherSuites` | Which TLS/SSL cipher suites to support
    `protected Set<String>` | `tlsPreferredProtocols` | Which TLS/SSL protocols to support
    `protected boolean` | `useHttp11` | Do we use HTTP/1.1?
    `protected boolean` | `useHttp2` | Whether to use HTTP/2
    `protected boolean` | `useProxy` | Indicates if a proxy is used
    `protected String` | `userAgent` | The Nutch 'User-Agent' request header

    - ### Fields inherited from interface org.apache.nutch.protocol.Protocol

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `HttpBase()` | Creates a new instance of HttpBase
    `HttpBase​(org.slf4j.Logger logger)` | Creates a new instance of HttpBase

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `String` | `getAccept()` |
    `String` | `getAcceptCharset()` |
    `String` | `getAcceptLanguage()` | Value of "Accept-Language" request header sent by Nutch.
    `Configuration` | `getConf()` |
    `String` | `getCookie​(URL url)` | If per-host cookies are configured, this method will look it up for the given url.
    `int` | `getMaxContent()` |
    `int` | `getMaxDuration()` | The time limit to download the entire content, in seconds.
    `ProtocolOutput` | `getProtocolOutput​(Text url, CrawlDatum datum)` | Get the `ProtocolOutput` for a given url and crawldatum
    `String` | `getProxyHost()` |
    `int` | `getProxyPort()` |
    `protected abstract Response` | `getResponse​(URL url, CrawlDatum datum, boolean followRedirects)` |
    `crawlercommons.robots.BaseRobotRules` | `getRobotRules​(Text url, CrawlDatum datum, List<Content> robotsTxtContent)` | Retrieve robot rules applicable for this URL.
    `int` | `getTimeout()` |
    `Set<String>` | `getTlsPreferredCipherSuites()` |
    `Set<String>` | `getTlsPreferredProtocols()` |
    `boolean` | `getUseHttp11()` |
    `String` | `getUserAgent()` |
    `boolean` | `isCookieEnabled()` |
    `boolean` | `isIfModifiedSinceEnabled()` |
    `boolean` | `isStoreHttpHeaders()` |
    `boolean` | `isStoreHttpRequest()` |
    `boolean` | `isStoreIPAddress()` |
    `boolean` | `isStorePartialAsTruncated()` | Whether to save partial fetches as truncated content, cf.
    `boolean` | `isTlsCheckCertificates()` |
    `protected void` | `logConf()` |
    `protected static void` | `main​(HttpBase http, String[] args)` |
    `byte[]` | `processDeflateEncoded​(byte[] compressed, URL url)` |
    `byte[]` | `processGzipEncoded​(byte[] compressed, URL url)` |
    `void` | `setConf​(Configuration conf)` |
    `boolean` | `useProxy​(String host)` |
    `boolean` | `useProxy​(URI uri)` |
    `boolean` | `useProxy​(URL url)` |

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### RESPONSE_TIME

          public static final Text RESPONSE_TIME

    - #### COOKIE

          public static final Text COOKIE

    - #### BUFFER_SIZE

          public static final int BUFFER_SIZE

      See Also:
        Constant Field Values

    - #### proxyHost

          protected String proxyHost

      The proxy hostname.

    - #### proxyPort

          protected int proxyPort

      The proxy port.

    - #### proxyType

          protected Proxy.Type proxyType

      The proxy port.

    - #### proxyException

          protected HashMap<String,​String> proxyException

      The proxy exception list.

    - #### useProxy

          protected boolean useProxy

      Indicates if a proxy is used

    - #### timeout

          protected int timeout

      The network timeout in millisecond

    - #### maxContent

          protected int maxContent

      The length limit for downloaded content, in bytes.

    - #### maxDuration

          protected int maxDuration

      The time limit to download the entire content, in seconds.

    - #### partialAsTruncated

          protected boolean partialAsTruncated

      Whether to save partial fetches as truncated content.

    - #### userAgent

          protected String userAgent

      The Nutch 'User-Agent' request header

    - #### acceptLanguage

          protected String acceptLanguage

      The "Accept-Language" request header value.

    - #### acceptCharset

          protected String acceptCharset

      The "Accept-Charset" request header value.

    - #### accept

          protected String accept

      The "Accept" request header value.

    - #### useHttp11

          protected boolean useHttp11

      Do we use HTTP/1.1?

    - #### useHttp2

          protected boolean useHttp2

      Whether to use HTTP/2

    - #### responseTime

          protected boolean responseTime

      Record response time in CrawlDatum's meta data, see property http.store.responsetime.

    - #### storeIPAddress

          protected boolean storeIPAddress

      Record the IP address of the responding server, see property `store.ip.address`.

    - #### storeHttpRequest

          protected boolean storeHttpRequest

      Record the HTTP request in the metadata, see property `store.http.request`.

    - #### storeHttpHeaders

          protected boolean storeHttpHeaders

      Record the HTTP response header in the metadata, see property `store.http.headers`.

    - #### storeProtocolVersions

          protected boolean storeProtocolVersions

      Record the HTTP and SSL/TLS protocol versions and the SSL/TLS cipher suites, see property `store.protocol.versions`.

    - #### maxCrawlDelay

          protected long maxCrawlDelay

      Skip page if Crawl-Delay longer than this value.

    - #### tlsCheckCertificate

          protected boolean tlsCheckCertificate

      Whether to check TLS/SSL certificates

    - #### tlsPreferredProtocols

          protected Set<String> tlsPreferredProtocols

      Which TLS/SSL protocols to support

    - #### tlsPreferredCipherSuites

          protected Set<String> tlsPreferredCipherSuites

      Which TLS/SSL cipher suites to support

    - #### enableIfModifiedsinceHeader

          protected boolean enableIfModifiedsinceHeader

      Configuration directive for If-Modified-Since HTTP header

    - #### enableCookieHeader

          protected boolean enableCookieHeader

      Controls whether or not to set Cookie HTTP header based on CrawlDatum metadata

  - ### Constructor Detail

    - #### HttpBase

          public HttpBase()

      Creates a new instance of HttpBase

    - #### HttpBase

          public HttpBase​(org.slf4j.Logger logger)

      Creates a new instance of HttpBase
      Parameters:
        `logger` - the `Logger` to use in this HttpBase

  - ### Method Detail

    - #### setConf

          public void setConf​(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`

    - #### getConf

          public Configuration getConf()

      Specified by:
        `getConf` in interface `Configurable`

    - #### getProtocolOutput

          public ProtocolOutput getProtocolOutput​(Text url,
                                                  CrawlDatum datum)

      Description copied from interface: `Protocol`
      Get the `ProtocolOutput` for a given url and crawldatum
      Specified by:
        `getProtocolOutput` in interface `Protocol`
      Parameters:
        `url` - canonical url
        `datum` - associated `CrawlDatum`
      Returns:
        the `ProtocolOutput`

    - #### getProxyHost

          public String getProxyHost()

    - #### getProxyPort

          public int getProxyPort()

    - #### useProxy

          public boolean useProxy​(URL url)

    - #### useProxy

          public boolean useProxy​(URI uri)

    - #### useProxy

          public boolean useProxy​(String host)

    - #### getTimeout

          public int getTimeout()

    - #### isIfModifiedSinceEnabled

          public boolean isIfModifiedSinceEnabled()

    - #### isCookieEnabled

          public boolean isCookieEnabled()

    - #### isStoreIPAddress

          public boolean isStoreIPAddress()

    - #### isStoreHttpRequest

          public boolean isStoreHttpRequest()

    - #### isStoreHttpHeaders

          public boolean isStoreHttpHeaders()

    - #### getMaxContent

          public int getMaxContent()

    - #### getMaxDuration

          public int getMaxDuration()

      The time limit to download the entire content, in seconds. See the property `http.time.limit`.
      Returns:
        the maximum duration

    - #### isStorePartialAsTruncated

          public boolean isStorePartialAsTruncated()

      Whether to save partial fetches as truncated content, cf. the property `http.partial.truncated`.
      Returns:
        true if partially fetched truncated content is stored

    - #### getUserAgent

          public String getUserAgent()

    - #### getCookie

          public String getCookie​(URL url)

      If per-host cookies are configured, this method will look it up for the given url.
      Parameters:
        `url` - the url to look-up a cookie for
      Returns:
        the cookie or null

    - #### getAcceptLanguage

          public String getAcceptLanguage()

      Value of "Accept-Language" request header sent by Nutch.
      Returns:
        The value of the header "Accept-Language" header.

    - #### getAcceptCharset

          public String getAcceptCharset()

    - #### getAccept

          public String getAccept()

    - #### getUseHttp11

          public boolean getUseHttp11()

    - #### isTlsCheckCertificates

          public boolean isTlsCheckCertificates()

    - #### getTlsPreferredCipherSuites

          public Set<String> getTlsPreferredCipherSuites()

    - #### getTlsPreferredProtocols

          public Set<String> getTlsPreferredProtocols()

    - #### logConf

          protected void logConf()

    - #### processGzipEncoded

          public byte[] processGzipEncoded​(byte[] compressed,
                                           URL url)
                                    throws IOException

      Throws:
        `IOException`

    - #### processDeflateEncoded

          public byte[] processDeflateEncoded​(byte[] compressed,
                                              URL url)
                                       throws IOException

      Throws:
        `IOException`

    - #### main

          protected static void main​(HttpBase http,
                                     String[] args)
                              throws Exception

      Throws:
        `Exception`

    - #### getResponse

          protected abstract Response getResponse​(URL url,
                                                  CrawlDatum datum,
                                                  boolean followRedirects)
                                           throws ProtocolException,
                                                  IOException

      Throws:
        `ProtocolException`
        `IOException`

    - #### getRobotRules

          public crawlercommons.robots.BaseRobotRules getRobotRules​(Text url,
                                                                    CrawlDatum datum,
                                                                    List<Content> robotsTxtContent)

      Description copied from interface: `Protocol`
      Retrieve robot rules applicable for this URL.
      Specified by:
        `getRobotRules` in interface `Protocol`
      Parameters:
        `url` - URL to check
        `datum` - page datum
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robot rules (specific for this URL or default), never null