org.apache.nutch.protocol.http.api

## Class HttpBase

- java.lang.Object
- - org.apache.nutch.protocol.http.api.HttpBase

- All Implemented Interfaces:
    Configurable, FieldPluggable, Pluggable, Protocol
  Direct Known Subclasses:
    Http, Http

      public abstract class HttpBase
      extends java.lang.Object
      implements Protocol

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected java.lang.String` | `accept` The "Accept" request header value.
    `protected java.lang.String` | `acceptCharset` The "Accept-Language" request header value.
    `protected java.lang.String` | `acceptLanguage` The "Accept-Language" request header value.
    `static int` | `BUFFER_SIZE`
    `protected int` | `maxContent` The length limit for downloaded content, in bytes.
    `protected java.lang.String` | `proxyHost` The proxy hostname.
    `protected int` | `proxyPort` The proxy port.
    `protected boolean` | `responseTime` Response Time
    `protected int` | `timeout` The network timeout in millisecond
    `protected java.util.Set<java.lang.String>` | `tlsPreferredCipherSuites` Which TLS/SSL cipher suites to support
    `protected java.util.Set<java.lang.String>` | `tlsPreferredProtocols` Which TLS/SSL protocols to support
    `protected boolean` | `useHttp11` Do we use HTTP/1.1?
    `protected boolean` | `useProxy` Indicates if a proxy is used
    `protected java.lang.String` | `userAgent` The Nutch 'User-Agent' request header

    - ### Fields inherited from interface org.apache.nutch.protocol.Protocol

      `CHECK_BLOCKING, CHECK_ROBOTS, X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `HttpBase()` Creates a new instance of HttpBase
    `HttpBase(org.slf4j.Logger logger)` Creates a new instance of HttpBase

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `java.lang.String` | `getAccept()`
    `java.lang.String` | `getAcceptCharset()`
    `java.lang.String` | `getAcceptLanguage()` Value of "Accept-Language" request header sent by Nutch.
    `Configuration` | `getConf()`
    `int` | `getMaxContent()`
    `ProtocolOutput` | `getProtocolOutput(java.lang.String url, WebPage page)`
    `java.lang.String` | `getProxyHost()`
    `int` | `getProxyPort()`
    `protected abstract Response` | `getResponse(java.net.URL url, WebPage page, boolean followRedirects)`
    `crawlercommons.robots.BaseRobotRules` | `getRobotRules(java.lang.String url, WebPage page)` Retrieve robot rules applicable for this url.
    `int` | `getTimeout()`
    `java.util.Set<java.lang.String>` | `getTlsPreferredCipherSuites()`
    `java.util.Set<java.lang.String>` | `getTlsPreferredProtocols()`
    `boolean` | `getUseHttp11()`
    `java.lang.String` | `getUserAgent()`
    `protected void` | `logConf()`
    `protected static void` | `main(HttpBase http, java.lang.String[] args)`
    `byte[]` | `processDeflateEncoded(byte[] compressed, java.net.URL url)`
    `byte[]` | `processGzipEncoded(byte[] compressed, java.net.URL url)`
    `void` | `setConf(Configuration conf)`
    `boolean` | `useProxy()`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.nutch.plugin.FieldPluggable

      `getFields`

- - ### Field Detail

    - #### BUFFER_SIZE

          public static final int BUFFER_SIZE

      See Also:
        Constant Field Values

    - #### proxyHost

          protected java.lang.String proxyHost

      The proxy hostname.

    - #### proxyPort

          protected int proxyPort

      The proxy port.

    - #### useProxy

          protected boolean useProxy

      Indicates if a proxy is used

    - #### timeout

          protected int timeout

      The network timeout in millisecond

    - #### maxContent

          protected int maxContent

      The length limit for downloaded content, in bytes.

    - #### userAgent

          protected java.lang.String userAgent

      The Nutch 'User-Agent' request header

    - #### acceptLanguage

          protected java.lang.String acceptLanguage

      The "Accept-Language" request header value.

    - #### acceptCharset

          protected java.lang.String acceptCharset

      The "Accept-Language" request header value.

    - #### accept

          protected java.lang.String accept

      The "Accept" request header value.

    - #### useHttp11

          protected boolean useHttp11

      Do we use HTTP/1.1?

    - #### responseTime

          protected boolean responseTime

      Response Time

    - #### tlsPreferredProtocols

          protected java.util.Set<java.lang.String> tlsPreferredProtocols

      Which TLS/SSL protocols to support

    - #### tlsPreferredCipherSuites

          protected java.util.Set<java.lang.String> tlsPreferredCipherSuites

      Which TLS/SSL cipher suites to support

  - ### Constructor Detail

    - #### HttpBase

          public HttpBase()

      Creates a new instance of HttpBase

    - #### HttpBase

          public HttpBase(org.slf4j.Logger logger)

      Creates a new instance of HttpBase

  - ### Method Detail

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`

    - #### getConf

          public Configuration getConf()

      Specified by:
        `getConf` in interface `Configurable`

    - #### getProtocolOutput

          public ProtocolOutput getProtocolOutput(java.lang.String url,
                                                  WebPage page)

      Specified by:
        `getProtocolOutput` in interface `Protocol`

    - #### getProxyHost

          public java.lang.String getProxyHost()

    - #### getProxyPort

          public int getProxyPort()

    - #### useProxy

          public boolean useProxy()

    - #### getTimeout

          public int getTimeout()

    - #### getMaxContent

          public int getMaxContent()

    - #### getUserAgent

          public java.lang.String getUserAgent()

    - #### getAcceptLanguage

          public java.lang.String getAcceptLanguage()

      Value of "Accept-Language" request header sent by Nutch.
      Returns:
        The value of the header "Accept-Language" header.

    - #### getAcceptCharset

          public java.lang.String getAcceptCharset()

    - #### getAccept

          public java.lang.String getAccept()

    - #### getUseHttp11

          public boolean getUseHttp11()

    - #### getTlsPreferredCipherSuites

          public java.util.Set<java.lang.String> getTlsPreferredCipherSuites()

    - #### getTlsPreferredProtocols

          public java.util.Set<java.lang.String> getTlsPreferredProtocols()

    - #### logConf

          protected void logConf()

    - #### processGzipEncoded

          public byte[] processGzipEncoded(byte[] compressed,
                                           java.net.URL url)
                                    throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### processDeflateEncoded

          public byte[] processDeflateEncoded(byte[] compressed,
                                              java.net.URL url)
                                       throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### main

          protected static void main(HttpBase http,
                                     java.lang.String[] args)
                              throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### getResponse

          protected abstract Response getResponse(java.net.URL url,
                                                  WebPage page,
                                                  boolean followRedirects)
                                           throws ProtocolException,
                                                  java.io.IOException

      Throws:
        `ProtocolException`
        `java.io.IOException`

    - #### getRobotRules

          public crawlercommons.robots.BaseRobotRules getRobotRules(java.lang.String url,
                                                                    WebPage page)

      Description copied from interface: `Protocol`
      Retrieve robot rules applicable for this url.
      Specified by:
        `getRobotRules` in interface `Protocol`
      Parameters:
        `url` - url to check
      Returns:
        robot rules (specific for this url or default), never null

Copyright © 2019 The Apache Software Foundation