org.apache.nutch.protocol.http.api

Class HttpBase

java.lang.Object

org.apache.nutch.protocol.http.api.HttpBase

All Implemented Interfaces:

Configurable, FieldPluggable, Pluggable, Protocol

Direct Known Subclasses:

Http, Http

public abstract class HttpBase
extends Object
implements Protocol

Field Summary

Fields

Modifier and Type	Field and Description
protected String	accept The "Accept" request header value.
protected String	acceptLanguage The "Accept-Language" request header value.
static int	BUFFER_SIZE
protected int	maxContent The length limit for downloaded content, in bytes.
protected String	proxyHost The proxy hostname.
protected int	proxyPort The proxy port.
protected boolean	responseTime Response Time
protected int	timeout The network timeout in millisecond
protected Set<String>	tlsPreferredCipherSuites Which TLS/SSL cipher suites to support
protected Set<String>	tlsPreferredProtocols Which TLS/SSL protocols to support
protected boolean	useHttp11 Do we use HTTP/1.1?
protected boolean	useProxy Indicates if a proxy is used
protected String	userAgent The Nutch 'User-Agent' request header

Fields inherited from interface org.apache.nutch.protocol.Protocol

CHECK_BLOCKING, CHECK_ROBOTS, X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
HttpBase() Creates a new instance of HttpBase
HttpBase(org.slf4j.Logger logger) Creates a new instance of HttpBase

Method Summary

Methods

Modifier and Type	Method and Description
String	getAccept()
String	getAcceptLanguage() Value of "Accept-Language" request header sent by Nutch.
Configuration	getConf()
int	getMaxContent()
ProtocolOutput	getProtocolOutput(String url, WebPage page)
String	getProxyHost()
int	getProxyPort()
protected abstract Response	getResponse(URL url, WebPage page, boolean followRedirects)
crawlercommons.robots.BaseRobotRules	getRobotRules(String url, WebPage page) Retrieve robot rules applicable for this url.
int	getTimeout()
Set<String>	getTlsPreferredCipherSuites()
Set<String>	getTlsPreferredProtocols()
boolean	getUseHttp11()
String	getUserAgent()
protected void	logConf()
protected static void	main(HttpBase http, String[] args)
byte[]	processDeflateEncoded(byte[] compressed, URL url)
byte[]	processGzipEncoded(byte[] compressed, URL url)
void	setConf(Configuration conf)
boolean	useProxy()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.nutch.plugin.FieldPluggable

getFields

Field Detail

BUFFER_SIZE

public static final int BUFFER_SIZE

See Also:

Constant Field Values

proxyHost

protected String proxyHost

The proxy hostname.

proxyPort

protected int proxyPort

The proxy port.

useProxy

protected boolean useProxy

Indicates if a proxy is used

timeout

protected int timeout

The network timeout in millisecond

maxContent

protected int maxContent

The length limit for downloaded content, in bytes.

userAgent

protected String userAgent

The Nutch 'User-Agent' request header

acceptLanguage

protected String acceptLanguage

The "Accept-Language" request header value.

accept

protected String accept

The "Accept" request header value.

useHttp11

protected boolean useHttp11

Do we use HTTP/1.1?

responseTime

protected boolean responseTime

Response Time

tlsPreferredProtocols

protected Set<String> tlsPreferredProtocols

Which TLS/SSL protocols to support

tlsPreferredCipherSuites

protected Set<String> tlsPreferredCipherSuites

Which TLS/SSL cipher suites to support

Constructor Detail

HttpBase

public HttpBase()

Creates a new instance of HttpBase

HttpBase

public HttpBase(org.slf4j.Logger logger)

Creates a new instance of HttpBase

Method Detail

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

getProtocolOutput

public ProtocolOutput getProtocolOutput(String url,
                               WebPage page)

Specified by:

getProtocolOutput in interface Protocol

getProxyHost

public String getProxyHost()

getProxyPort

public int getProxyPort()

useProxy

public boolean useProxy()

getTimeout

public int getTimeout()

getMaxContent

public int getMaxContent()

getUserAgent

public String getUserAgent()

getAcceptLanguage

public String getAcceptLanguage()

Value of "Accept-Language" request header sent by Nutch.

Returns:

The value of the header "Accept-Language" header.

getAccept

public String getAccept()

getUseHttp11

public boolean getUseHttp11()

getTlsPreferredCipherSuites

public Set<String> getTlsPreferredCipherSuites()

getTlsPreferredProtocols

public Set<String> getTlsPreferredProtocols()

logConf

protected void logConf()

processGzipEncoded

public byte[] processGzipEncoded(byte[] compressed,
                        URL url)
                          throws IOException

Throws:

IOException

processDeflateEncoded

public byte[] processDeflateEncoded(byte[] compressed,
                           URL url)
                             throws IOException

Throws:

IOException

main

protected static void main(HttpBase http,
        String[] args)
                    throws Exception

Throws:

Exception

getResponse

protected abstract Response getResponse(URL url,
                   WebPage page,
                   boolean followRedirects)
                                 throws ProtocolException,
                                        IOException

Throws:

ProtocolException

IOException

getRobotRules

public crawlercommons.robots.BaseRobotRules getRobotRules(String url,
                                                 WebPage page)

Description copied from interface: Protocol

Retrieve robot rules applicable for this url.

Specified by:

getRobotRules in interface Protocol

Parameters:

url - url to check

Returns:

robot rules (specific for this url or default), never null