org.apache.nutch.protocol.http.api

Class HttpBase

java.lang.Object

org.apache.nutch.protocol.http.api.HttpBase

All Implemented Interfaces:

Configurable, Pluggable, Protocol

Direct Known Subclasses:

Http, Http, Http, Http, Http

public abstract class HttpBase
extends java.lang.Object
implements Protocol

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	accept The "Accept" request header value.
protected java.lang.String	acceptCharset The "Accept-Language" request header value.
protected java.lang.String	acceptLanguage The "Accept-Language" request header value.
static int	BUFFER_SIZE
static Text	COOKIE
protected boolean	enableCookieHeader Controls whether or not to set Cookie HTTP header based on CrawlDatum metadata
protected boolean	enableIfModifiedsinceHeader Configuration directive for If-Modified-Since HTTP header
protected int	maxContent The length limit for downloaded content, in bytes.
protected long	maxCrawlDelay Skip page if Crawl-Delay longer than this value.
protected java.util.HashMap	proxyException The proxy exception list.
protected java.lang.String	proxyHost The proxy hostname.
protected int	proxyPort The proxy port.
static Text	RESPONSE_TIME
protected boolean	responseTime Record response time in CrawlDatum's meta data, see property http.store.responsetime.
protected int	timeout The network timeout in millisecond
protected java.util.Set<java.lang.String>	tlsPreferredCipherSuites Which TLS/SSL cipher suites to support
protected java.util.Set<java.lang.String>	tlsPreferredProtocols Which TLS/SSL protocols to support
protected boolean	useHttp11 Do we use HTTP/1.1?
protected boolean	useProxy Indicates if a proxy is used
protected java.lang.String	userAgent The Nutch 'User-Agent' request header

Fields inherited from interface org.apache.nutch.protocol.Protocol

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
HttpBase() Creates a new instance of HttpBase
HttpBase(org.slf4j.Logger logger) Creates a new instance of HttpBase

Method Summary

Modifier and Type	Method and Description
java.lang.String	getAccept()
java.lang.String	getAcceptCharset()
java.lang.String	getAcceptLanguage() Value of "Accept-Language" request header sent by Nutch.
Configuration	getConf()
int	getMaxContent()
ProtocolOutput	getProtocolOutput(Text url, CrawlDatum datum) Returns the Content for a fetchlist entry.
java.lang.String	getProxyHost()
int	getProxyPort()
protected abstract Response	getResponse(java.net.URL url, CrawlDatum datum, boolean followRedirects)
crawlercommons.robots.BaseRobotRules	getRobotRules(Text url, CrawlDatum datum, java.util.List<Content> robotsTxtContent) Retrieve robot rules applicable for this URL.
int	getTimeout()
java.util.Set<java.lang.String>	getTlsPreferredCipherSuites()
java.util.Set<java.lang.String>	getTlsPreferredProtocols()
boolean	getUseHttp11()
java.lang.String	getUserAgent()
boolean	isCookieEnabled()
boolean	isIfModifiedSinceEnabled()
protected void	logConf()
protected static void	main(HttpBase http, java.lang.String[] args)
byte[]	processDeflateEncoded(byte[] compressed, java.net.URL url)
byte[]	processGzipEncoded(byte[] compressed, java.net.URL url)
void	setConf(Configuration conf)
boolean	useProxy(java.net.URL url)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RESPONSE_TIME

public static final Text RESPONSE_TIME

COOKIE

public static final Text COOKIE

BUFFER_SIZE

public static final int BUFFER_SIZE

See Also:

Constant Field Values

proxyHost

protected java.lang.String proxyHost

The proxy hostname.

proxyPort

protected int proxyPort

The proxy port.

proxyException

protected java.util.HashMap proxyException

The proxy exception list.

useProxy

protected boolean useProxy

Indicates if a proxy is used

timeout

protected int timeout

The network timeout in millisecond

maxContent

protected int maxContent

The length limit for downloaded content, in bytes.

userAgent

protected java.lang.String userAgent

The Nutch 'User-Agent' request header

acceptLanguage

protected java.lang.String acceptLanguage

The "Accept-Language" request header value.

acceptCharset

protected java.lang.String acceptCharset

The "Accept-Language" request header value.

accept

protected java.lang.String accept

The "Accept" request header value.

useHttp11

protected boolean useHttp11

Do we use HTTP/1.1?

responseTime

protected boolean responseTime

Record response time in CrawlDatum's meta data, see property http.store.responsetime.

maxCrawlDelay

protected long maxCrawlDelay

Skip page if Crawl-Delay longer than this value.

tlsPreferredProtocols

protected java.util.Set<java.lang.String> tlsPreferredProtocols

Which TLS/SSL protocols to support

tlsPreferredCipherSuites

protected java.util.Set<java.lang.String> tlsPreferredCipherSuites

Which TLS/SSL cipher suites to support

enableIfModifiedsinceHeader

protected boolean enableIfModifiedsinceHeader

Configuration directive for If-Modified-Since HTTP header

enableCookieHeader

protected boolean enableCookieHeader

Controls whether or not to set Cookie HTTP header based on CrawlDatum metadata

Constructor Detail

HttpBase

public HttpBase()

Creates a new instance of HttpBase

HttpBase

public HttpBase(org.slf4j.Logger logger)

Creates a new instance of HttpBase

Method Detail

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

getProtocolOutput

public ProtocolOutput getProtocolOutput(Text url,
                                        CrawlDatum datum)

Description copied from interface: Protocol

Returns the Content for a fetchlist entry.

Specified by:

getProtocolOutput in interface Protocol

getProxyHost

public java.lang.String getProxyHost()

getProxyPort

public int getProxyPort()

useProxy

public boolean useProxy(java.net.URL url)

getTimeout

public int getTimeout()

isIfModifiedSinceEnabled

public boolean isIfModifiedSinceEnabled()

isCookieEnabled

public boolean isCookieEnabled()

getMaxContent

public int getMaxContent()

getUserAgent

public java.lang.String getUserAgent()

getAcceptLanguage

public java.lang.String getAcceptLanguage()

Value of "Accept-Language" request header sent by Nutch.

Returns:

The value of the header "Accept-Language" header.

getAcceptCharset

public java.lang.String getAcceptCharset()

getAccept

public java.lang.String getAccept()

getUseHttp11

public boolean getUseHttp11()

getTlsPreferredCipherSuites

public java.util.Set<java.lang.String> getTlsPreferredCipherSuites()

getTlsPreferredProtocols

public java.util.Set<java.lang.String> getTlsPreferredProtocols()

logConf

protected void logConf()

processGzipEncoded

public byte[] processGzipEncoded(byte[] compressed,
                                 java.net.URL url)
                          throws java.io.IOException

Throws:

java.io.IOException

processDeflateEncoded

public byte[] processDeflateEncoded(byte[] compressed,
                                    java.net.URL url)
                             throws java.io.IOException

Throws:

java.io.IOException

main

protected static void main(HttpBase http,
                           java.lang.String[] args)
                    throws java.lang.Exception

Throws:

java.lang.Exception

getResponse

protected abstract Response getResponse(java.net.URL url,
                                        CrawlDatum datum,
                                        boolean followRedirects)
                                 throws ProtocolException,
                                        java.io.IOException

Throws:

ProtocolException

java.io.IOException

getRobotRules

public crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                          CrawlDatum datum,
                                                          java.util.List<Content> robotsTxtContent)

Description copied from interface: Protocol

Retrieve robot rules applicable for this URL.

Specified by:

getRobotRules in interface Protocol

Parameters:

url - URL to check

datum - page datum

robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.

Returns:

robot rules (specific for this URL or default), never null