org.apache.nutch.tools

Class AbstractCommonCrawlFormat

java.lang.Object

org.apache.nutch.tools.AbstractCommonCrawlFormat

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, CommonCrawlFormat

Direct Known Subclasses:

CommonCrawlFormatJackson, CommonCrawlFormatJettinson, CommonCrawlFormatSimple, CommonCrawlFormatWARC

public abstract class AbstractCommonCrawlFormat
extends java.lang.Object
implements CommonCrawlFormat

Abstract class that implements { @see org.apache.nutch.tools.CommonCrawlFormat } interface.

Field Summary

Fields

Modifier and Type	Field and Description
protected Configuration	conf
protected Content	content
protected java.util.List<java.lang.String>	inLinks
protected boolean	jsonArray
protected java.lang.String	keyPrefix
protected static org.slf4j.Logger	LOG
protected Metadata	metadata
protected boolean	reverseKey
protected java.lang.String	reverseKeyValue
protected boolean	simpleDateFormat
protected java.lang.String	url

Constructor Summary

Constructors

Constructor and Description
AbstractCommonCrawlFormat(java.lang.String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config)

Method Summary

Modifier and Type	Method and Description
void	close() Optional method that could be implemented if the actual format needs some close procedure.
protected abstract void	closeArray(java.lang.String key, boolean nested, boolean newline)
protected abstract void	closeObject(java.lang.String key)
protected abstract java.lang.String	generateJson()
protected java.lang.String	getImported()
java.util.List<java.lang.String>	getInLinks() gets set of inlinks
java.lang.String	getJsonData()
java.lang.String	getJsonData(java.lang.String url, Content content, Metadata metadata) Returns a string representation of the JSON structure of the URL content
java.lang.String	getJsonData(java.lang.String url, Content content, Metadata metadata, ParseData parseData) Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
protected java.lang.String	getKey()
protected java.lang.String	getMethod()
protected java.lang.String	getRequestAccept()
protected java.lang.String	getRequestAcceptEncoding()
protected java.lang.String	getRequestAcceptLanguage()
protected java.lang.String	getRequestContactEmail()
protected java.lang.String	getRequestContactName()
protected java.lang.String	getRequestHostAddress()
protected java.lang.String	getRequestHostName()
protected java.lang.String	getRequestRobots()
protected java.lang.String	getRequestSoftware()
protected java.lang.String	getRequestUserAgent()
protected java.lang.String	getResponseAddress()
protected java.lang.String	getResponseContent()
protected java.lang.String	getResponseContentEncoding()
protected java.lang.String	getResponseContentType()
protected java.lang.String	getResponseDate()
protected java.lang.String	getResponseHostName()
protected java.lang.String	getResponseServer()
protected java.lang.String	getResponseStatus()
protected java.lang.String	getTimestamp()
protected java.lang.String	getUrl()
void	setInLinks(java.util.List<java.lang.String> inLinks) sets inlinks of this document
protected abstract void	startArray(java.lang.String key, boolean nested, boolean newline)
protected abstract void	startObject(java.lang.String key)
protected abstract void	writeArrayValue(java.lang.String value)
protected abstract void	writeKeyNull(java.lang.String key)
protected abstract void	writeKeyValue(java.lang.String key, java.lang.String value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

url

protected java.lang.String url

content

protected Content content

metadata

protected Metadata metadata

conf

protected Configuration conf

keyPrefix

protected java.lang.String keyPrefix

simpleDateFormat

protected boolean simpleDateFormat

jsonArray

protected boolean jsonArray

reverseKey

protected boolean reverseKey

reverseKeyValue

protected java.lang.String reverseKeyValue

inLinks

protected java.util.List<java.lang.String> inLinks

Constructor Detail

AbstractCommonCrawlFormat

public AbstractCommonCrawlFormat(java.lang.String url,
                                 Content content,
                                 Metadata metadata,
                                 Configuration nutchConf,
                                 CommonCrawlConfig config)
                          throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getJsonData

public java.lang.String getJsonData(java.lang.String url,
                                    Content content,
                                    Metadata metadata)
                             throws java.io.IOException

Description copied from interface: CommonCrawlFormat

Returns a string representation of the JSON structure of the URL content

Specified by:

getJsonData in interface CommonCrawlFormat

Returns:

Throws:

java.io.IOException

getJsonData

public java.lang.String getJsonData(java.lang.String url,
                                    Content content,
                                    Metadata metadata,
                                    ParseData parseData)
                             throws java.io.IOException

Description copied from interface: CommonCrawlFormat

Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL

Specified by:

getJsonData in interface CommonCrawlFormat

Returns:

Throws:

java.io.IOException

getJsonData

public java.lang.String getJsonData()
                             throws java.io.IOException

Specified by:

getJsonData in interface CommonCrawlFormat

Throws:

java.io.IOException

writeKeyValue

protected abstract void writeKeyValue(java.lang.String key,
                                      java.lang.String value)
                               throws java.io.IOException

Throws:

java.io.IOException

writeKeyNull

protected abstract void writeKeyNull(java.lang.String key)
                              throws java.io.IOException

Throws:

java.io.IOException

startArray

protected abstract void startArray(java.lang.String key,
                                   boolean nested,
                                   boolean newline)
                            throws java.io.IOException

Throws:

java.io.IOException

closeArray

protected abstract void closeArray(java.lang.String key,
                                   boolean nested,
                                   boolean newline)
                            throws java.io.IOException

Throws:

java.io.IOException

writeArrayValue

protected abstract void writeArrayValue(java.lang.String value)
                                 throws java.io.IOException

Throws:

java.io.IOException

startObject

protected abstract void startObject(java.lang.String key)
                             throws java.io.IOException

Throws:

java.io.IOException

closeObject

protected abstract void closeObject(java.lang.String key)
                             throws java.io.IOException

Throws:

java.io.IOException

generateJson

protected abstract java.lang.String generateJson()
                                          throws java.io.IOException

Throws:

java.io.IOException

getUrl

protected java.lang.String getUrl()

getTimestamp

protected java.lang.String getTimestamp()

getMethod

protected java.lang.String getMethod()

getRequestHostName

protected java.lang.String getRequestHostName()

getRequestHostAddress

protected java.lang.String getRequestHostAddress()

getRequestSoftware

protected java.lang.String getRequestSoftware()

getRequestRobots

protected java.lang.String getRequestRobots()

getRequestContactName

protected java.lang.String getRequestContactName()

getRequestContactEmail

protected java.lang.String getRequestContactEmail()

getRequestAccept

protected java.lang.String getRequestAccept()

getRequestAcceptEncoding

protected java.lang.String getRequestAcceptEncoding()

getRequestAcceptLanguage

protected java.lang.String getRequestAcceptLanguage()

getRequestUserAgent

protected java.lang.String getRequestUserAgent()

getResponseStatus

protected java.lang.String getResponseStatus()

getResponseHostName

protected java.lang.String getResponseHostName()

getResponseAddress

protected java.lang.String getResponseAddress()

getResponseContentEncoding

protected java.lang.String getResponseContentEncoding()

getResponseContentType

protected java.lang.String getResponseContentType()

getInLinks

public java.util.List<java.lang.String> getInLinks()

Description copied from interface: CommonCrawlFormat

gets set of inlinks

Specified by:

getInLinks in interface CommonCrawlFormat

Returns:

gets inlinks of this document

setInLinks

public void setInLinks(java.util.List<java.lang.String> inLinks)

Description copied from interface: CommonCrawlFormat

sets inlinks of this document

Specified by:

setInLinks in interface CommonCrawlFormat

Parameters:

inLinks - list of inlinks

getResponseDate

protected java.lang.String getResponseDate()

getResponseServer

protected java.lang.String getResponseServer()

getResponseContent

protected java.lang.String getResponseContent()

getKey

protected java.lang.String getKey()

getImported

protected java.lang.String getImported()

close

public void close()

Description copied from interface: CommonCrawlFormat

Optional method that could be implemented if the actual format needs some close procedure.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface CommonCrawlFormat