org.apache.nutch.tools

Class CommonCrawlFormatWARC

java.lang.Object

org.apache.nutch.tools.AbstractCommonCrawlFormat

org.apache.nutch.tools.CommonCrawlFormatWARC

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, CommonCrawlFormat

public class CommonCrawlFormatWARC
extends AbstractCommonCrawlFormat

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	MAX_WARC_FILE_SIZE
static java.lang.String	TEMPLATE

Fields inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

conf, content, inLinks, jsonArray, keyPrefix, LOG, metadata, reverseKey, reverseKeyValue, simpleDateFormat, url

Constructor Summary

Constructors

Constructor and Description
CommonCrawlFormatWARC(Configuration nutchConf, CommonCrawlConfig config)
CommonCrawlFormatWARC(java.lang.String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config, ParseData parseData)

Method Summary

Modifier and Type	Method and Description
void	close() Optional method that could be implemented if the actual format needs some close procedure.
protected void	closeArray(java.lang.String key, boolean nested, boolean newline)
protected void	closeObject(java.lang.String key)
protected java.lang.String	generateJson()
java.lang.String	getJsonData()
java.lang.String	getJsonData(java.lang.String url, Content content, Metadata metadata, ParseData parseData) Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
protected void	startArray(java.lang.String key, boolean nested, boolean newline)
protected void	startObject(java.lang.String key)
protected void	writeArrayValue(java.lang.String value)
protected void	writeKeyNull(java.lang.String key)
protected void	writeKeyValue(java.lang.String key, java.lang.String value)
protected java.net.URI	writeRequest(java.net.URI id)
protected java.net.URI	writeResponse()

Methods inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

getImported, getInLinks, getJsonData, getKey, getMethod, getRequestAccept, getRequestAcceptEncoding, getRequestAcceptLanguage, getRequestContactEmail, getRequestContactName, getRequestHostAddress, getRequestHostName, getRequestRobots, getRequestSoftware, getRequestUserAgent, getResponseAddress, getResponseContent, getResponseContentEncoding, getResponseContentType, getResponseDate, getResponseHostName, getResponseServer, getResponseStatus, getTimestamp, getUrl, setInLinks

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_WARC_FILE_SIZE

public static final java.lang.String MAX_WARC_FILE_SIZE

See Also:

Constant Field Values

TEMPLATE

public static final java.lang.String TEMPLATE

See Also:

Constant Field Values

Constructor Detail

CommonCrawlFormatWARC

public CommonCrawlFormatWARC(Configuration nutchConf,
                             CommonCrawlConfig config)
                      throws java.io.IOException

Throws:

java.io.IOException

CommonCrawlFormatWARC

public CommonCrawlFormatWARC(java.lang.String url,
                             Content content,
                             Metadata metadata,
                             Configuration nutchConf,
                             CommonCrawlConfig config,
                             ParseData parseData)
                      throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getJsonData

public java.lang.String getJsonData(java.lang.String url,
                                    Content content,
                                    Metadata metadata,
                                    ParseData parseData)
                             throws java.io.IOException

Description copied from interface: CommonCrawlFormat

Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL

Specified by:

getJsonData in interface CommonCrawlFormat

Overrides:

getJsonData in class AbstractCommonCrawlFormat

Returns:

Throws:

java.io.IOException

getJsonData

public java.lang.String getJsonData()
                             throws java.io.IOException

Specified by:

getJsonData in interface CommonCrawlFormat

Overrides:

getJsonData in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

writeResponse

protected java.net.URI writeResponse()
                              throws java.io.IOException,
                                     java.text.ParseException

Throws:

java.io.IOException

java.text.ParseException

writeRequest

protected java.net.URI writeRequest(java.net.URI id)
                             throws java.io.IOException,
                                    java.text.ParseException

Throws:

java.io.IOException

java.text.ParseException

generateJson

protected java.lang.String generateJson()
                                 throws java.io.IOException

Specified by:

generateJson in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

writeKeyValue

protected void writeKeyValue(java.lang.String key,
                             java.lang.String value)
                      throws java.io.IOException

Specified by:

writeKeyValue in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

writeKeyNull

protected void writeKeyNull(java.lang.String key)
                     throws java.io.IOException

Specified by:

writeKeyNull in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

startArray

protected void startArray(java.lang.String key,
                          boolean nested,
                          boolean newline)
                   throws java.io.IOException

Specified by:

startArray in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

closeArray

protected void closeArray(java.lang.String key,
                          boolean nested,
                          boolean newline)
                   throws java.io.IOException

Specified by:

closeArray in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

writeArrayValue

protected void writeArrayValue(java.lang.String value)
                        throws java.io.IOException

Specified by:

writeArrayValue in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

startObject

protected void startObject(java.lang.String key)
                    throws java.io.IOException

Specified by:

startObject in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

closeObject

protected void closeObject(java.lang.String key)
                    throws java.io.IOException

Specified by:

closeObject in class AbstractCommonCrawlFormat

Throws:

java.io.IOException

close

public void close()

Description copied from interface: CommonCrawlFormat

Optional method that could be implemented if the actual format needs some close procedure.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface CommonCrawlFormat

Overrides:

close in class AbstractCommonCrawlFormat