org.apache.nutch.tools

Interface CommonCrawlFormat

All Superinterfaces:

java.lang.AutoCloseable, java.io.Closeable

All Known Implementing Classes:

AbstractCommonCrawlFormat, CommonCrawlFormatJackson, CommonCrawlFormatJettinson, CommonCrawlFormatSimple, CommonCrawlFormatWARC

public interface CommonCrawlFormat
extends java.io.Closeable

Interface for all CommonCrawl formatter. It provides the signature for the method used to get JSON data.

Author:

gtotaro

Method Summary

Modifier and Type	Method and Description
void	close() Optional method that could be implemented if the actual format needs some close procedure.
java.util.List<java.lang.String>	getInLinks() gets set of inlinks
java.lang.String	getJsonData()
java.lang.String	getJsonData(java.lang.String url, Content content, Metadata metadata) Returns a string representation of the JSON structure of the URL content
java.lang.String	getJsonData(java.lang.String url, Content content, Metadata metadata, ParseData parseData) Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
void	setInLinks(java.util.List<java.lang.String> inLinks) sets inlinks of this document

Method Detail

getJsonData

java.lang.String getJsonData()
                      throws java.io.IOException

Throws:

java.io.IOException

getJsonData

java.lang.String getJsonData(java.lang.String url,
                             Content content,
                             Metadata metadata)
                      throws java.io.IOException

Returns a string representation of the JSON structure of the URL content

Parameters:

url -

content -

metadata -

Returns:

Throws:

java.io.IOException

getJsonData

java.lang.String getJsonData(java.lang.String url,
                             Content content,
                             Metadata metadata,
                             ParseData parseData)
                      throws java.io.IOException

Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL

Parameters:

url -

content -

metadata -

Returns:

Throws:

java.io.IOException

setInLinks

void setInLinks(java.util.List<java.lang.String> inLinks)

sets inlinks of this document

Parameters:

inLinks - list of inlinks

getInLinks

java.util.List<java.lang.String> getInLinks()

gets set of inlinks

Returns:

gets inlinks of this document

close

void close()

Optional method that could be implemented if the actual format needs some close procedure.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable