org.apache.nutch.tools

## Class AbstractCommonCrawlFormat

- java.lang.Object
- - org.apache.nutch.tools.AbstractCommonCrawlFormat

- All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, CommonCrawlFormat
  Direct Known Subclasses:
    CommonCrawlFormatJackson, CommonCrawlFormatJettinson, CommonCrawlFormatSimple, CommonCrawlFormatWARC

      public abstract class AbstractCommonCrawlFormat
      extends java.lang.Object
      implements CommonCrawlFormat

  Abstract class that implements { @see org.apache.nutch.tools.CommonCrawlFormat } interface.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected Configuration` | `conf`
    `protected Content` | `content`
    `protected java.util.List<java.lang.String>` | `inLinks`
    `protected boolean` | `jsonArray`
    `protected java.lang.String` | `keyPrefix`
    `protected static org.slf4j.Logger` | `LOG`
    `protected Metadata` | `metadata`
    `protected boolean` | `reverseKey`
    `protected java.lang.String` | `reverseKeyValue`
    `protected boolean` | `simpleDateFormat`
    `protected java.lang.String` | `url`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `AbstractCommonCrawlFormat(java.lang.String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config)`

  - ### Method Summary

    All Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()` Optional method that could be implemented if the actual format needs some close procedure.
    `protected abstract void` | `closeArray(java.lang.String key, boolean nested, boolean newline)`
    `protected abstract void` | `closeObject(java.lang.String key)`
    `protected abstract java.lang.String` | `generateJson()`
    `protected java.lang.String` | `getImported()`
    `java.util.List<java.lang.String>` | `getInLinks()` gets set of inlinks
    `java.lang.String` | `getJsonData()`
    `java.lang.String` | `getJsonData(java.lang.String url, Content content, Metadata metadata)` Returns a string representation of the JSON structure of the URL content
    `java.lang.String` | `getJsonData(java.lang.String url, Content content, Metadata metadata, ParseData parseData)` Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
    `protected java.lang.String` | `getKey()`
    `protected java.lang.String` | `getMethod()`
    `protected java.lang.String` | `getRequestAccept()`
    `protected java.lang.String` | `getRequestAcceptEncoding()`
    `protected java.lang.String` | `getRequestAcceptLanguage()`
    `protected java.lang.String` | `getRequestContactEmail()`
    `protected java.lang.String` | `getRequestContactName()`
    `protected java.lang.String` | `getRequestHostAddress()`
    `protected java.lang.String` | `getRequestHostName()`
    `protected java.lang.String` | `getRequestRobots()`
    `protected java.lang.String` | `getRequestSoftware()`
    `protected java.lang.String` | `getRequestUserAgent()`
    `protected java.lang.String` | `getResponseAddress()`
    `protected java.lang.String` | `getResponseContent()`
    `protected java.lang.String` | `getResponseContentEncoding()`
    `protected java.lang.String` | `getResponseContentType()`
    `protected java.lang.String` | `getResponseDate()`
    `protected java.lang.String` | `getResponseHostName()`
    `protected java.lang.String` | `getResponseServer()`
    `protected java.lang.String` | `getResponseStatus()`
    `protected java.lang.String` | `getTimestamp()`
    `protected java.lang.String` | `getUrl()`
    `void` | `setInLinks(java.util.List<java.lang.String> inLinks)` sets inlinks of this document
    `protected abstract void` | `startArray(java.lang.String key, boolean nested, boolean newline)`
    `protected abstract void` | `startObject(java.lang.String key)`
    `protected abstract void` | `writeArrayValue(java.lang.String value)`
    `protected abstract void` | `writeKeyNull(java.lang.String key)`
    `protected abstract void` | `writeKeyValue(java.lang.String key, java.lang.String value)`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### LOG

          protected static final org.slf4j.Logger LOG

    - #### url

          protected java.lang.String url

    - #### content

          protected Content content

    - #### metadata

          protected Metadata metadata

    - #### conf

          protected Configuration conf

    - #### keyPrefix

          protected java.lang.String keyPrefix

    - #### simpleDateFormat

          protected boolean simpleDateFormat

    - #### jsonArray

          protected boolean jsonArray

    - #### reverseKey

          protected boolean reverseKey

    - #### reverseKeyValue

          protected java.lang.String reverseKeyValue

    - #### inLinks

          protected java.util.List<java.lang.String> inLinks

  - ### Constructor Detail

    - #### AbstractCommonCrawlFormat

          public AbstractCommonCrawlFormat(java.lang.String url,
                                           Content content,
                                           Metadata metadata,
                                           Configuration nutchConf,
                                           CommonCrawlConfig config)
                                    throws java.io.IOException

      Throws:
        `java.io.IOException`

  - ### Method Detail

    - #### getJsonData

          public java.lang.String getJsonData(java.lang.String url,
                                              Content content,
                                              Metadata metadata)
                                       throws java.io.IOException

      Description copied from interface: `CommonCrawlFormat`
      Returns a string representation of the JSON structure of the URL content
      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Returns:
      Throws:
        `java.io.IOException`

    - #### getJsonData

          public java.lang.String getJsonData(java.lang.String url,
                                              Content content,
                                              Metadata metadata,
                                              ParseData parseData)
                                       throws java.io.IOException

      Description copied from interface: `CommonCrawlFormat`
      Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Returns:
      Throws:
        `java.io.IOException`

    - #### getJsonData

          public java.lang.String getJsonData()
                                       throws java.io.IOException

      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### writeKeyValue

          protected abstract void writeKeyValue(java.lang.String key,
                                                java.lang.String value)
                                         throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### writeKeyNull

          protected abstract void writeKeyNull(java.lang.String key)
                                        throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### startArray

          protected abstract void startArray(java.lang.String key,
                                             boolean nested,
                                             boolean newline)
                                      throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### closeArray

          protected abstract void closeArray(java.lang.String key,
                                             boolean nested,
                                             boolean newline)
                                      throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### writeArrayValue

          protected abstract void writeArrayValue(java.lang.String value)
                                           throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### startObject

          protected abstract void startObject(java.lang.String key)
                                       throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### closeObject

          protected abstract void closeObject(java.lang.String key)
                                       throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### generateJson

          protected abstract java.lang.String generateJson()
                                                    throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### getUrl

          protected java.lang.String getUrl()

    - #### getTimestamp

          protected java.lang.String getTimestamp()

    - #### getMethod

          protected java.lang.String getMethod()

    - #### getRequestHostName

          protected java.lang.String getRequestHostName()

    - #### getRequestHostAddress

          protected java.lang.String getRequestHostAddress()

    - #### getRequestSoftware

          protected java.lang.String getRequestSoftware()

    - #### getRequestRobots

          protected java.lang.String getRequestRobots()

    - #### getRequestContactName

          protected java.lang.String getRequestContactName()

    - #### getRequestContactEmail

          protected java.lang.String getRequestContactEmail()

    - #### getRequestAccept

          protected java.lang.String getRequestAccept()

    - #### getRequestAcceptEncoding

          protected java.lang.String getRequestAcceptEncoding()

    - #### getRequestAcceptLanguage

          protected java.lang.String getRequestAcceptLanguage()

    - #### getRequestUserAgent

          protected java.lang.String getRequestUserAgent()

    - #### getResponseStatus

          protected java.lang.String getResponseStatus()

    - #### getResponseHostName

          protected java.lang.String getResponseHostName()

    - #### getResponseAddress

          protected java.lang.String getResponseAddress()

    - #### getResponseContentEncoding

          protected java.lang.String getResponseContentEncoding()

    - #### getResponseContentType

          protected java.lang.String getResponseContentType()

    - #### getInLinks

          public java.util.List<java.lang.String> getInLinks()

      Description copied from interface: `CommonCrawlFormat`
      gets set of inlinks
      Specified by:
        `getInLinks` in interface `CommonCrawlFormat`
      Returns:
        gets inlinks of this document

    - #### setInLinks

          public void setInLinks(java.util.List<java.lang.String> inLinks)

      Description copied from interface: `CommonCrawlFormat`
      sets inlinks of this document
      Specified by:
        `setInLinks` in interface `CommonCrawlFormat`
      Parameters:
        `inLinks` - list of inlinks

    - #### getResponseDate

          protected java.lang.String getResponseDate()

    - #### getResponseServer

          protected java.lang.String getResponseServer()

    - #### getResponseContent

          protected java.lang.String getResponseContent()

    - #### getKey

          protected java.lang.String getKey()

    - #### getImported

          protected java.lang.String getImported()

    - #### close

          public void close()

      Description copied from interface: `CommonCrawlFormat`
      Optional method that could be implemented if the actual format needs some close procedure.
      Specified by:
        `close` in interface `java.io.Closeable`
      Specified by:
        `close` in interface `java.lang.AutoCloseable`
      Specified by:
        `close` in interface `CommonCrawlFormat`

Copyright © 2019 The Apache Software Foundation