org.apache.nutch.tools

## Class CommonCrawlFormatWARC

- java.lang.Object
- - org.apache.nutch.tools.AbstractCommonCrawlFormat
  - - org.apache.nutch.tools.CommonCrawlFormatWARC

- All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, CommonCrawlFormat

      public class CommonCrawlFormatWARC
      extends AbstractCommonCrawlFormat

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static java.lang.String` | `MAX_WARC_FILE_SIZE`
    `static java.lang.String` | `TEMPLATE`

    - ### Fields inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

      `conf, content, inLinks, jsonArray, keyPrefix, LOG, metadata, reverseKey, reverseKeyValue, simpleDateFormat, url`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CommonCrawlFormatWARC(Configuration nutchConf, CommonCrawlConfig config)`
    `CommonCrawlFormatWARC(java.lang.String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config, ParseData parseData)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()` Optional method that could be implemented if the actual format needs some close procedure.
    `protected void` | `closeArray(java.lang.String key, boolean nested, boolean newline)`
    `protected void` | `closeObject(java.lang.String key)`
    `protected java.lang.String` | `generateJson()`
    `java.lang.String` | `getJsonData()`
    `java.lang.String` | `getJsonData(java.lang.String url, Content content, Metadata metadata, ParseData parseData)` Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
    `protected void` | `startArray(java.lang.String key, boolean nested, boolean newline)`
    `protected void` | `startObject(java.lang.String key)`
    `protected void` | `writeArrayValue(java.lang.String value)`
    `protected void` | `writeKeyNull(java.lang.String key)`
    `protected void` | `writeKeyValue(java.lang.String key, java.lang.String value)`
    `protected java.net.URI` | `writeRequest(java.net.URI id)`
    `protected java.net.URI` | `writeResponse()`

    - ### Methods inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

      `getImported, getInLinks, getJsonData, getKey, getMethod, getRequestAccept, getRequestAcceptEncoding, getRequestAcceptLanguage, getRequestContactEmail, getRequestContactName, getRequestHostAddress, getRequestHostName, getRequestRobots, getRequestSoftware, getRequestUserAgent, getResponseAddress, getResponseContent, getResponseContentEncoding, getResponseContentType, getResponseDate, getResponseHostName, getResponseServer, getResponseStatus, getTimestamp, getUrl, setInLinks`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### MAX_WARC_FILE_SIZE

          public static final java.lang.String MAX_WARC_FILE_SIZE

      See Also:
        Constant Field Values

    - #### TEMPLATE

          public static final java.lang.String TEMPLATE

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### CommonCrawlFormatWARC

          public CommonCrawlFormatWARC(Configuration nutchConf,
                                       CommonCrawlConfig config)
                                throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### CommonCrawlFormatWARC

          public CommonCrawlFormatWARC(java.lang.String url,
                                       Content content,
                                       Metadata metadata,
                                       Configuration nutchConf,
                                       CommonCrawlConfig config,
                                       ParseData parseData)
                                throws java.io.IOException

      Throws:
        `java.io.IOException`

  - ### Method Detail

    - #### getJsonData

          public java.lang.String getJsonData(java.lang.String url,
                                              Content content,
                                              Metadata metadata,
                                              ParseData parseData)
                                       throws java.io.IOException

      Description copied from interface: `CommonCrawlFormat`
      Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Overrides:
        `getJsonData` in class `AbstractCommonCrawlFormat`
      Returns:
      Throws:
        `java.io.IOException`

    - #### getJsonData

          public java.lang.String getJsonData()
                                       throws java.io.IOException

      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Overrides:
        `getJsonData` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### writeResponse

          protected java.net.URI writeResponse()
                                        throws java.io.IOException,
                                               java.text.ParseException

      Throws:
        `java.io.IOException`
        `java.text.ParseException`

    - #### writeRequest

          protected java.net.URI writeRequest(java.net.URI id)
                                       throws java.io.IOException,
                                              java.text.ParseException

      Throws:
        `java.io.IOException`
        `java.text.ParseException`

    - #### generateJson

          protected java.lang.String generateJson()
                                           throws java.io.IOException

      Specified by:
        `generateJson` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### writeKeyValue

          protected void writeKeyValue(java.lang.String key,
                                       java.lang.String value)
                                throws java.io.IOException

      Specified by:
        `writeKeyValue` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### writeKeyNull

          protected void writeKeyNull(java.lang.String key)
                               throws java.io.IOException

      Specified by:
        `writeKeyNull` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### startArray

          protected void startArray(java.lang.String key,
                                    boolean nested,
                                    boolean newline)
                             throws java.io.IOException

      Specified by:
        `startArray` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### closeArray

          protected void closeArray(java.lang.String key,
                                    boolean nested,
                                    boolean newline)
                             throws java.io.IOException

      Specified by:
        `closeArray` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### writeArrayValue

          protected void writeArrayValue(java.lang.String value)
                                  throws java.io.IOException

      Specified by:
        `writeArrayValue` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### startObject

          protected void startObject(java.lang.String key)
                              throws java.io.IOException

      Specified by:
        `startObject` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### closeObject

          protected void closeObject(java.lang.String key)
                              throws java.io.IOException

      Specified by:
        `closeObject` in class `AbstractCommonCrawlFormat`
      Throws:
        `java.io.IOException`

    - #### close

          public void close()

      Description copied from interface: `CommonCrawlFormat`
      Optional method that could be implemented if the actual format needs some close procedure.
      Specified by:
        `close` in interface `java.io.Closeable`
      Specified by:
        `close` in interface `java.lang.AutoCloseable`
      Specified by:
        `close` in interface `CommonCrawlFormat`
      Overrides:
        `close` in class `AbstractCommonCrawlFormat`

Copyright © 2019 The Apache Software Foundation