org.apache.nutch.tools

## Class CommonCrawlFormatWARC

- java.lang.Object
- - org.apache.nutch.tools.AbstractCommonCrawlFormat
  - - org.apache.nutch.tools.CommonCrawlFormatWARC

- All Implemented Interfaces:
    Closeable, AutoCloseable, CommonCrawlFormat

      public class CommonCrawlFormatWARC
      extends AbstractCommonCrawlFormat

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `MAX_WARC_FILE_SIZE`
    `static String` | `TEMPLATE`

    - ### Fields inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

      `conf, content, inLinks, jsonArray, keyPrefix, LOG, metadata, reverseKey, reverseKeyValue, simpleDateFormat, url`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CommonCrawlFormatWARC(Configuration nutchConf, CommonCrawlConfig config)`
    `CommonCrawlFormatWARC(String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config, ParseData parseData)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()` Optional method that could be implemented if the actual format needs some close procedure.
    `protected void` | `closeArray(String key, boolean nested, boolean newline)`
    `protected void` | `closeObject(String key)`
    `protected String` | `generateJson()`
    `String` | `getJsonData()`
    `String` | `getJsonData(String url, Content content, Metadata metadata, ParseData parseData)` Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
    `protected void` | `startArray(String key, boolean nested, boolean newline)`
    `protected void` | `startObject(String key)`
    `protected void` | `writeArrayValue(String value)`
    `protected void` | `writeKeyNull(String key)`
    `protected void` | `writeKeyValue(String key, String value)`
    `protected URI` | `writeRequest(URI id)`
    `protected URI` | `writeResponse()`

    - ### Methods inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

      `getImported, getInLinks, getJsonData, getKey, getMethod, getRequestAccept, getRequestAcceptEncoding, getRequestAcceptLanguage, getRequestContactEmail, getRequestContactName, getRequestHostAddress, getRequestHostName, getRequestRobots, getRequestSoftware, getRequestUserAgent, getResponseAddress, getResponseContent, getResponseContentEncoding, getResponseContentType, getResponseDate, getResponseHostName, getResponseServer, getResponseStatus, getTimestamp, getUrl, setInLinks`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### MAX_WARC_FILE_SIZE

          public static final String MAX_WARC_FILE_SIZE

      See Also:
        Constant Field Values

    - #### TEMPLATE

          public static final String TEMPLATE

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### CommonCrawlFormatWARC

          public CommonCrawlFormatWARC(Configuration nutchConf,
                                       CommonCrawlConfig config)
                                throws IOException

      Throws:
        `IOException`

    - #### CommonCrawlFormatWARC

          public CommonCrawlFormatWARC(String url,
                                       Content content,
                                       Metadata metadata,
                                       Configuration nutchConf,
                                       CommonCrawlConfig config,
                                       ParseData parseData)
                                throws IOException

      Throws:
        `IOException`

  - ### Method Detail

    - #### getJsonData

          public String getJsonData(String url,
                                    Content content,
                                    Metadata metadata,
                                    ParseData parseData)
                             throws IOException

      Description copied from interface: `CommonCrawlFormat`
      Returns a string representation of the JSON structure of the URL content takes into account the parsed metadata about the URL
      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Overrides:
        `getJsonData` in class `AbstractCommonCrawlFormat`
      Returns:
      Throws:
        `IOException`

    - #### getJsonData

          public String getJsonData()
                             throws IOException

      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Overrides:
        `getJsonData` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeResponse

          protected URI writeResponse()
                               throws IOException,
                                      ParseException

      Throws:
        `IOException`
        `ParseException`

    - #### writeRequest

          protected URI writeRequest(URI id)
                              throws IOException,
                                     ParseException

      Throws:
        `IOException`
        `ParseException`

    - #### generateJson

          protected String generateJson()
                                 throws IOException

      Specified by:
        `generateJson` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeKeyValue

          protected void writeKeyValue(String key,
                                       String value)
                                throws IOException

      Specified by:
        `writeKeyValue` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeKeyNull

          protected void writeKeyNull(String key)
                               throws IOException

      Specified by:
        `writeKeyNull` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### startArray

          protected void startArray(String key,
                                    boolean nested,
                                    boolean newline)
                             throws IOException

      Specified by:
        `startArray` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### closeArray

          protected void closeArray(String key,
                                    boolean nested,
                                    boolean newline)
                             throws IOException

      Specified by:
        `closeArray` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeArrayValue

          protected void writeArrayValue(String value)
                                  throws IOException

      Specified by:
        `writeArrayValue` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### startObject

          protected void startObject(String key)
                              throws IOException

      Specified by:
        `startObject` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### closeObject

          protected void closeObject(String key)
                              throws IOException

      Specified by:
        `closeObject` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### close

          public void close()

      Description copied from interface: `CommonCrawlFormat`
      Optional method that could be implemented if the actual format needs some close procedure.
      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in interface `CommonCrawlFormat`
      Overrides:
        `close` in class `AbstractCommonCrawlFormat`

Copyright © 2021 The Apache Software Foundation