org.apache.nutch.tools

## Class WARCUtils

- java.lang.Object
- - org.apache.nutch.tools.WARCUtils

-     public class WARCUtils
      extends java.lang.Object

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static java.lang.String` | `COLONSP`
    `static java.lang.String` | `CONFORMS_TO`
    `static java.lang.String` | `CRLF`
    `static java.lang.String` | `FORMAT`
    `static org.archive.uid.UUIDGenerator` | `generator`
    `static java.lang.String` | `HOSTNAME`
    `static java.lang.String` | `HTTP_HEADER_FROM`
    `static java.lang.String` | `HTTP_HEADER_USER_AGENT`
    `static java.lang.String` | `IP`
    `static java.lang.String` | `OPERATOR`
    `protected static java.util.regex.Pattern` | `PROBLEMATIC_HEADERS`
    `static java.lang.String` | `ROBOTS`
    `static java.lang.String` | `SOFTWARE`
    `protected static java.lang.String` | `X_HIDE_HEADER`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `WARCUtils()`

  - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static org.archive.io.warc.WARCRecordInfo` | `docToMetadata(NutchDocument doc)`
    `static java.lang.String` | `fixHttpHeaders(java.lang.String headers, int contentLength)` Modify verbatim HTTP response headers: fix, remove or replace headers `Content-Length`, `Content-Encoding` and `Transfer-Encoding` which may confuse WARC readers.
    `static java.lang.String` | `getAgentString(java.lang.String name, java.lang.String version, java.lang.String description, java.lang.String URL, java.lang.String email)`
    `static java.lang.String` | `getHostname(Configuration conf)`
    `static java.lang.String` | `getIPAddress(Configuration conf)`
    `static org.archive.util.anvl.ANVLRecord` | `getWARCInfoContent(Configuration conf)`
    `static byte[]` | `toByteArray(org.archive.format.http.HttpHeaders headers)`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### SOFTWARE

          public static final java.lang.String SOFTWARE

      See Also:
        Constant Field Values

    - #### HTTP_HEADER_FROM

          public static final java.lang.String HTTP_HEADER_FROM

      See Also:
        Constant Field Values

    - #### HTTP_HEADER_USER_AGENT

          public static final java.lang.String HTTP_HEADER_USER_AGENT

      See Also:
        Constant Field Values

    - #### HOSTNAME

          public static final java.lang.String HOSTNAME

      See Also:
        Constant Field Values

    - #### ROBOTS

          public static final java.lang.String ROBOTS

      See Also:
        Constant Field Values

    - #### OPERATOR

          public static final java.lang.String OPERATOR

      See Also:
        Constant Field Values

    - #### FORMAT

          public static final java.lang.String FORMAT

      See Also:
        Constant Field Values

    - #### CONFORMS_TO

          public static final java.lang.String CONFORMS_TO

      See Also:
        Constant Field Values

    - #### IP

          public static final java.lang.String IP

      See Also:
        Constant Field Values

    - #### generator

          public static final org.archive.uid.UUIDGenerator generator

    - #### CRLF

          public static final java.lang.String CRLF

      See Also:
        Constant Field Values

    - #### COLONSP

          public static final java.lang.String COLONSP

      See Also:
        Constant Field Values

    - #### PROBLEMATIC_HEADERS

          protected static final java.util.regex.Pattern PROBLEMATIC_HEADERS

    - #### X_HIDE_HEADER

          protected static final java.lang.String X_HIDE_HEADER

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### WARCUtils

          public WARCUtils()

  - ### Method Detail

    - #### getWARCInfoContent

          public static final org.archive.util.anvl.ANVLRecord getWARCInfoContent(Configuration conf)

    - #### getHostname

          public static final java.lang.String getHostname(Configuration conf)
                                                    throws java.net.UnknownHostException

      Throws:
        `java.net.UnknownHostException`

    - #### getIPAddress

          public static final java.lang.String getIPAddress(Configuration conf)
                                                     throws java.net.UnknownHostException

      Throws:
        `java.net.UnknownHostException`

    - #### toByteArray

          public static final byte[] toByteArray(org.archive.format.http.HttpHeaders headers)
                                          throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### getAgentString

          public static final java.lang.String getAgentString(java.lang.String name,
                                                              java.lang.String version,
                                                              java.lang.String description,
                                                              java.lang.String URL,
                                                              java.lang.String email)

    - #### docToMetadata

          public static final org.archive.io.warc.WARCRecordInfo docToMetadata(NutchDocument doc)
                                                                        throws java.io.UnsupportedEncodingException

      Throws:
        `java.io.UnsupportedEncodingException`

    - #### fixHttpHeaders

          public static final java.lang.String fixHttpHeaders(java.lang.String headers,
                                                              int contentLength)

      Modify verbatim HTTP response headers: fix, remove or replace headers `Content-Length`, `Content-Encoding` and `Transfer-Encoding` which may confuse WARC readers. Ensure that returned header end with a single empty line (`\r\n\r\n`).
      Parameters:
        `headers` - HTTP 1.1 or 1.0 response header string, CR-LF-separated lines, first line is status line
      Returns:
        safe HTTP response header

Copyright © 2019 The Apache Software Foundation