org.apache.nutch.crawl

## Class CrawlDatum

- java.lang.Object
- - org.apache.nutch.crawl.CrawlDatum

- All Implemented Interfaces:
    Cloneable, Comparable<CrawlDatum>, Writable, WritableComparable<CrawlDatum>

      public class CrawlDatum
      extends Object
      implements WritableComparable<CrawlDatum>, Cloneable

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `CrawlDatum.Comparator` A Comparator optimized for CrawlDatum.

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `FETCH_DIR_NAME`
    `static String` | `GENERATE_DIR_NAME`
    `static String` | `PARSE_DIR_NAME`
    `static HashMap<Byte,String>` | `statNames`
    `static byte` | `STATUS_DB_DUPLICATE` Page was marked as being a duplicate of another page
    `static byte` | `STATUS_DB_FETCHED` Page was successfully fetched.
    `static byte` | `STATUS_DB_GONE` Page no longer exists.
    `static byte` | `STATUS_DB_MAX` Maximum value of DB-related status.
    `static byte` | `STATUS_DB_NOTMODIFIED` Page was successfully fetched and found not modified.
    `static byte` | `STATUS_DB_ORPHAN` Page was marked as orphan, e.g.
    `static byte` | `STATUS_DB_REDIR_PERM` Page permanently redirects to other page.
    `static byte` | `STATUS_DB_REDIR_TEMP` Page temporarily redirects to other page.
    `static byte` | `STATUS_DB_UNFETCHED` Page was not fetched yet.
    `static byte` | `STATUS_FETCH_GONE` Fetching unsuccessful - page is gone.
    `static byte` | `STATUS_FETCH_MAX` Maximum value of fetch-related status.
    `static byte` | `STATUS_FETCH_NOTMODIFIED` Fetching successful - page is not modified.
    `static byte` | `STATUS_FETCH_REDIR_PERM` Fetching permanently redirected to other page.
    `static byte` | `STATUS_FETCH_REDIR_TEMP` Fetching temporarily redirected to other page.
    `static byte` | `STATUS_FETCH_RETRY` Fetching unsuccessful, needs to be retried (transient errors).
    `static byte` | `STATUS_FETCH_SUCCESS` Fetching was successful.
    `static byte` | `STATUS_INJECTED` Page was newly injected.
    `static byte` | `STATUS_LINKED` Page discovered through a link.
    `static byte` | `STATUS_PARSE_META` Page got metadata from a parser
    `static byte` | `STATUS_SIGNATURE` Page signature.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CrawlDatum()`
    `CrawlDatum(int status, int fetchInterval)`
    `CrawlDatum(int status, int fetchInterval, float score)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Object` | `clone()`
    `int` | `compareTo(CrawlDatum that)` Sort by decreasing score.
    `boolean` | `equals(Object o)`
    `boolean` | `evaluate(org.apache.commons.jexl3.JexlExpression expr, String url)`
    `int` | `getFetchInterval()`
    `long` | `getFetchTime()` Returns either the time of the last fetch, or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.
    `MapWritable` | `getMetaData()` returns a MapWritable if it was set or read in @see readFields(DataInput), returns empty map in case CrawlDatum was freshly created (lazily instantiated).
    `long` | `getModifiedTime()`
    `byte` | `getRetriesSinceFetch()`
    `float` | `getScore()`
    `byte[]` | `getSignature()`
    `byte` | `getStatus()`
    `static byte` | `getStatusByName(String name)`
    `static String` | `getStatusName(byte value)`
    `static boolean` | `hasDbStatus(CrawlDatum datum)`
    `static boolean` | `hasFetchStatus(CrawlDatum datum)`
    `int` | `hashCode()`
    `void` | `putAllMetaData(CrawlDatum other)` Add all metadata from other CrawlDatum to this CrawlDatum.
    `static CrawlDatum` | `read(DataInput in)`
    `void` | `readFields(DataInput in)`
    `void` | `set(CrawlDatum that)` Copy the contents of another instance into this instance.
    `void` | `setFetchInterval(float fetchInterval)`
    `void` | `setFetchInterval(int fetchInterval)`
    `void` | `setFetchTime(long fetchTime)` Sets either the time of the last fetch or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.
    `void` | `setMetaData(MapWritable mapWritable)`
    `void` | `setModifiedTime(long modifiedTime)`
    `void` | `setRetriesSinceFetch(int retries)`
    `void` | `setScore(float score)`
    `void` | `setSignature(byte[] signature)`
    `void` | `setStatus(int status)`
    `String` | `toString()`
    `void` | `write(DataOutput out)`

    - ### Methods inherited from class java.lang.Object

      `finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### GENERATE_DIR_NAME

          public static final String GENERATE_DIR_NAME

      See Also:
        Constant Field Values

    - #### FETCH_DIR_NAME

          public static final String FETCH_DIR_NAME

      See Also:
        Constant Field Values

    - #### PARSE_DIR_NAME

          public static final String PARSE_DIR_NAME

      See Also:
        Constant Field Values

    - #### STATUS_DB_UNFETCHED

          public static final byte STATUS_DB_UNFETCHED

      Page was not fetched yet.
      See Also:
        Constant Field Values

    - #### STATUS_DB_FETCHED

          public static final byte STATUS_DB_FETCHED

      Page was successfully fetched.
      See Also:
        Constant Field Values

    - #### STATUS_DB_GONE

          public static final byte STATUS_DB_GONE

      Page no longer exists.
      See Also:
        Constant Field Values

    - #### STATUS_DB_REDIR_TEMP

          public static final byte STATUS_DB_REDIR_TEMP

      Page temporarily redirects to other page.
      See Also:
        Constant Field Values

    - #### STATUS_DB_REDIR_PERM

          public static final byte STATUS_DB_REDIR_PERM

      Page permanently redirects to other page.
      See Also:
        Constant Field Values

    - #### STATUS_DB_NOTMODIFIED

          public static final byte STATUS_DB_NOTMODIFIED

      Page was successfully fetched and found not modified.
      See Also:
        Constant Field Values

    - #### STATUS_DB_DUPLICATE

          public static final byte STATUS_DB_DUPLICATE

      Page was marked as being a duplicate of another page
      See Also:
        Constant Field Values

    - #### STATUS_DB_ORPHAN

          public static final byte STATUS_DB_ORPHAN

      Page was marked as orphan, e.g. has no inlinks anymore
      See Also:
        Constant Field Values

    - #### STATUS_DB_MAX

          public static final byte STATUS_DB_MAX

      Maximum value of DB-related status.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_SUCCESS

          public static final byte STATUS_FETCH_SUCCESS

      Fetching was successful.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_RETRY

          public static final byte STATUS_FETCH_RETRY

      Fetching unsuccessful, needs to be retried (transient errors).
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_REDIR_TEMP

          public static final byte STATUS_FETCH_REDIR_TEMP

      Fetching temporarily redirected to other page.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_REDIR_PERM

          public static final byte STATUS_FETCH_REDIR_PERM

      Fetching permanently redirected to other page.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_GONE

          public static final byte STATUS_FETCH_GONE

      Fetching unsuccessful - page is gone.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_NOTMODIFIED

          public static final byte STATUS_FETCH_NOTMODIFIED

      Fetching successful - page is not modified.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_MAX

          public static final byte STATUS_FETCH_MAX

      Maximum value of fetch-related status.
      See Also:
        Constant Field Values

    - #### STATUS_SIGNATURE

          public static final byte STATUS_SIGNATURE

      Page signature.
      See Also:
        Constant Field Values

    - #### STATUS_INJECTED

          public static final byte STATUS_INJECTED

      Page was newly injected.
      See Also:
        Constant Field Values

    - #### STATUS_LINKED

          public static final byte STATUS_LINKED

      Page discovered through a link.
      See Also:
        Constant Field Values

    - #### STATUS_PARSE_META

          public static final byte STATUS_PARSE_META

      Page got metadata from a parser
      See Also:
        Constant Field Values

    - #### statNames

          public static final HashMap<Byte,String> statNames

  - ### Constructor Detail

    - #### CrawlDatum

          public CrawlDatum()

    - #### CrawlDatum

          public CrawlDatum(int status,
                            int fetchInterval)

    - #### CrawlDatum

          public CrawlDatum(int status,
                            int fetchInterval,
                            float score)

  - ### Method Detail

    - #### hasDbStatus

          public static boolean hasDbStatus(CrawlDatum datum)

    - #### hasFetchStatus

          public static boolean hasFetchStatus(CrawlDatum datum)

    - #### getStatus

          public byte getStatus()

    - #### getStatusName

          public static String getStatusName(byte value)

    - #### getStatusByName

          public static byte getStatusByName(String name)

    - #### setStatus

          public void setStatus(int status)

    - #### getFetchTime

          public long getFetchTime()

      Returns either the time of the last fetch, or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.

    - #### setFetchTime

          public void setFetchTime(long fetchTime)

      Sets either the time of the last fetch or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.

    - #### getModifiedTime

          public long getModifiedTime()

    - #### setModifiedTime

          public void setModifiedTime(long modifiedTime)

    - #### getRetriesSinceFetch

          public byte getRetriesSinceFetch()

    - #### setRetriesSinceFetch

          public void setRetriesSinceFetch(int retries)

    - #### getFetchInterval

          public int getFetchInterval()

    - #### setFetchInterval

          public void setFetchInterval(int fetchInterval)

    - #### setFetchInterval

          public void setFetchInterval(float fetchInterval)

    - #### getScore

          public float getScore()

    - #### setScore

          public void setScore(float score)

    - #### getSignature

          public byte[] getSignature()

    - #### setSignature

          public void setSignature(byte[] signature)

    - #### setMetaData

          public void setMetaData(MapWritable mapWritable)

    - #### putAllMetaData

          public void putAllMetaData(CrawlDatum other)

      Add all metadata from other CrawlDatum to this CrawlDatum.
      Parameters:
        `other` - CrawlDatum

    - #### getMetaData

          public MapWritable getMetaData()

      returns a MapWritable if it was set or read in @see readFields(DataInput), returns empty map in case CrawlDatum was freshly created (lazily instantiated).

    - #### read

          public static CrawlDatum read(DataInput in)
                                 throws IOException

      Throws:
        `IOException`

    - #### readFields

          public void readFields(DataInput in)
                          throws IOException

      Specified by:
        `readFields` in interface `Writable`
      Throws:
        `IOException`

    - #### write

          public void write(DataOutput out)
                     throws IOException

      Specified by:
        `write` in interface `Writable`
      Throws:
        `IOException`

    - #### set

          public void set(CrawlDatum that)

      Copy the contents of another instance into this instance.

    - #### compareTo

          public int compareTo(CrawlDatum that)

      Sort by decreasing score.
      Specified by:
        `compareTo` in interface `Comparable<CrawlDatum>`

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `Object`

    - #### equals

          public boolean equals(Object o)

      Overrides:
        `equals` in class `Object`

    - #### hashCode

          public int hashCode()

      Overrides:
        `hashCode` in class `Object`

    - #### clone

          public Object clone()

      Overrides:
        `clone` in class `Object`

    - #### evaluate

          public boolean evaluate(org.apache.commons.jexl3.JexlExpression expr,
                                  String url)

Copyright © 2021 The Apache Software Foundation