org.apache.nutch.crawl

## Interface FetchSchedule

- All Superinterfaces:
    Configurable
  All Known Implementing Classes:
    AbstractFetchSchedule, AdaptiveFetchSchedule, DefaultFetchSchedule

      public interface FetchSchedule
      extends Configurable

  This interface defines the contract for implementations that manipulate fetch times and re-fetch intervals.
  Author:
    Andrzej Bialecki

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `SECONDS_PER_DAY`
    `static int` | `STATUS_MODIFIED` Page is known to have been modified since our last visit.
    `static int` | `STATUS_NOTMODIFIED` Page is known to remain unmodified since our last visit.
    `static int` | `STATUS_UNKNOWN` It is unknown whether page was changed since our last visit.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `long` | `calculateLastFetchTime(WebPage page)` Calculates last fetch time of the given CrawlDatum.
    `void` | `forceRefetch(String url, WebPage row, boolean asap)` This method resets fetchTime, fetchInterval, modifiedTime and page signature, so that it forces refetching.
    `Collection<WebPage.Field>` | `getFields()`
    `void` | `initializeSchedule(String url, WebPage page)` Initialize fetch schedule related data.
    `void` | `setFetchSchedule(String url, WebPage page, long prevFetchTime, long prevModifiedTime, long fetchTime, long modifiedTime, int state)` Sets the `fetchInterval` and `fetchTime` on a successfully fetched page.
    `void` | `setPageGoneSchedule(String url, WebPage page, long prevFetchTime, long prevModifiedTime, long fetchTime)` This method specifies how to schedule refetching of pages marked as GONE.
    `void` | `setPageRetrySchedule(String url, WebPage page, long prevFetchTime, long prevModifiedTime, long fetchTime)` This method adjusts the fetch schedule if fetching needs to be re-tried due to transient errors.
    `boolean` | `shouldFetch(String url, WebPage page, long curTime)` This method provides information whether the page is suitable for selection in the current fetchlist.

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Field Detail

    - #### STATUS_UNKNOWN

          static final int STATUS_UNKNOWN

      It is unknown whether page was changed since our last visit.
      See Also:
        Constant Field Values

    - #### STATUS_MODIFIED

          static final int STATUS_MODIFIED

      Page is known to have been modified since our last visit.
      See Also:
        Constant Field Values

    - #### STATUS_NOTMODIFIED

          static final int STATUS_NOTMODIFIED

      Page is known to remain unmodified since our last visit.
      See Also:
        Constant Field Values

    - #### SECONDS_PER_DAY

          static final int SECONDS_PER_DAY

      See Also:
        Constant Field Values

  - ### Method Detail

    - #### initializeSchedule

          void initializeSchedule(String url,
                                WebPage page)

      Initialize fetch schedule related data. Implementations should at least set the `fetchTime` and `fetchInterval`. The default implementation set the `fetchTime` to now, using the default `fetchInterval`.
      Parameters:
        `url` - URL of the page.
        `page` -

    - #### setFetchSchedule

          void setFetchSchedule(String url,
                              WebPage page,
                              long prevFetchTime,
                              long prevModifiedTime,
                              long fetchTime,
                              long modifiedTime,
                              int state)

      Sets the `fetchInterval` and `fetchTime` on a successfully fetched page. Implementations may use supplied arguments to support different re-fetching schedules.
      Parameters:
        `url` - url of the page
        `page` -
        `prevFetchTime` - previous value of fetch time, or -1 if not available
        `prevModifiedTime` - previous value of modifiedTime, or -1 if not available
        `fetchTime` - the latest time, when the page was recently re-fetched. Most FetchSchedule implementations should update the value in
        `datum` - to something greater than this value.
        `modifiedTime` - last time the content was modified. This information comes from the protocol implementations, or is set to < 0 if not available. Most FetchSchedule implementations should update the value in
        `datum` - to this value.
        `state` - if `STATUS_MODIFIED`, then the content is considered to be "changed" before the `fetchTime`, if `STATUS_NOTMODIFIED` then the content is known to be unchanged. This information may be obtained by comparing page signatures before and after fetching. If this is set to `STATUS_UNKNOWN`, then it is unknown whether the page was changed; implementations are free to follow a sensible default behavior.

    - #### setPageGoneSchedule

          void setPageGoneSchedule(String url,
                                 WebPage page,
                                 long prevFetchTime,
                                 long prevModifiedTime,
                                 long fetchTime)

      This method specifies how to schedule refetching of pages marked as GONE. Default implementation increases fetchInterval by 50%, and if it exceeds the `maxInterval` it calls `#forceRefetch(Text, CrawlDatum, boolean)`.
      Parameters:
        `url` - URL of the page
        `page` -

    - #### setPageRetrySchedule

          void setPageRetrySchedule(String url,
                                  WebPage page,
                                  long prevFetchTime,
                                  long prevModifiedTime,
                                  long fetchTime)

      This method adjusts the fetch schedule if fetching needs to be re-tried due to transient errors. The default implementation sets the next fetch time 1 day in the future and increases the retry counter.Set
      Parameters:
        `url` - URL of the page
        `page` -
        `prevFetchTime` - previous fetch time
        `prevModifiedTime` - previous modified time
        `fetchTime` - current fetch time

    - #### calculateLastFetchTime

          long calculateLastFetchTime(WebPage page)

      Calculates last fetch time of the given CrawlDatum.
      Returns:
        the date as a long.

    - #### shouldFetch

          boolean shouldFetch(String url,
                            WebPage page,
                            long curTime)

      This method provides information whether the page is suitable for selection in the current fetchlist. NOTE: a true return value does not guarantee that the page will be fetched, it just allows it to be included in the further selection process based on scores. The default implementation checks `fetchTime`, if it is higher than the
      Parameters:
        `curTime` - it returns false, and true otherwise. It will also check that fetchTime is not too remote (more than `maxInterval`
        `url` - URL of the page
        `row` - url's row
        `curTime` - reference time (usually set to the time when the fetchlist generation process was started).
      Returns:
        true, if the page should be considered for inclusion in the current fetchlist, otherwise false.

    - #### forceRefetch

          void forceRefetch(String url,
                          WebPage row,
                          boolean asap)

      This method resets fetchTime, fetchInterval, modifiedTime and page signature, so that it forces refetching.
      Parameters:
        `url` - URL of the page
        `page` -
        `asap` - if true, force refetch as soon as possible - this sets the fetchTime to now. If false, force refetch whenever the next fetch time is set.

    - #### getFields

          Collection<WebPage.Field> getFields()

Copyright © 2015 The Apache Software Foundation