org.apache.nutch.crawl

## Class AdaptiveFetchSchedule

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.crawl.AbstractFetchSchedule
    - - org.apache.nutch.crawl.AdaptiveFetchSchedule

- All Implemented Interfaces:
    Configurable, FetchSchedule

      public class AdaptiveFetchSchedule
      extends AbstractFetchSchedule

  This class implements an adaptive re-fetch algorithm. This works as follows:
  - for pages that has changed since the last fetchTime, decrease their fetchInterval by a factor of DEC_FACTOR (default value is 0.2f).
  - for pages that haven't changed since the last fetchTime, increase their fetchInterval by a factor of INC_FACTOR (default value is 0.2f).
    If SYNC_DELTA property is true, then:
    - calculate a `delta = fetchTime - modifiedTime`
    - try to synchronize with the time of change, by shifting the next fetchTime by a fraction of the difference between the last modification time and the last fetch time. I.e. the next fetch time will be set to `fetchTime + fetchInterval - delta * SYNC_DELTA_RATE`
    - if the adjusted fetch interval is bigger than the delta, then `fetchInterval = delta`.
  - the minimum value of fetchInterval may not be smaller than MIN_INTERVAL (default is 1 minute).
  - the maximum value of fetchInterval may not be bigger than MAX_INTERVAL (default is 365 days).

  NOTE: values of DEC_FACTOR and INC_FACTOR higher than 0.4f may destabilize the algorithm, so that the fetch interval either increases or decreases infinitely, with little relevance to the page changes.

  Author:
    Andrzej Bialecki

- - ### Field Summary

    - ### Fields inherited from class org.apache.nutch.crawl.AbstractFetchSchedule

      `defaultInterval, maxInterval`

    - ### Fields inherited from interface org.apache.nutch.crawl.FetchSchedule

      `SECONDS_PER_DAY, STATUS_MODIFIED, STATUS_NOTMODIFIED, STATUS_UNKNOWN`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `AdaptiveFetchSchedule()`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `setConf(Configuration conf)`
    `void` | `setFetchSchedule(java.lang.String url, WebPage page, long prevFetchTime, long prevModifiedTime, long fetchTime, long modifiedTime, int state)` Sets the `fetchInterval` and `fetchTime` on a successfully fetched page.

    - ### Methods inherited from class org.apache.nutch.crawl.AbstractFetchSchedule

      `calculateLastFetchTime, forceRefetch, getFields, initializeSchedule, setPageGoneSchedule, setPageRetrySchedule, shouldFetch`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf`

- - ### Constructor Detail

    - #### AdaptiveFetchSchedule

          public AdaptiveFetchSchedule()

  - ### Method Detail

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`
      Overrides:
        `setConf` in class `AbstractFetchSchedule`

    - #### setFetchSchedule

          public void setFetchSchedule(java.lang.String url,
                                       WebPage page,
                                       long prevFetchTime,
                                       long prevModifiedTime,
                                       long fetchTime,
                                       long modifiedTime,
                                       int state)

      Description copied from class: `AbstractFetchSchedule`
      Sets the `fetchInterval` and `fetchTime` on a successfully fetched page. NOTE: this implementation resets the retry counter - extending classes should call super.setFetchSchedule() to preserve this behavior.
      Specified by:
        `setFetchSchedule` in interface `FetchSchedule`
      Overrides:
        `setFetchSchedule` in class `AbstractFetchSchedule`
      Parameters:
        `url` - url of the page
        `page` - `WebPage` object relative to the URL
        `prevFetchTime` - previous value of fetch time, or -1 if not available
        `prevModifiedTime` - previous value of modifiedTime, or -1 if not available
        `fetchTime` - the latest time, when the page was recently re-fetched.
        `modifiedTime` - last time the content was modified. This information comes from the protocol implementations, or is set to < 0 if not available.
        `state` - if `FetchSchedule.STATUS_MODIFIED`, then the content is considered to be "changed" before the `fetchTime`, if `FetchSchedule.STATUS_NOTMODIFIED` then the content is known to be unchanged. This information may be obtained by comparing page signatures before and after fetching. If this is set to `FetchSchedule.STATUS_UNKNOWN`, then it is unknown whether the page was changed; implementations are free to follow a sensible default behavior.

Copyright © 2019 The Apache Software Foundation