Package org.apache.nutch.scoring.depth

## Class DepthScoringFilter

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.scoring.depth.DepthScoringFilter

- All Implemented Interfaces:
    `Configurable`, `Pluggable`, `ScoringFilter`

      public class DepthScoringFilter
      extends Configured
      implements ScoringFilter

  This scoring filter limits the number of hops from the initial seed urls. If the number of hops exceeds the depth (either the default value, or the one set in the injector file) then all outlinks from that url are discarded, effectively stopping further crawling along this path.

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static int` | `DEFAULT_MAX_DEPTH` |
    `static String` | `DEPTH_KEY` |
    `static Text` | `DEPTH_KEY_W` |
    `static String` | `MAX_DEPTH_KEY` |
    `static Text` | `MAX_DEPTH_KEY_W` |

    - ### Fields inherited from interface org.apache.nutch.scoring.ScoringFilter

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `DepthScoringFilter()` |

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `CrawlDatum` | `distributeScoreToOutlinks​(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,​CrawlDatum>> targets, CrawlDatum adjust, int allCount)` | Distribute score value from the current page to all its outlinked pages.
    `float` | `generatorSortValue​(Text url, CrawlDatum datum, float initSort)` | This method prepares a sort value for the purpose of sorting and selecting top N scoring pages during fetchlist generation.
    `float` | `indexerScore​(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)` | This method calculates a indexed document score/boost.
    `void` | `initialScore​(Text url, CrawlDatum datum)` | Set an initial score for newly discovered pages.
    `void` | `injectedScore​(Text url, CrawlDatum datum)` | Set an initial score for newly injected pages.
    `void` | `passScoreAfterParsing​(Text url, Content content, Parse parse)` | Currently a part of score distribution is performed using only data coming from the parsing process.
    `void` | `passScoreBeforeParsing​(Text url, CrawlDatum datum, Content content)` | This method takes all relevant score information from the current datum (coming from a generated fetchlist) and stores it into `Content` metadata.
    `void` | `setConf​(Configuration conf)` |
    `void` | `updateDbScore​(Text url, CrawlDatum old, CrawlDatum datum, List<CrawlDatum> inlinked)` | This method calculates a new score of CrawlDatum during CrawlDb update, based on the initial value of the original CrawlDatum, and also score values contributed by inlinked pages.

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf`

    - ### Methods inherited from interface org.apache.nutch.scoring.ScoringFilter

      `orphanedScore`

- - ### Field Detail

    - #### DEPTH_KEY

          public static final String DEPTH_KEY

      See Also:
        Constant Field Values

    - #### DEPTH_KEY_W

          public static final Text DEPTH_KEY_W

    - #### MAX_DEPTH_KEY

          public static final String MAX_DEPTH_KEY

      See Also:
        Constant Field Values

    - #### MAX_DEPTH_KEY_W

          public static final Text MAX_DEPTH_KEY_W

    - #### DEFAULT_MAX_DEPTH

          public static final int DEFAULT_MAX_DEPTH

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### DepthScoringFilter

          public DepthScoringFilter()

  - ### Method Detail

    - #### setConf

          public void setConf​(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`
      Overrides:
        `setConf` in class `Configured`

    - #### distributeScoreToOutlinks

          public CrawlDatum distributeScoreToOutlinks​(Text fromUrl,
                                                      ParseData parseData,
                                                      Collection<Map.Entry<Text,​CrawlDatum>> targets,
                                                      CrawlDatum adjust,
                                                      int allCount)
                                               throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      Distribute score value from the current page to all its outlinked pages.
      Specified by:
        `distributeScoreToOutlinks` in interface `ScoringFilter`
      Parameters:
        `fromUrl` - url of the source page
        `parseData` - ParseData instance, which stores relevant score value(s) in its metadata. NOTE: filters may modify this in-place, all changes will be persisted.
        `targets` - <url, CrawlDatum> pairs. NOTE: filters can modify this in-place, all changes will be persisted.
        `adjust` - a CrawlDatum instance, initially null, which implementations may use to pass adjustment values to the original CrawlDatum. When creating this instance, set its status to `CrawlDatum.STATUS_LINKED`.
        `allCount` - number of all collected outlinks from the source page
      Returns:
        if needed, implementations may return an instance of CrawlDatum, with status `CrawlDatum.STATUS_LINKED`, which contains adjustments to be applied to the original CrawlDatum score(s) and metadata. This can be null if not needed.
      Throws:
        `ScoringFilterException` - there is a fatal error distributing score data from the current page to all of its outlinks

    - #### generatorSortValue

          public float generatorSortValue​(Text url,
                                          CrawlDatum datum,
                                          float initSort)
                                   throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      This method prepares a sort value for the purpose of sorting and selecting top N scoring pages during fetchlist generation.
      Specified by:
        `generatorSortValue` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - page's datum, should not be modified
        `initSort` - initial sort value, or a value from previous filters in chain
      Returns:
        a sort value for use in sorting and selecting the top N scoring pages during fetchlist generation
      Throws:
        `ScoringFilterException` - if there is a fatal error preparing the sort value

    - #### indexerScore

          public float indexerScore​(Text url,
                                    NutchDocument doc,
                                    CrawlDatum dbDatum,
                                    CrawlDatum fetchDatum,
                                    Parse parse,
                                    Inlinks inlinks,
                                    float initScore)
                             throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      This method calculates a indexed document score/boost.
      Specified by:
        `indexerScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `doc` - indexed document. NOTE: this already contains all information collected by indexing filters. Implementations may modify this instance, in order to store/remove some information.
        `dbDatum` - current page from CrawlDb. NOTE:
        - changes made to this instance are not persisted
        - may be null if indexing is done without CrawlDb or if the segment is generated not from the CrawlDb (via FreeGenerator).
        `fetchDatum` - datum from FetcherOutput (containing among others the fetching status)
        `parse` - parsing result. NOTE: changes made to this instance are not persisted.
        `inlinks` - current inlinks from LinkDb. NOTE: changes made to this instance are not persisted.
        `initScore` - initial boost value for the indexed document.
      Returns:
        boost value for the indexed document. This value is passed as an argument to the next scoring filter in chain. NOTE: implementations may also express other scoring strategies by modifying the indexed document directly.
      Throws:
        `ScoringFilterException` - if there is a fatal error whilst calculating the indexed document score/boost

    - #### initialScore

          public void initialScore​(Text url,
                                   CrawlDatum datum)
                            throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      Set an initial score for newly discovered pages. Note: newly discovered pages have at least one inlink with its score contribution, so filter implementations may choose to set initial score to zero (unknown value), and then the inlink score contribution will set the "real" value of the new page.
      Specified by:
        `initialScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - new datum. Filters will modify it in-place.
      Throws:
        `ScoringFilterException` - if there is a fatal error setting an initial score for newly discovered pages

    - #### injectedScore

          public void injectedScore​(Text url,
                                    CrawlDatum datum)
                             throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      Set an initial score for newly injected pages. Note: newly injected pages may have no inlinks, so filter implementations may wish to set this score to a non-zero value, to give newly injected pages some initial credit.
      Specified by:
        `injectedScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - new datum. Filters will modify it in-place.
      Throws:
        `ScoringFilterException` - if there is a fatal error setting an initial score for newly injected pages

    - #### passScoreAfterParsing

          public void passScoreAfterParsing​(Text url,
                                            Content content,
                                            Parse parse)
                                     throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      Currently a part of score distribution is performed using only data coming from the parsing process. We need this method in order to ensure the presence of score data in these steps.
      Specified by:
        `passScoreAfterParsing` in interface `ScoringFilter`
      Parameters:
        `url` - page url
        `content` - original content. NOTE: modifications to this value are not persisted.
        `parse` - target instance to copy the score information to. Implementations may modify this in-place, primarily by setting some metadata properties.
      Throws:
        `ScoringFilterException` - if there is a fatal error processing score data in subsequent steps after parsing

    - #### passScoreBeforeParsing

          public void passScoreBeforeParsing​(Text url,
                                             CrawlDatum datum,
                                             Content content)
                                      throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      This method takes all relevant score information from the current datum (coming from a generated fetchlist) and stores it into `Content` metadata. This is needed in order to pass this value(s) to the mechanism that distributes it to outlinked pages.
      Specified by:
        `passScoreBeforeParsing` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - source datum. NOTE: modifications to this value are not persisted.
        `content` - instance of content. Implementations may modify this in-place, primarily by setting some metadata properties.
      Throws:
        `ScoringFilterException` - if there is a fatal error injecting score information from the current datum into `Content` metadata

    - #### updateDbScore

          public void updateDbScore​(Text url,
                                    CrawlDatum old,
                                    CrawlDatum datum,
                                    List<CrawlDatum> inlinked)
                             throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      This method calculates a new score of CrawlDatum during CrawlDb update, based on the initial value of the original CrawlDatum, and also score values contributed by inlinked pages.
      Specified by:
        `updateDbScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `old` - original datum, with original score. May be null if this is a newly discovered page. If not null, filters should use score values from this parameter as the starting values - the `datum` parameter may contain values that are no longer valid, if other updates occurred between generation and this update.
        `datum` - the new datum, with the original score saved at the time when fetchlist was generated. Filters should update this in-place, and it will be saved in the crawldb.
        `inlinked` - (partial) list of CrawlDatum-s (with their scores) from links pointing to this page, found in the current update batch.
      Throws:
        `ScoringFilterException` - there is a fatal error calculating a new score of `CrawlDatum` during CrawlDb update