org.apache.nutch.scoring.urlmeta

## Class URLMetaScoringFilter

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.scoring.urlmeta.URLMetaScoringFilter

- All Implemented Interfaces:
    Configurable, Pluggable, ScoringFilter

      public class URLMetaScoringFilter
      extends Configured
      implements ScoringFilter

  For documentation: `org.apache.nutch.scoring.urlmeta`

- - ### Field Summary

    - ### Fields inherited from interface org.apache.nutch.scoring.ScoringFilter

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `URLMetaScoringFilter()`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `CrawlDatum` | `distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount)` This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the parseData object.
    `float` | `generatorSortValue(Text url, CrawlDatum datum, float initSort)` Boilerplate
    `Configuration` | `getConf()` Boilerplate
    `float` | `indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)` Boilerplate
    `void` | `initialScore(Text url, CrawlDatum datum)` Boilerplate
    `void` | `injectedScore(Text url, CrawlDatum datum)` Boilerplate
    `void` | `passScoreAfterParsing(Text url, Content content, Parse parse)` Takes the metadata, which was lumped inside the content, and replicates it within your parse data.
    `void` | `passScoreBeforeParsing(Text url, CrawlDatum datum, Content content)` Takes the metadata, specified in your "urlmeta.tags" property, from the datum object and injects it into the content.
    `void` | `setConf(Configuration conf)` handles conf assignment and pulls the value assignment from the "urlmeta.tags" property
    `void` | `updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List<CrawlDatum> inlinked)` Boilerplate

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.nutch.scoring.ScoringFilter

      `orphanedScore`

- - ### Constructor Detail

    - #### URLMetaScoringFilter

          public URLMetaScoringFilter()

  - ### Method Detail

    - #### distributeScoreToOutlinks

          public CrawlDatum distributeScoreToOutlinks(Text fromUrl,
                                                      ParseData parseData,
                                                      Collection<Map.Entry<Text,CrawlDatum>> targets,
                                                      CrawlDatum adjust,
                                                      int allCount)
                                               throws ScoringFilterException

      This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the parseData object. If they exist, this will be propagated into your 'targets' Collection's ["outlinks"] attributes.
      Specified by:
        `distributeScoreToOutlinks` in interface `ScoringFilter`
      Parameters:
        `fromUrl` - url of the source page
        `parseData` - ParseData instance, which stores relevant score value(s) in its metadata. NOTE: filters may modify this in-place, all changes will be persisted.
        `targets` - <url, CrawlDatum> pairs. NOTE: filters can modify this in-place, all changes will be persisted.
        `adjust` - a CrawlDatum instance, initially null, which implementations may use to pass adjustment values to the original CrawlDatum. When creating this instance, set its status to `CrawlDatum.STATUS_LINKED`.
        `allCount` - number of all collected outlinks from the source page
      Returns:
        if needed, implementations may return an instance of CrawlDatum, with status `CrawlDatum.STATUS_LINKED`, which contains adjustments to be applied to the original CrawlDatum score(s) and metadata. This can be null if not needed.
      Throws:
        `ScoringFilterException`
      See Also:
        `ScoringFilter.distributeScoreToOutlinks(org.apache.hadoop.io.Text, org.apache.nutch.parse.ParseData, java.util.Collection<java.util.Map.Entry<org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum>>, org.apache.nutch.crawl.CrawlDatum, int)`

    - #### passScoreBeforeParsing

          public void passScoreBeforeParsing(Text url,
                                             CrawlDatum datum,
                                             Content content)

      Takes the metadata, specified in your "urlmeta.tags" property, from the datum object and injects it into the content. This is transfered to the parseData object.
      Specified by:
        `passScoreBeforeParsing` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - source datum. NOTE: modifications to this value are not persisted.
        `content` - instance of content. Implementations may modify this in-place, primarily by setting some metadata properties.
      See Also:
        `ScoringFilter.passScoreBeforeParsing(org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum, org.apache.nutch.protocol.Content)`, `passScoreAfterParsing(org.apache.hadoop.io.Text, org.apache.nutch.protocol.Content, org.apache.nutch.parse.Parse)`

    - #### passScoreAfterParsing

          public void passScoreAfterParsing(Text url,
                                            Content content,
                                            Parse parse)

      Takes the metadata, which was lumped inside the content, and replicates it within your parse data.
      Specified by:
        `passScoreAfterParsing` in interface `ScoringFilter`
      Parameters:
        `url` - page url
        `content` - original content. NOTE: modifications to this value are not persisted.
        `parse` - target instance to copy the score information to. Implementations may modify this in-place, primarily by setting some metadata properties.
      See Also:
        `passScoreBeforeParsing(org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum, org.apache.nutch.protocol.Content)`, `ScoringFilter.passScoreAfterParsing(org.apache.hadoop.io.Text, org.apache.nutch.protocol.Content, org.apache.nutch.parse.Parse)`

    - #### generatorSortValue

          public float generatorSortValue(Text url,
                                          CrawlDatum datum,
                                          float initSort)
                                   throws ScoringFilterException

      Boilerplate
      Specified by:
        `generatorSortValue` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - page's datum, should not be modified
        `initSort` - initial sort value, or a value from previous filters in chain
      Throws:
        `ScoringFilterException`

    - #### indexerScore

          public float indexerScore(Text url,
                                    NutchDocument doc,
                                    CrawlDatum dbDatum,
                                    CrawlDatum fetchDatum,
                                    Parse parse,
                                    Inlinks inlinks,
                                    float initScore)
                             throws ScoringFilterException

      Boilerplate
      Specified by:
        `indexerScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `doc` - indexed document. NOTE: this already contains all information collected by indexing filters. Implementations may modify this instance, in order to store/remove some information.
        `dbDatum` - current page from CrawlDb. NOTE:
        - changes made to this instance are not persisted
        - may be null if indexing is done without CrawlDb or if the segment is generated not from the CrawlDb (via FreeGenerator).
        `fetchDatum` - datum from FetcherOutput (containing among others the fetching status)
        `parse` - parsing result. NOTE: changes made to this instance are not persisted.
        `inlinks` - current inlinks from LinkDb. NOTE: changes made to this instance are not persisted.
        `initScore` - initial boost value for the indexed document.
      Returns:
        boost value for the indexed document. This value is passed as an argument to the next scoring filter in chain. NOTE: implementations may also express other scoring strategies by modifying the indexed document directly.
      Throws:
        `ScoringFilterException`

    - #### initialScore

          public void initialScore(Text url,
                                   CrawlDatum datum)
                            throws ScoringFilterException

      Boilerplate
      Specified by:
        `initialScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - new datum. Filters will modify it in-place.
      Throws:
        `ScoringFilterException`

    - #### injectedScore

          public void injectedScore(Text url,
                                    CrawlDatum datum)
                             throws ScoringFilterException

      Boilerplate
      Specified by:
        `injectedScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - new datum. Filters will modify it in-place.
      Throws:
        `ScoringFilterException`

    - #### updateDbScore

          public void updateDbScore(Text url,
                                    CrawlDatum old,
                                    CrawlDatum datum,
                                    List<CrawlDatum> inlinked)
                             throws ScoringFilterException

      Boilerplate
      Specified by:
        `updateDbScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `old` - original datum, with original score. May be null if this is a newly discovered page. If not null, filters should use score values from this parameter as the starting values - the `datum` parameter may contain values that are no longer valid, if other updates occurred between generation and this update.
        `datum` - the new datum, with the original score saved at the time when fetchlist was generated. Filters should update this in-place, and it will be saved in the crawldb.
        `inlinked` - (partial) list of CrawlDatum-s (with their scores) from links pointing to this page, found in the current update batch.
      Throws:
        `ScoringFilterException`

    - #### setConf

          public void setConf(Configuration conf)

      handles conf assignment and pulls the value assignment from the "urlmeta.tags" property
      Specified by:
        `setConf` in interface `Configurable`
      Overrides:
        `setConf` in class `Configured`

    - #### getConf

          public Configuration getConf()

      Boilerplate
      Specified by:
        `getConf` in interface `Configurable`
      Overrides:
        `getConf` in class `Configured`

Copyright © 2021 The Apache Software Foundation