Package org.apache.lucene.search.similarities

# Class DFRSimilarity

java.lang.Object
org.apache.lucene.search.similarities.Similarity
org.apache.lucene.search.similarities.SimilarityBase
org.apache.lucene.search.similarities.DFRSimilarity

public class DFRSimilarity extends SimilarityBase
Implements the divergence from randomness (DFR) framework introduced in Gianni Amati and Cornelis Joost Van Rijsbergen. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst. 20, 4 (October 2002), 357-389.

The DFR scoring formula is composed of three separate components: the basic model, the aftereffect and an additional normalization component, represented by the classes `BasicModel`, `AfterEffect` and `Normalization`, respectively. The names of these classes were chosen to match the names of their counterparts in the Terrier IR engine.

To construct a DFRSimilarity, you must specify the implementations for all three components of DFR:

1. `BasicModel`: Basic model of information content:
   - `BasicModelG`: Geometric approximation of Bose-Einstein
   - `BasicModelIn`: Inverse document frequency
   - `BasicModelIne`: Inverse expected document frequency [mixture of Poisson and IDF]
   - `BasicModelIF`: Inverse term frequency [approximation of I(ne)]
2. `AfterEffect`: First normalization of information gain:
   - `AfterEffectL`: Laplace's law of succession
   - `AfterEffectB`: Ratio of two Bernoulli processes
3. `Normalization`: Second (length) normalization:
   - `NormalizationH1`: Uniform distribution of term frequency
   - `NormalizationH2`: term frequency density inversely related to length
   - `NormalizationH3`: term frequency normalization provided by Dirichlet prior
   - `NormalizationZ`: term frequency normalization provided by a Zipfian relation
   - `Normalization.NoNormalization`: no second normalization

Note that qtf, the multiplicity of term-occurrence in the query, is not handled by this implementation.

Note that basic models BE (Limiting form of Bose-Einstein), P (Poisson approximation of the Binomial) and D (Divergence approximation of the Binomial) are not implemented because their formula couldn't be written in a way that makes scores non-decreasing with the normalized term frequency.

See Also:
  - `BasicModel`
  - `AfterEffect`
  - `Normalization`
WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity

  `Similarity.BulkSimScorer, Similarity.SimScorer`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected final AfterEffect`
  `afterEffect`
  The first normalization of the information content.
  `protected final BasicModel`
  `basicModel`
  The basic model for information content.
  `protected final Normalization`
  `normalization`
  The term frequency normalization.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `DFRSimilarity(BasicModel basicModel, AfterEffect afterEffect, Normalization normalization)`
  Creates DFRSimilarity from the three components and using default discountOverlaps value.
  `DFRSimilarity(BasicModel basicModel, AfterEffect afterEffect, Normalization normalization, boolean discountOverlaps)`
  Creates DFRSimilarity from the three components and with the specified discountOverlaps value.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected void`
  `explain(List<Explanation> subs, BasicStats stats, double freq, double docLen)`
  Subclasses should implement this method to explain the score.
  `protected Explanation`
  `explain(BasicStats stats, Explanation freq, double docLen)`
  Explains the score.
  `AfterEffect`
  `getAfterEffect()`
  Returns the first normalization
  `BasicModel`
  `getBasicModel()`
  Returns the basic model of information content
  `Normalization`
  `getNormalization()`
  Returns the second normalization
  `protected double`
  `score(BasicStats stats, double freq, double docLen)`
  Scores the document `doc`.
  `String`
  `toString()`
  Subclasses must override this method to return the name of the Similarity and preferably the values of parameters (if any) as well.

  ### Methods inherited from class org.apache.lucene.search.similarities.SimilarityBase

  `fillBasicStats, log2, newStats, scorer`

  ### Methods inherited from class org.apache.lucene.search.similarities.Similarity

  `computeNorm, getDiscountOverlaps`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### basicModel

    protected final BasicModel basicModel
    The basic model for information content.

  - ### afterEffect

    protected final AfterEffect afterEffect
    The first normalization of the information content.

  - ### normalization

    protected final Normalization normalization
    The term frequency normalization.

- ## Constructor Details

  - ### DFRSimilarity

    public DFRSimilarity(BasicModel basicModel, AfterEffect afterEffect, Normalization normalization)
    Creates DFRSimilarity from the three components and using default discountOverlaps value.

    Note that `null` values are not allowed: if you want no normalization, instead pass `Normalization.NoNormalization`.

    Parameters:
      `basicModel` - Basic model of information content
      `afterEffect` - First normalization of information gain
      `normalization` - Second (length) normalization

  - ### DFRSimilarity

    public DFRSimilarity(BasicModel basicModel, AfterEffect afterEffect, Normalization normalization, boolean discountOverlaps)
    Creates DFRSimilarity from the three components and with the specified discountOverlaps value.

    Note that `null` values are not allowed: if you want no normalization, instead pass `Normalization.NoNormalization`.

    Parameters:
      `basicModel` - Basic model of information content
      `afterEffect` - First normalization of information gain
      `normalization` - Second (length) normalization
      `discountOverlaps` - True if overlap tokens (tokens with a position of increment of zero) are discounted from the document's length.

- ## Method Details

  - ### score

    protected double score(BasicStats stats, double freq, double docLen)
    Description copied from class: `SimilarityBase`
    Scores the document `doc`.

    Subclasses must apply their scoring formula in this class.

    Specified by:
      `score` in class `SimilarityBase`
    Parameters:
      `stats` - the corpus level statistics.
      `freq` - the term frequency.
      `docLen` - the document length.
    Returns:
      the score.

  - ### explain

    protected void explain(List<Explanation> subs, BasicStats stats, double freq, double docLen)
    Description copied from class: `SimilarityBase`
    Subclasses should implement this method to explain the score. `expl` already contains the score, the name of the class and the doc id, as well as the term frequency and its explanation; subclasses can add additional clauses to explain details of their scoring formulae.

    The default implementation does nothing.

    Overrides:
      `explain` in class `SimilarityBase`
    Parameters:
      `subs` - the list of details of the explanation to extend
      `stats` - the corpus level statistics.
      `freq` - the term frequency.
      `docLen` - the document length.

  - ### explain

    protected Explanation explain(BasicStats stats, Explanation freq, double docLen)
    Description copied from class: `SimilarityBase`
    Explains the score. The implementation here provides a basic explanation in the format score(name-of-similarity, doc=doc-id, freq=term-frequency), computed from:, and attaches the score (computed via the `SimilarityBase.score(BasicStats, double, double)` method) and the explanation for the term frequency. Subclasses content with this format may add additional details in `SimilarityBase.explain(List, BasicStats, double, double)`.
    Overrides:
      `explain` in class `SimilarityBase`
    Parameters:
      `stats` - the corpus level statistics.
      `freq` - the term frequency and its explanation.
      `docLen` - the document length.
    Returns:
      the explanation.

  - ### toString

    public String toString()
    Description copied from class: `SimilarityBase`
    Subclasses must override this method to return the name of the Similarity and preferably the values of parameters (if any) as well.
    Specified by:
      `toString` in class `SimilarityBase`

  - ### getBasicModel

    public BasicModel getBasicModel()
    Returns the basic model of information content

  - ### getAfterEffect

    public AfterEffect getAfterEffect()
    Returns the first normalization

  - ### getNormalization

    public Normalization getNormalization()
    Returns the second normalization