Package org.apache.lucene.analysis.morph

# Class Viterbi<T extends Token,U extends Viterbi.Position>

java.lang.Object
org.apache.lucene.analysis.morph.Viterbi<T,U>

Type Parameters:
  `T` - output token class
  `U` - position class

Direct Known Subclasses:
  `ViterbiNBest`

public abstract class Viterbi<T extends Token,U extends Viterbi.Position> extends Object
Performs Viterbi algorithm for morphological Tokenizers, which split texts by Hidden Markov Model or Conditional Random Fields.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `Viterbi.Position`
  Holds all back pointers arriving to this position.
  `static final class`
  `Viterbi.WrappedPositionArray<U extends Viterbi.Position>`
  Holds partial graph (array of positions) for calculating the minimum cost path

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected final RollingCharBuffer`
  `buffer`
  `protected final ConnectionCosts`
  `costs`
  `protected boolean`
  `enableSpacePenaltyFactor`
  `protected boolean`
  `end`
  `protected int`
  `lastBackTracePos`
  `protected static final int`
  `MAX_UNKNOWN_WORD_LENGTH`
  `protected boolean`
  `outputLongestUserEntryOnly`
  `protected boolean`
  `outputNBest`
  `protected final List<T>`
  `pending`
  `protected int`
  `pos`
  `protected final Viterbi.WrappedPositionArray<U>`
  `positions`
  `protected static final boolean`
  `VERBOSE`
  `protected final IntsRef`
  `wordIdRef`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `Viterbi(TokenInfoFST fst, FST.BytesReader fstReader, BinaryDictionary<? extends MorphData> dictionary, TokenInfoFST userFST, FST.BytesReader userFSTReader, Dictionary<? extends MorphData> userDictionary, ConnectionCosts costs, Class<U> positionImpl)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected final void`
  `add(MorphData morphData, Viterbi.Position fromPosData, int wordPos, int endPos, int wordID, TokenType type, boolean addPenalty)`
  Add a token on the minimum cost path to the pending token list.
  `protected abstract void`
  `backtrace(Viterbi.Position endPosData, int fromIDX)`
  Backtrace from the provided position, back to the last time we back-traced, accumulating the resulting tokens to the pending list.
  `protected void`
  `backtraceNBest(Viterbi.Position endPosData, boolean useEOS)`
  Backtrace the n-best path.
  `protected int`
  `computePenalty(int pos, int length)`
  Returns the penalty for a specific input region
  `protected int`
  `computeSpacePenalty(MorphData morphData, int wordID, int numSpaces)`
  Returns the space penalty.
  `protected void`
  `fixupPendingList()`
  Remove duplicated tokens from the pending list; this is needed because `backtrace(Position, int)` and `backtraceNBest(Position, boolean)` can add same tokens to the list.
  `final void`
  `forward()`
  Incrementally parse some more characters.
  `List<T>`
  `getPending()`
  `int`
  `getPos()`
  `boolean`
  `isEnd()`
  `boolean`
  `isOutputNBest()`
  `protected abstract int`
  `processUnknownWord(boolean anyMatches, Viterbi.Position posData)`
  Add unknown words to the position graph.
  `void`
  `resetBuffer(Reader reader)`
  `void`
  `resetState()`
  `protected boolean`
  `shouldSkipProcessUnknownWord(int unknownWordEndIndex, Viterbi.Position posData)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### VERBOSE

    protected static final boolean VERBOSE
    See Also:
      - Constant Field Values

  - ### MAX_UNKNOWN_WORD_LENGTH

    protected static final int MAX_UNKNOWN_WORD_LENGTH
    See Also:
      - Constant Field Values

  - ### costs

    protected final ConnectionCosts costs

  - ### wordIdRef

    protected final IntsRef wordIdRef

  - ### buffer

    protected final RollingCharBuffer buffer

  - ### positions

    protected final Viterbi.WrappedPositionArray<U extends Viterbi.Position> positions

  - ### end

    protected boolean end

  - ### lastBackTracePos

    protected int lastBackTracePos

  - ### pos

    protected int pos

  - ### pending

    protected final List<T extends Token> pending

  - ### outputNBest

    protected boolean outputNBest

  - ### enableSpacePenaltyFactor

    protected boolean enableSpacePenaltyFactor

  - ### outputLongestUserEntryOnly

    protected boolean outputLongestUserEntryOnly

- ## Constructor Details

  - ### Viterbi

    protected Viterbi(TokenInfoFST fst, FST.BytesReader fstReader, BinaryDictionary<? extends MorphData> dictionary, TokenInfoFST userFST, FST.BytesReader userFSTReader, Dictionary<? extends MorphData> userDictionary, ConnectionCosts costs, Class<U> positionImpl)

- ## Method Details

  - ### forward

    public final void forward() throws IOException
    Incrementally parse some more characters. This runs the viterbi search forwards "enough" so that we generate some more tokens. How much forward depends on the chars coming in, since some chars could cause longer-lasting ambiguity in the parsing. Once the ambiguity is resolved, then we back trace, produce the pending tokens, and return.
    Throws:
      `IOException`

  - ### shouldSkipProcessUnknownWord

    protected boolean shouldSkipProcessUnknownWord(int unknownWordEndIndex, Viterbi.Position posData)

  - ### processUnknownWord

    protected abstract int processUnknownWord(boolean anyMatches, Viterbi.Position posData) throws IOException
    Add unknown words to the position graph.
    Returns:
      word length
    Throws:
      `IOException`

  - ### backtrace

    protected abstract void backtrace(Viterbi.Position endPosData, int fromIDX) throws IOException
    Backtrace from the provided position, back to the last time we back-traced, accumulating the resulting tokens to the pending list. The pending list is then in-reverse (last token should be returned first).
    Throws:
      `IOException`

  - ### backtraceNBest

    protected void backtraceNBest(Viterbi.Position endPosData, boolean useEOS) throws IOException
    Backtrace the n-best path. Subclasses that support n-best paths should implement this method.
    Throws:
      `IOException`

  - ### fixupPendingList

    protected void fixupPendingList()
    Remove duplicated tokens from the pending list; this is needed because `backtrace(Position, int)` and `backtraceNBest(Position, boolean)` can add same tokens to the list. Subclasses that support n-best paths should implement this method.

  - ### add

    protected final void add(MorphData morphData, Viterbi.Position fromPosData, int wordPos, int endPos, int wordID, TokenType type, boolean addPenalty) throws IOException
    Add a token on the minimum cost path to the pending token list.
    Throws:
      `IOException`

  - ### computeSpacePenalty

    protected int computeSpacePenalty(MorphData morphData, int wordID, int numSpaces)
    Returns the space penalty.

  - ### computePenalty

    protected int computePenalty(int pos, int length) throws IOException
    Returns the penalty for a specific input region
    Throws:
      `IOException`

  - ### getPos

    public int getPos()

  - ### isEnd

    public boolean isEnd()

  - ### getPending

    public List<T> getPending()

  - ### isOutputNBest

    public boolean isOutputNBest()

  - ### resetBuffer

    public void resetBuffer(Reader reader)

  - ### resetState

    public void resetState()