org.apache.lucene.analysis.ngram

## Class NGramTokenFilter

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.TokenFilter
      - - org.apache.lucene.analysis.ngram.NGramTokenFilter

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public final class NGramTokenFilter
      extends TokenFilter

  Tokenizes the input into n-grams of the given size(s). As of Lucene 4.4, this token filter:
  - handles supplementary characters correctly,
  - emits all n-grams for the same token at the same position,
  - does not modify offsets,
  - sorts n-grams by their offset in the original token first, then increasing length (meaning that "abc" will give "a", "ab", "abc", "b", "bc", "c").

  You can make this filter use the old behavior by using `Lucene43NGramTokenFilter` but this is not recommended as it will lead to broken `TokenStream`s that will cause highlighting bugs.

  If you were using this `TokenFilter` to perform partial highlighting, this won't work anymore since this filter doesn't update offsets. You should modify your analysis chain to use `NGramTokenizer`, and potentially override `NGramTokenizer.isTokenChar(int)` to perform pre-tokenization.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

      `AttributeSource.State`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `DEFAULT_MAX_NGRAM_SIZE`
    `static int` | `DEFAULT_MIN_NGRAM_SIZE`

    - ### Fields inherited from class org.apache.lucene.analysis.TokenFilter

      `input`

    - ### Fields inherited from class org.apache.lucene.analysis.TokenStream

      `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `NGramTokenFilter(TokenStream input)` Creates NGramTokenFilter with default min and max n-grams.
    `NGramTokenFilter(TokenStream input, int minGram, int maxGram)` Creates NGramTokenFilter with given min and max n-grams.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `boolean` | `incrementToken()` Returns the next token in the stream, or null at EOS.
    `void` | `reset()`

    - ### Methods inherited from class org.apache.lucene.analysis.TokenFilter

      `close, end`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_MIN_NGRAM_SIZE

          public static final int DEFAULT_MIN_NGRAM_SIZE

      See Also:
        Constant Field Values

    - #### DEFAULT_MAX_NGRAM_SIZE

          public static final int DEFAULT_MAX_NGRAM_SIZE

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### NGramTokenFilter

          public NGramTokenFilter(TokenStream input,
                          int minGram,
                          int maxGram)

      Creates NGramTokenFilter with given min and max n-grams.
      Parameters:
        `input` - `TokenStream` holding the input to be tokenized
        `minGram` - the smallest n-gram to generate
        `maxGram` - the largest n-gram to generate

    - #### NGramTokenFilter

          public NGramTokenFilter(TokenStream input)

      Creates NGramTokenFilter with default min and max n-grams.
      Parameters:
        `input` - `TokenStream` holding the input to be tokenized

  - ### Method Detail

    - #### incrementToken

          public final boolean incrementToken()
                                       throws IOException

      Returns the next token in the stream, or null at EOS.
      Specified by:
        `incrementToken` in class `TokenStream`
      Throws:
        `IOException`

    - #### reset

          public void reset()
                     throws IOException

      Overrides:
        `reset` in class `TokenFilter`
      Throws:
        `IOException`

Copyright © 2000-2017 Apache Software Foundation. All Rights Reserved.