Package org.apache.lucene.analysis.cjk

# Class CJKBigramFilter

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.TokenFilter
org.apache.lucene.analysis.cjk.CJKBigramFilter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `Unwrappable<TokenStream>`

public final class CJKBigramFilter extends TokenFilter
Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

CJK types are set by these tokenizers, but you can also use `CJKBigramFilter(TokenStream, int)` to explicitly control which of the CJK scripts are turned into bigrams.

By default, when a CJK character has no adjacent characters to form a bigram, it is output in unigram form. If you want to always output both unigrams and bigrams, set the `outputUnigrams` flag in `CJKBigramFilter(TokenStream, int, boolean)`. This can be used for a combined unigram+bigram approach.

Unlike ICUTokenizer, StandardTokenizer does not split at script boundaries. Korean Hangul characters are treated the same as many other scripts' letters, and as a result, StandardTokenizer can produce tokens that mix Hangul and non-Hangul characters, e.g. "한국abc". Such mixed-script tokens are typed as `<ALPHANUM>` rather than `<HANGUL>`, and as a result, will not be converted to bigrams by CJKBigramFilter.

In all cases, all non-CJK input is passed thru unmodified.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `DOUBLE_TYPE`
  when we emit a bigram, it's then marked as this type
  `static final int`
  `HAN`
  bigram flag for Han Ideographs
  `static final int`
  `HANGUL`
  bigram flag for Hangul
  `static final int`
  `HIRAGANA`
  bigram flag for Hiragana
  `static final int`
  `KATAKANA`
  bigram flag for Katakana
  `static final String`
  `SINGLE_TYPE`
  when we emit a unigram, it's then marked as this type

  ### Fields inherited from class org.apache.lucene.analysis.TokenFilter

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `CJKBigramFilter(TokenStream in)`
  Calls `CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)`
  `CJKBigramFilter(TokenStream in, int flags)`
  Calls `CJKBigramFilter(in, flags, false)`
  `CJKBigramFilter(TokenStream in, int flags, boolean outputUnigrams)`
  Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `boolean`
  `incrementToken()`
  `void`
  `reset()`

  ### Methods inherited from class org.apache.lucene.analysis.TokenFilter

  `close, end, unwrap`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### HAN

    public static final int HAN
    bigram flag for Han Ideographs
    See Also:
      - Constant Field Values

  - ### HIRAGANA

    public static final int HIRAGANA
    bigram flag for Hiragana
    See Also:
      - Constant Field Values

  - ### KATAKANA

    public static final int KATAKANA
    bigram flag for Katakana
    See Also:
      - Constant Field Values

  - ### HANGUL

    public static final int HANGUL
    bigram flag for Hangul
    See Also:
      - Constant Field Values

  - ### DOUBLE_TYPE

    public static final String DOUBLE_TYPE
    when we emit a bigram, it's then marked as this type
    See Also:
      - Constant Field Values

  - ### SINGLE_TYPE

    public static final String SINGLE_TYPE
    when we emit a unigram, it's then marked as this type
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### CJKBigramFilter

    public CJKBigramFilter(TokenStream in)
    Calls `CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)`

  - ### CJKBigramFilter

    public CJKBigramFilter(TokenStream in, int flags)
    Calls `CJKBigramFilter(in, flags, false)`

  - ### CJKBigramFilter

    public CJKBigramFilter(TokenStream in, int flags, boolean outputUnigrams)
    Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.
    Parameters:
      `flags` - OR'ed set from `HAN`, `HIRAGANA`, `KATAKANA`, `HANGUL`
      `outputUnigrams` - true if unigrams for the selected writing systems should also be output. when this is false, this is only done when there are no adjacent characters to form a bigram.

- ## Method Details

  - ### incrementToken

    public boolean incrementToken() throws IOException
    Specified by:
      `incrementToken` in class `TokenStream`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    Overrides:
      `reset` in class `TokenFilter`
    Throws:
      `IOException`